pg.ddx.io pgsql-bugs@postgresql.org mailing list archive
help / color / mirror / Atom feedLogical replication 'invalid memory alloc request size 1585837200' after upgrading to 17.5
94+ messages / 10 participants
[nested] [flat]
* Logical replication 'invalid memory alloc request size 1585837200' after upgrading to 17.5
@ 2025-05-19 11:46 Duncan Sands <duncan.sands@deepbluecap.com>
0 siblings, 3 replies; 94+ messages in thread
From: Duncan Sands @ 2025-05-19 11:46 UTC (permalink / raw)
To: pgsql-bugs@lists.postgresql.org
PostgreSQL v17.5 (Ubuntu 17.5-1.pgdg24.04+1); Ubuntu 24.04.2 LTS (kernel
6.8.0); x86-64
Good morning from DeepBlueCapital. Soon after upgrading to 17.5 from 17.4, we
started seeing logical replication failures with publisher errors like this:
ERROR: invalid memory alloc request size 1196493216
(the exact size varies). Here is a typical log extract from the publisher:
2025-05-19 10:30:14 CEST \[1348336-465] remote\_production\_user\@blue DEBUG:
00000: write FB03/349DEF90 flush FB03/349DEF90 apply FB03/349DEF90 reply\_time
2025-05-19 10:30:07.467048+02
2025-05-19 10:30:14 CEST \[1348336-466] remote\_production\_user\@blue LOCATION:
ProcessStandbyReplyMessage, walsender.c:2431
2025-05-19 10:30:14 CEST \[1348336-467] remote\_production\_user\@blue DEBUG:
00000: skipped replication of an empty transaction with XID: 207637565
2025-05-19 10:30:14 CEST \[1348336-468] remote\_production\_user\@blue CONTEXT:
slot "jnb\_production", output plugin "pgoutput", in the commit callback,
associated LSN FB03/349FF938
2025-05-19 10:30:14 CEST \[1348336-469] remote\_production\_user\@blue LOCATION:
pgoutput\_commit\_txn, pgoutput.c:629
2025-05-19 10:30:14 CEST \[1348336-470] remote\_production\_user\@blue DEBUG:
00000: UpdateDecodingStats: updating stats 0x5ae1616c17a8 0 0 0 0 1 0 1 191
2025-05-19 10:30:14 CEST \[1348336-471] remote\_production\_user\@blue LOCATION:
UpdateDecodingStats, logical.c:1943
2025-05-19 10:30:14 CEST \[1348336-472] remote\_production\_user\@blue DEBUG:
00000: found top level transaction 207637519, with catalog changes
2025-05-19 10:30:14 CEST \[1348336-473] remote\_production\_user\@blue LOCATION:
SnapBuildCommitTxn, snapbuild.c:1150
2025-05-19 10:30:14 CEST \[1348336-474] remote\_production\_user\@blue DEBUG:
00000: adding a new snapshot and invalidations to 207616976 at FB03/34A1AAE0
2025-05-19 10:30:14 CEST \[1348336-475] remote\_production\_user\@blue LOCATION:
SnapBuildDistributeSnapshotAndInval, snapbuild.c:915
2025-05-19 10:30:14 CEST \[1348336-476] remote\_production\_user\@blue ERROR:
XX000: invalid memory alloc request size 1196493216
If I'm reading it right, things go wrong on the publisher while preparing the
message, i.e. it's not a subscriber problem.
This particular instance was triggered by a large number of catalog
invalidations: I dumped what I think is the relevant WAL with "pg_waldump -s
FB03/34A1AAE0 -p 17/main/ --xid=207637519" and the output was a single long line:
rmgr: Transaction len (rec/tot): 10665/ 10665, tx: 207637519, lsn:
FB03/34A1AAE0, prev FB03/34A1A8C8, desc: COMMIT 2025-05-19 08:10:12.880599 CEST;
dropped stats: 2/17426/661557718 2/17426/661557717 2/17426/661557714
2/17426/661557678 2/17426/661557677 2/17426/661557674 2/17426/661557673
2/17426/661557672 2/17426/661557669 2/17426/661557618 2/17426/661557617
2/17426/661557614; inval msgs: catcache 80 catcache 79 catcache 80 catcache 79
catcache 55 catcache 54 catcache 7 catcache 6 catcache 7 catcache 6 catcache 7
catcache 6 catcache 7 catcache 6 catcache 7 catcache 6 catcache 7 catcache 6
catcache 7 catcache 6 catcache 7 catcache 6 catcache 7 catcache 6 catcache 7
catcache 6 catcache 7 catcache 6 catcache 7 catcache 6 catcache 7 catcache 6
catcache 55 catcache 54 catcache 7 catcache 6 catcache 7 catcache 6 catcache 7
catcache 6 catcache 7 catcache 6 catcache 7 catcache 6 catcache 7 catcache 6
catcache 7 catcache 6 catcache 7 catcache 6 catcache 7 catcache 6 catcache 55
catcache 54 catcache 7 catcache 6 catcache 7 catcache 6 catcache 32 catcache 55
catcache 54 catcache 55 catcache 54 catcache 55 catcache 54 catcache 80 catcache
79 catcache 80 catcache 79 catcache 55 catcache 54 catcache 7 catcache 6
catcache 7 catcache 6 catcache 7 catcache 6 catcache 7 catcache 6 catcache 7
catcache 6 catcache 7 catcache 6 catcache 7 catcache 6 catcache 7 catcache 6
catcache 7 catcache 6 catcache 7 catcache 6 catcache 7 catcache 6 catcache 7
catcache 6 catcache 7 catcache 6 catcache 55 catcache 54 catcache 7 catcache 6
catcache 7 catcache 6 catcache 7 catcache 6 catcache 7 catcache 6 catcache 7
catcache 6 catcache 7 catcache 6 catcache 7 catcache 6 catcache 7 catcache 6
catcache 7 catcache 6 catcache 55 catcache 54 catcache 7 catcache 6 catcache 7
catcache 6 catcache 32 catcache 55 catcache 54 catcache 55 catcache 54 catcache
55 catcache 54 catcache 63 catcache 63 catcache 63 catcache 63 catcache 63
catcache 63 catcache 63 catcache 55 catcache 54 catcache 80 catcache 79 catcache
80 catcache 79 catcache 55 catcache 54 catcache 7 catcache 6 catcache 7 catcache
6 catcache 7 catcache 6 catcache 7 catcache 6 catcache 7 catcache 6 catcache 7
catcache 6 catcache 7 catcache 6 catcache 7 catcache 6 catcache 7 catcache 6
catcache 7 catcache 6 catcache 7 catcache 6 catcache 7 catcache 6 catcache 7
catcache 6 catcache 7 catcache 6 catcache 7 catcache 6 catcache 7 catcache 6
catcache 7 catcache 6 catcache 55 catcache 54 catcache 7 catcache 6 catcache 7
catcache 6 catcache 7 catcache 6 catcache 7 catcache 6 catcache 7 catcache 6
catcache 7 catcache 6 catcache 7 catcache 6 catcache 7 catcache 6 catcache 7
catcache 6 catcache 55 catcache 54 catcache 7 catcache 6 catcache 7 catcache 6
catcache 32 catcache 55 catcache 54 catcache 55 catcache 54 catcache 55 catcache
54 catcache 80 catcache 79 catcache 80 catcache 79 catcache 55 catcache 54
catcache 7 catcache 6 catcache 7 catcache 6 catcache 7 catcache 6 catcache 7
catcache 6 catcache 7 catcache 6 catcache 7 catcache 6 catcache 7 catcache 6
catcache 7 catcache 6 catcache 7 catcache 6 catcache 7 catcache 6 catcache 7
catcache 6 catcache 7 catcache 6 catcache 7 catcache 6 catcache 7 catcache 6
catcache 7 catcache 6 catcache 7 catcache 6 catcache 7 catcache 6 catcache 55
catcache 54 catcache 7 catcache 6 catcache 7 catcache 6 catcache 7 catcache 6
catcache 7 catcache 6 catcache 7 catcache 6 catcache 7 catcache 6 catcache 7
catcache 6 catcache 7 catcache 6 catcache 7 catcache 6 catcache 55 catcache 54
catcache 7 catcache 6 catcache 7 catcache 6 catcache 32 catcache 55 catcache 54
catcache 55 catcache 54 catcache 55 catcache 54 catcache 63 catcache 63 catcache
63 catcache 63 catcache 63 catcache 63 catcache 63 catcache 63 catcache 63
catcache 63 catcache 63 catcache 55 catcache 54 catcache 32 catcache 7 catcache
6 catcache 7 catcache 6 catcache 55 catcache 54 catcache 7 catcache 6 catcache 7
catcache 6 catcache 7 catcache 6 catcache 7 catcache 6 catcache 7 catcache 6
catcache 7 catcache 6 catcache 7 catcache 6 catcache 7 catcache 6 catcache 7
catcache 6 catcache 55 catcache 54 catcache 80 catcache 79 catcache 80 catcache
79 catcache 63 catcache 63 catcache 63 catcache 63 catcache 63 catcache 63
catcache 63 catcache 63 catcache 63 catcache 63 catcache 63 catcache 7 catcache
6 catcache 7 catcache 6 catcache 7 catcache 6 catcache 7 catcache 6 catcache 7
catcache 6 catcache 7 catcache 6 catcache 7 catcache 6 catcache 7 catcache 6
catcache 7 catcache 6 catcache 7 catcache 6 catcache 7 catcache 6 catcache 7
catcache 6 catcache 7 catcache 6 catcache 7 catcache 6 catcache 7 catcache 6
catcache 7 catcache 6 catcache 7 catcache 6 catcache 55 catcache 54 catcache 32
catcache 7 catcache 6 catcache 7 catcache 6 catcache 55 catcache 54 catcache 7
catcache 6 catcache 7 catcache 6 catcache 7 catcache 6 catcache 7 catcache 6
catcache 7 catcache 6 catcache 7 catcache 6 catcache 7 catcache 6 catcache 7
catcache 6 catcache 7 catcache 6 catcache 55 catcache 54 catcache 80 catcache 79
catcache 80 catcache 79 catcache 7 catcache 6 catcache 7 catcache 6 catcache 7
catcache 6 catcache 7 catcache 6 catcache 7 catcache 6 catcache 7 catcache 6
catcache 7 catcache 6 catcache 7 catcache 6 catcache 7 catcache 6 catcache 7
catcache 6 catcache 7 catcache 6 catcache 7 catcache 6 catcache 7 catcache 6
catcache 7 catcache 6 catcache 7 catcache 6 catcache 7 catcache 6 catcache 7
catcache 6 catcache 55 catcache 54 catcache 32 catcache 7 catcache 6 catcache 7
catcache 6 catcache 55 catcache 54 catcache 7 catcache 6 catcache 7 catcache 6
catcache 7 catcache 6 catcache 7 catcache 6 catcache 7 catcache 6 catcache 7
catcache 6 catcache 7 catcache 6 catcache 7 catcache 6 catcache 7 catcache 6
catcache 55 catcache 54 catcache 80 catcache 79 catcache 80 catcache 79 catcache
63 catcache 63 catcache 63 catcache 63 catcache 63 catcache 63 catcache 63
catcache 7 catcache 6 catcache 7 catcache 6 catcache 7 catcache 6 catcache 7
catcache 6 catcache 7 catcache 6 catcache 7 catcache 6 catcache 7 catcache 6
catcache 7 catcache 6 catcache 7 catcache 6 catcache 7 catcache 6 catcache 7
catcache 6 catcache 7 catcache 6 catcache 7 catcache 6 catcache 55 catcache 54
catcache 32 catcache 7 catcache 6 catcache 7 catcache 6 catcache 55 catcache 54
catcache 7 catcache 6 catcache 7 catcache 6 catcache 7 catcache 6 catcache 7
catcache 6 catcache 7 catcache 6 catcache 7 catcache 6 catcache 7 catcache 6
catcache 7 catcache 6 catcache 7 catcache 6 catcache 55 catcache 54 catcache 80
catcache 79 catcache 80 catcache 79 catcache 7 catcache 6 catcache 7 catcache 6
catcache 7 catcache 6 catcache 7 catcache 6 catcache 7 catcache 6 catcache 7
catcache 6 catcache 7 catcache 6 catcache 7 catcache 6 catcache 7 catcache 6
catcache 7 catcache 6 catcache 7 catcache 6 catcache 7 catcache 6 catcache 7
catcache 6 catcache 55 catcache 54 snapshot 2608 relcache 661557614 snapshot
1214 relcache 661557617 relcache 661557618 relcache 661557617 snapshot 2608
relcache 661557617 relcache 661557618 relcache 661557614 snapshot 2608 snapshot
2608 relcache 661557669 snapshot 1214 relcache 661557672 relcache 661557673
relcache 661557672 snapshot 2608 relcache 661557672 relcache 661557673 relcache
661557669 snapshot 2608 relcache 661557669 snapshot 2608 relcache 661557674
snapshot 1214 relcache 661557677 relcache 661557678 relcache 661557677 snapshot
2608 relcache 661557677 relcache 661557678 relcache 661557674 snapshot 2608
snapshot 2608 relcache 661557714 snapshot 1214 relcache 661557717 relcache
661557718 relcache 661557717 snapshot 2608 relcache 661557717 relcache 661557718
relcache 661557714 snapshot 2608 relcache 661557714 relcache 661557718 relcache
661557717 snapshot 2608 relcache 661557717 snapshot 2608 snapshot 2608 snapshot
2608 relcache 661557714 snapshot 2608 snapshot 1214 relcache 661557678 relcache
661557677 snapshot 2608 relcache 661557677 snapshot 2608 snapshot 2608 snapshot
2608 relcache 661557674 snapshot 2608 snapshot 1214 relcache 661557673 relcache
661557672 snapshot 2608 relcache 661557672 snapshot 2608 snapshot 2608 snapshot
2608 relcache 661557669 snapshot 2608 snapshot 1214 relcache 661557618 relcache
661557617 snapshot 2608 relcache 661557617 snapshot 2608 snapshot 2608 snapshot
2608 relcache 661557614 snapshot 2608 snapshot 1214
While it is long, it doesn't seem to merit allocating anything like 1GB of
memory. So I'm guessing that postgres is miscalculating the required size somehow.
If I skip over this LSN, for example by dropping the subscription and recreating
it anew, then things go fine for a while before hitting another "invalid memory
alloc request", i.e. it wasn't just a one-off. On the other hand, after
downgrading to 17.4, subscribers spontaneously recovered and the issue has gone
way. Since I didn't skip over the last LSN of this kind, presumably 17.4
successfully serialized a message for the same problematic bit of WAL that
caused 17.5 to blow up, which suggests a regression between 17.4 and 17.5.
Best wishes, Duncan.
^ permalink raw reply [nested|flat] 94+ messages in thread
* Re: Logical replication 'invalid memory alloc request size 1585837200' after upgrading to 17.5
@ 2025-05-21 05:46 Shlok Kyal <shlok.kyal.oss@gmail.com>
parent: Duncan Sands <duncan.sands@deepbluecap.com>
2 siblings, 0 replies; 94+ messages in thread
From: Shlok Kyal @ 2025-05-21 05:46 UTC (permalink / raw)
To: Duncan Sands <duncan.sands@deepbluecap.com>; +Cc: pgsql-bugs@lists.postgresql.org
On Mon, 19 May 2025 at 20:08, Duncan Sands <duncan.sands@deepbluecap.com> wrote:
>
> PostgreSQL v17.5 (Ubuntu 17.5-1.pgdg24.04+1); Ubuntu 24.04.2 LTS (kernel
> 6.8.0); x86-64
>
> Good morning from DeepBlueCapital. Soon after upgrading to 17.5 from 17.4, we
> started seeing logical replication failures with publisher errors like this:
>
> ERROR: invalid memory alloc request size 1196493216
>
> (the exact size varies). Here is a typical log extract from the publisher:
>
> 2025-05-19 10:30:14 CEST \[1348336-465] remote\_production\_user\@blue DEBUG:
> 00000: write FB03/349DEF90 flush FB03/349DEF90 apply FB03/349DEF90 reply\_time
> 2025-05-19 10:30:07.467048+02
> 2025-05-19 10:30:14 CEST \[1348336-466] remote\_production\_user\@blue LOCATION:
> ProcessStandbyReplyMessage, walsender.c:2431
> 2025-05-19 10:30:14 CEST \[1348336-467] remote\_production\_user\@blue DEBUG:
> 00000: skipped replication of an empty transaction with XID: 207637565
> 2025-05-19 10:30:14 CEST \[1348336-468] remote\_production\_user\@blue CONTEXT:
> slot "jnb\_production", output plugin "pgoutput", in the commit callback,
> associated LSN FB03/349FF938
> 2025-05-19 10:30:14 CEST \[1348336-469] remote\_production\_user\@blue LOCATION:
> pgoutput\_commit\_txn, pgoutput.c:629
> 2025-05-19 10:30:14 CEST \[1348336-470] remote\_production\_user\@blue DEBUG:
> 00000: UpdateDecodingStats: updating stats 0x5ae1616c17a8 0 0 0 0 1 0 1 191
> 2025-05-19 10:30:14 CEST \[1348336-471] remote\_production\_user\@blue LOCATION:
> UpdateDecodingStats, logical.c:1943
> 2025-05-19 10:30:14 CEST \[1348336-472] remote\_production\_user\@blue DEBUG:
> 00000: found top level transaction 207637519, with catalog changes
> 2025-05-19 10:30:14 CEST \[1348336-473] remote\_production\_user\@blue LOCATION:
> SnapBuildCommitTxn, snapbuild.c:1150
> 2025-05-19 10:30:14 CEST \[1348336-474] remote\_production\_user\@blue DEBUG:
> 00000: adding a new snapshot and invalidations to 207616976 at FB03/34A1AAE0
> 2025-05-19 10:30:14 CEST \[1348336-475] remote\_production\_user\@blue LOCATION:
> SnapBuildDistributeSnapshotAndInval, snapbuild.c:915
> 2025-05-19 10:30:14 CEST \[1348336-476] remote\_production\_user\@blue ERROR:
> XX000: invalid memory alloc request size 1196493216
>
> If I'm reading it right, things go wrong on the publisher while preparing the
> message, i.e. it's not a subscriber problem.
>
> This particular instance was triggered by a large number of catalog
> invalidations: I dumped what I think is the relevant WAL with "pg_waldump -s
> FB03/34A1AAE0 -p 17/main/ --xid=207637519" and the output was a single long line:
>
> rmgr: Transaction len (rec/tot): 10665/ 10665, tx: 207637519, lsn:
> FB03/34A1AAE0, prev FB03/34A1A8C8, desc: COMMIT 2025-05-19 08:10:12.880599 CEST;
> dropped stats: 2/17426/661557718 2/17426/661557717 2/17426/661557714
> 2/17426/661557678 2/17426/661557677 2/17426/661557674 2/17426/661557673
> 2/17426/661557672 2/17426/661557669 2/17426/661557618 2/17426/661557617
> 2/17426/661557614; inval msgs: catcache 80 catcache 79 catcache 80 catcache 79
> catcache 55 catcache 54 catcache 7 catcache 6 catcache 7 catcache 6 catcache 7
> catcache 6 catcache 7 catcache 6 catcache 7 catcache 6 catcache 7 catcache 6
> catcache 7 catcache 6 catcache 7 catcache 6 catcache 7 catcache 6 catcache 7
> catcache 6 catcache 7 catcache 6 catcache 7 catcache 6 catcache 7 catcache 6
> catcache 55 catcache 54 catcache 7 catcache 6 catcache 7 catcache 6 catcache 7
> catcache 6 catcache 7 catcache 6 catcache 7 catcache 6 catcache 7 catcache 6
> catcache 7 catcache 6 catcache 7 catcache 6 catcache 7 catcache 6 catcache 55
> catcache 54 catcache 7 catcache 6 catcache 7 catcache 6 catcache 32 catcache 55
> catcache 54 catcache 55 catcache 54 catcache 55 catcache 54 catcache 80 catcache
> 79 catcache 80 catcache 79 catcache 55 catcache 54 catcache 7 catcache 6
> catcache 7 catcache 6 catcache 7 catcache 6 catcache 7 catcache 6 catcache 7
> catcache 6 catcache 7 catcache 6 catcache 7 catcache 6 catcache 7 catcache 6
> catcache 7 catcache 6 catcache 7 catcache 6 catcache 7 catcache 6 catcache 7
> catcache 6 catcache 7 catcache 6 catcache 55 catcache 54 catcache 7 catcache 6
> catcache 7 catcache 6 catcache 7 catcache 6 catcache 7 catcache 6 catcache 7
> catcache 6 catcache 7 catcache 6 catcache 7 catcache 6 catcache 7 catcache 6
> catcache 7 catcache 6 catcache 55 catcache 54 catcache 7 catcache 6 catcache 7
> catcache 6 catcache 32 catcache 55 catcache 54 catcache 55 catcache 54 catcache
> 55 catcache 54 catcache 63 catcache 63 catcache 63 catcache 63 catcache 63
> catcache 63 catcache 63 catcache 55 catcache 54 catcache 80 catcache 79 catcache
> 80 catcache 79 catcache 55 catcache 54 catcache 7 catcache 6 catcache 7 catcache
> 6 catcache 7 catcache 6 catcache 7 catcache 6 catcache 7 catcache 6 catcache 7
> catcache 6 catcache 7 catcache 6 catcache 7 catcache 6 catcache 7 catcache 6
> catcache 7 catcache 6 catcache 7 catcache 6 catcache 7 catcache 6 catcache 7
> catcache 6 catcache 7 catcache 6 catcache 7 catcache 6 catcache 7 catcache 6
> catcache 7 catcache 6 catcache 55 catcache 54 catcache 7 catcache 6 catcache 7
> catcache 6 catcache 7 catcache 6 catcache 7 catcache 6 catcache 7 catcache 6
> catcache 7 catcache 6 catcache 7 catcache 6 catcache 7 catcache 6 catcache 7
> catcache 6 catcache 55 catcache 54 catcache 7 catcache 6 catcache 7 catcache 6
> catcache 32 catcache 55 catcache 54 catcache 55 catcache 54 catcache 55 catcache
> 54 catcache 80 catcache 79 catcache 80 catcache 79 catcache 55 catcache 54
> catcache 7 catcache 6 catcache 7 catcache 6 catcache 7 catcache 6 catcache 7
> catcache 6 catcache 7 catcache 6 catcache 7 catcache 6 catcache 7 catcache 6
> catcache 7 catcache 6 catcache 7 catcache 6 catcache 7 catcache 6 catcache 7
> catcache 6 catcache 7 catcache 6 catcache 7 catcache 6 catcache 7 catcache 6
> catcache 7 catcache 6 catcache 7 catcache 6 catcache 7 catcache 6 catcache 55
> catcache 54 catcache 7 catcache 6 catcache 7 catcache 6 catcache 7 catcache 6
> catcache 7 catcache 6 catcache 7 catcache 6 catcache 7 catcache 6 catcache 7
> catcache 6 catcache 7 catcache 6 catcache 7 catcache 6 catcache 55 catcache 54
> catcache 7 catcache 6 catcache 7 catcache 6 catcache 32 catcache 55 catcache 54
> catcache 55 catcache 54 catcache 55 catcache 54 catcache 63 catcache 63 catcache
> 63 catcache 63 catcache 63 catcache 63 catcache 63 catcache 63 catcache 63
> catcache 63 catcache 63 catcache 55 catcache 54 catcache 32 catcache 7 catcache
> 6 catcache 7 catcache 6 catcache 55 catcache 54 catcache 7 catcache 6 catcache 7
> catcache 6 catcache 7 catcache 6 catcache 7 catcache 6 catcache 7 catcache 6
> catcache 7 catcache 6 catcache 7 catcache 6 catcache 7 catcache 6 catcache 7
> catcache 6 catcache 55 catcache 54 catcache 80 catcache 79 catcache 80 catcache
> 79 catcache 63 catcache 63 catcache 63 catcache 63 catcache 63 catcache 63
> catcache 63 catcache 63 catcache 63 catcache 63 catcache 63 catcache 7 catcache
> 6 catcache 7 catcache 6 catcache 7 catcache 6 catcache 7 catcache 6 catcache 7
> catcache 6 catcache 7 catcache 6 catcache 7 catcache 6 catcache 7 catcache 6
> catcache 7 catcache 6 catcache 7 catcache 6 catcache 7 catcache 6 catcache 7
> catcache 6 catcache 7 catcache 6 catcache 7 catcache 6 catcache 7 catcache 6
> catcache 7 catcache 6 catcache 7 catcache 6 catcache 55 catcache 54 catcache 32
> catcache 7 catcache 6 catcache 7 catcache 6 catcache 55 catcache 54 catcache 7
> catcache 6 catcache 7 catcache 6 catcache 7 catcache 6 catcache 7 catcache 6
> catcache 7 catcache 6 catcache 7 catcache 6 catcache 7 catcache 6 catcache 7
> catcache 6 catcache 7 catcache 6 catcache 55 catcache 54 catcache 80 catcache 79
> catcache 80 catcache 79 catcache 7 catcache 6 catcache 7 catcache 6 catcache 7
> catcache 6 catcache 7 catcache 6 catcache 7 catcache 6 catcache 7 catcache 6
> catcache 7 catcache 6 catcache 7 catcache 6 catcache 7 catcache 6 catcache 7
> catcache 6 catcache 7 catcache 6 catcache 7 catcache 6 catcache 7 catcache 6
> catcache 7 catcache 6 catcache 7 catcache 6 catcache 7 catcache 6 catcache 7
> catcache 6 catcache 55 catcache 54 catcache 32 catcache 7 catcache 6 catcache 7
> catcache 6 catcache 55 catcache 54 catcache 7 catcache 6 catcache 7 catcache 6
> catcache 7 catcache 6 catcache 7 catcache 6 catcache 7 catcache 6 catcache 7
> catcache 6 catcache 7 catcache 6 catcache 7 catcache 6 catcache 7 catcache 6
> catcache 55 catcache 54 catcache 80 catcache 79 catcache 80 catcache 79 catcache
> 63 catcache 63 catcache 63 catcache 63 catcache 63 catcache 63 catcache 63
> catcache 7 catcache 6 catcache 7 catcache 6 catcache 7 catcache 6 catcache 7
> catcache 6 catcache 7 catcache 6 catcache 7 catcache 6 catcache 7 catcache 6
> catcache 7 catcache 6 catcache 7 catcache 6 catcache 7 catcache 6 catcache 7
> catcache 6 catcache 7 catcache 6 catcache 7 catcache 6 catcache 55 catcache 54
> catcache 32 catcache 7 catcache 6 catcache 7 catcache 6 catcache 55 catcache 54
> catcache 7 catcache 6 catcache 7 catcache 6 catcache 7 catcache 6 catcache 7
> catcache 6 catcache 7 catcache 6 catcache 7 catcache 6 catcache 7 catcache 6
> catcache 7 catcache 6 catcache 7 catcache 6 catcache 55 catcache 54 catcache 80
> catcache 79 catcache 80 catcache 79 catcache 7 catcache 6 catcache 7 catcache 6
> catcache 7 catcache 6 catcache 7 catcache 6 catcache 7 catcache 6 catcache 7
> catcache 6 catcache 7 catcache 6 catcache 7 catcache 6 catcache 7 catcache 6
> catcache 7 catcache 6 catcache 7 catcache 6 catcache 7 catcache 6 catcache 7
> catcache 6 catcache 55 catcache 54 snapshot 2608 relcache 661557614 snapshot
> 1214 relcache 661557617 relcache 661557618 relcache 661557617 snapshot 2608
> relcache 661557617 relcache 661557618 relcache 661557614 snapshot 2608 snapshot
> 2608 relcache 661557669 snapshot 1214 relcache 661557672 relcache 661557673
> relcache 661557672 snapshot 2608 relcache 661557672 relcache 661557673 relcache
> 661557669 snapshot 2608 relcache 661557669 snapshot 2608 relcache 661557674
> snapshot 1214 relcache 661557677 relcache 661557678 relcache 661557677 snapshot
> 2608 relcache 661557677 relcache 661557678 relcache 661557674 snapshot 2608
> snapshot 2608 relcache 661557714 snapshot 1214 relcache 661557717 relcache
> 661557718 relcache 661557717 snapshot 2608 relcache 661557717 relcache 661557718
> relcache 661557714 snapshot 2608 relcache 661557714 relcache 661557718 relcache
> 661557717 snapshot 2608 relcache 661557717 snapshot 2608 snapshot 2608 snapshot
> 2608 relcache 661557714 snapshot 2608 snapshot 1214 relcache 661557678 relcache
> 661557677 snapshot 2608 relcache 661557677 snapshot 2608 snapshot 2608 snapshot
> 2608 relcache 661557674 snapshot 2608 snapshot 1214 relcache 661557673 relcache
> 661557672 snapshot 2608 relcache 661557672 snapshot 2608 snapshot 2608 snapshot
> 2608 relcache 661557669 snapshot 2608 snapshot 1214 relcache 661557618 relcache
> 661557617 snapshot 2608 relcache 661557617 snapshot 2608 snapshot 2608 snapshot
> 2608 relcache 661557614 snapshot 2608 snapshot 1214
>
> While it is long, it doesn't seem to merit allocating anything like 1GB of
> memory. So I'm guessing that postgres is miscalculating the required size somehow.
>
> If I skip over this LSN, for example by dropping the subscription and recreating
> it anew, then things go fine for a while before hitting another "invalid memory
> alloc request", i.e. it wasn't just a one-off. On the other hand, after
> downgrading to 17.4, subscribers spontaneously recovered and the issue has gone
> way. Since I didn't skip over the last LSN of this kind, presumably 17.4
> successfully serialized a message for the same problematic bit of WAL that
> caused 17.5 to blow up, which suggests a regression between 17.4 and 17.5.
>
Hi Duncan,
Thanks for reporting this.
I tried adding around ~80000 invalidations but could not reproduce the issue.
Can you share the steps to reproduce the above scenario?
Thanks and Regards,
Shlok Kyal
^ permalink raw reply [nested|flat] 94+ messages in thread
* Re: Logical replication 'invalid memory alloc request size 1585837200' after upgrading to 17.5
@ 2025-05-21 05:48 Amit Kapila <amit.kapila16@gmail.com>
parent: Duncan Sands <duncan.sands@deepbluecap.com>
2 siblings, 2 replies; 94+ messages in thread
From: Amit Kapila @ 2025-05-21 05:48 UTC (permalink / raw)
To: Duncan Sands <duncan.sands@deepbluecap.com>; +Cc: pgsql-bugs@lists.postgresql.org
On Mon, May 19, 2025 at 8:08 PM Duncan Sands
<duncan.sands@deepbluecap.com> wrote:
>
> PostgreSQL v17.5 (Ubuntu 17.5-1.pgdg24.04+1); Ubuntu 24.04.2 LTS (kernel
> 6.8.0); x86-64
>
> Good morning from DeepBlueCapital. Soon after upgrading to 17.5 from 17.4, we
> started seeing logical replication failures with publisher errors like this:
>
> ERROR: invalid memory alloc request size 1196493216
>
> (the exact size varies). Here is a typical log extract from the publisher:
>
> 2025-05-19 10:30:14 CEST \[1348336-465] remote\_production\_user\@blue DEBUG:
> 00000: write FB03/349DEF90 flush FB03/349DEF90 apply FB03/349DEF90 reply\_time
> 2025-05-19 10:30:07.467048+02
> 2025-05-19 10:30:14 CEST \[1348336-466] remote\_production\_user\@blue LOCATION:
> ProcessStandbyReplyMessage, walsender.c:2431
> 2025-05-19 10:30:14 CEST \[1348336-467] remote\_production\_user\@blue DEBUG:
> 00000: skipped replication of an empty transaction with XID: 207637565
> 2025-05-19 10:30:14 CEST \[1348336-468] remote\_production\_user\@blue CONTEXT:
> slot "jnb\_production", output plugin "pgoutput", in the commit callback,
> associated LSN FB03/349FF938
> 2025-05-19 10:30:14 CEST \[1348336-469] remote\_production\_user\@blue LOCATION:
> pgoutput\_commit\_txn, pgoutput.c:629
> 2025-05-19 10:30:14 CEST \[1348336-470] remote\_production\_user\@blue DEBUG:
> 00000: UpdateDecodingStats: updating stats 0x5ae1616c17a8 0 0 0 0 1 0 1 191
> 2025-05-19 10:30:14 CEST \[1348336-471] remote\_production\_user\@blue LOCATION:
> UpdateDecodingStats, logical.c:1943
> 2025-05-19 10:30:14 CEST \[1348336-472] remote\_production\_user\@blue DEBUG:
> 00000: found top level transaction 207637519, with catalog changes
> 2025-05-19 10:30:14 CEST \[1348336-473] remote\_production\_user\@blue LOCATION:
> SnapBuildCommitTxn, snapbuild.c:1150
> 2025-05-19 10:30:14 CEST \[1348336-474] remote\_production\_user\@blue DEBUG:
> 00000: adding a new snapshot and invalidations to 207616976 at FB03/34A1AAE0
> 2025-05-19 10:30:14 CEST \[1348336-475] remote\_production\_user\@blue LOCATION:
> SnapBuildDistributeSnapshotAndInval, snapbuild.c:915
> 2025-05-19 10:30:14 CEST \[1348336-476] remote\_production\_user\@blue ERROR:
> XX000: invalid memory alloc request size 1196493216
>
> If I'm reading it right, things go wrong on the publisher while preparing the
> message, i.e. it's not a subscriber problem.
>
Right, I also think so.
> This particular instance was triggered by a large number of catalog
> invalidations: I dumped what I think is the relevant WAL with "pg_waldump -s
> FB03/34A1AAE0 -p 17/main/ --xid=207637519" and the output was a single long line:
>
...
...
>
> While it is long, it doesn't seem to merit allocating anything like 1GB of
> memory. So I'm guessing that postgres is miscalculating the required size somehow.
>
We fixed a bug in commit 4909b38af0 to distribute invalidation at the
transaction end to avoid data loss in certain cases, which could cause
such a problem. I am wondering that even prior to that commit, we
would eventually end up allocating the required memory for a
transaction for all the invalidations because of repalloc in
ReorderBufferAddInvalidations, so why it matter with this commit? One
possibility is that we need allocations for multiple in-progress
transactions now. I'll think more about this. It would be helpful if
you could share more details about the workload, or if possible, a
testcase or script using which we can reproduce this problem.
--
With Regards,
Amit Kapila.
^ permalink raw reply [nested|flat] 94+ messages in thread
* Re: Logical replication 'invalid memory alloc request size 1585837200' after upgrading to 17.5
@ 2025-05-21 11:12 Amit Kapila <amit.kapila16@gmail.com>
parent: Amit Kapila <amit.kapila16@gmail.com>
1 sibling, 2 replies; 94+ messages in thread
From: Amit Kapila @ 2025-05-21 11:12 UTC (permalink / raw)
To: Duncan Sands <duncan.sands@deepbluecap.com>; +Cc: pgsql-bugs@lists.postgresql.org, Sawada Masahiko <sawada.mshk@gmail.com>
On Wed, May 21, 2025 at 11:18 AM Amit Kapila <amit.kapila16@gmail.com> wrote:
>
> On Mon, May 19, 2025 at 8:08 PM Duncan Sands
> <duncan.sands@deepbluecap.com> wrote:
> >
> > While it is long, it doesn't seem to merit allocating anything like 1GB of
> > memory. So I'm guessing that postgres is miscalculating the required size somehow.
> >
>
> We fixed a bug in commit 4909b38af0 to distribute invalidation at the
> transaction end to avoid data loss in certain cases, which could cause
> such a problem. I am wondering that even prior to that commit, we
> would eventually end up allocating the required memory for a
> transaction for all the invalidations because of repalloc in
> ReorderBufferAddInvalidations, so why it matter with this commit? One
> possibility is that we need allocations for multiple in-progress
> transactions now.
>
I think the problem here is that when we are distributing
invalidations to a concurrent transaction, in addition to queuing the
invalidations as a change, we also copy the distributed invalidations
along with the original transaction's invalidations via repalloc in
ReorderBufferAddInvalidations. So, when there are many in-progress
transactions, each would try to copy all its accumulated invalidations
to the remaining in-progress transactions. This could lead to such an
increase in allocation request size. However, after queuing the
change, we don't need to copy it along with the original transaction's
invalidations. This is because the copy is only required when we don't
process any changes in cases like ReorderBufferForget(). I have
analyzed all such cases, and my analysis is as follows:
ReorderBufferForget()
------------------------------
It is okay not to perform the invalidations that we got from other
concurrent transactions during ReorderBufferForget. This is because
ReorderBufferForget executes invalidations when we skip the
transaction being decoded, as it is not from a database of interest.
So, we execute only to invalidate shared catalogs (See comment at the
caller of ReorderBufferForget). It is sufficient to execute such
invalidations in the source transaction only because the transaction
being skipped wouldn't have loaded anything in the shared catalog.
ReorderBufferAbort()
-----------------------------
ReorderBufferAbort() process invalidation when it has already streamed
some changes. Whenever it would have streamed the change, it would
have processed the concurrent transactions' invalidation messages that
happened before the statement that led to streaming. That should be
sufficient for us.
Consider the following variant of the original case that required the
distribution of invalidations:
1) S1: CREATE TABLE d(data text not null);
2) S1: INSERT INTO d VALUES('d1');
3) S2: BEGIN; INSERT INTO d VALUES('d2');
4) S1: ALTER PUBLICATION pb ADD TABLE d;
5) S2: INSERT INTO unrelated_tab VALUES(1);
6) S2: ROLLBACK;
7) S2: INSERT INTO d VALUES('d3');
8) S1: INSERT INTO d VALUES('d4');
The problem with the sequence is that the insert from 3) could be
decoded *after* 4) in step 5) due to streaming, and that to decode the
insert (which happened before the ALTER) the catalog snapshot and
cache state is from *before* the ALTER TABLE. Because the transaction
started in 3) doesn't modify any catalogs, no invalidations are
executed after decoding it. The result could be that the cache looks
like it did at 3), not like after 4). However, this won't create a
problem because while streaming at 5), we would execute invalidation
from S-1 due to the change added via message
REORDER_BUFFER_CHANGE_INVALIDATION in ReorderBufferAddInvalidations.
ReorderBufferInvalidate
--------------------------------
The reason is the same as ReorderBufferForget(), as it executes
invalidations for the same reason, but with a different function to
avoid the cleanup of the buffer at the end.
XLOG_XACT_INVALIDATIONS
-------------------------------------------
While processing XLOG_XACT_INVALIDATIONS, we don't need invalidations
accumulated from other xacts because this is a special case to execute
invalidations from a particular command (DDL) in a transaction. It
won't build any cache, so it can't create any invalid state.
--
With Regards,
Amit Kapila.
^ permalink raw reply [nested|flat] 94+ messages in thread
* Re: Logical replication 'invalid memory alloc request size 1585837200' after upgrading to 17.5
@ 2025-05-21 11:30 Duncan Sands <duncan.sands@deepbluecap.com>
parent: Amit Kapila <amit.kapila16@gmail.com>
1 sibling, 0 replies; 94+ messages in thread
From: Duncan Sands @ 2025-05-21 11:30 UTC (permalink / raw)
To: Amit Kapila <amit.kapila16@gmail.com>; +Cc: pgsql-bugs@lists.postgresql.org, Shlok Kyal <shlok.kyal.oss@gmail.com>
Hi Amit and Shlok, thanks for thinking about this issue. We are working on
reproducing it in our test environment. Since it seems likely to be related to
our primary database being very busy with lots of concurrency and large
transactions, we are starting by creating a streaming replication copy of our
primary server (this copy to run 17.5, with the primary on 17.4), with the idea
of then doing logical replication from the standby to see if we hit the same
issue. If so, that gives something to poke at, and we can move on to something
better from there.
Best wishes, Duncan.
On 21/05/2025 07:48, Amit Kapila wrote:
> On Mon, May 19, 2025 at 8:08 PM Duncan Sands
> <duncan.sands@deepbluecap.com> wrote:
>>
>> PostgreSQL v17.5 (Ubuntu 17.5-1.pgdg24.04+1); Ubuntu 24.04.2 LTS (kernel
>> 6.8.0); x86-64
>>
>> Good morning from DeepBlueCapital. Soon after upgrading to 17.5 from 17.4, we
>> started seeing logical replication failures with publisher errors like this:
>>
>> ERROR: invalid memory alloc request size 1196493216
>>
>> (the exact size varies). Here is a typical log extract from the publisher:
>>
>> 2025-05-19 10:30:14 CEST \[1348336-465] remote\_production\_user\@blue DEBUG:
>> 00000: write FB03/349DEF90 flush FB03/349DEF90 apply FB03/349DEF90 reply\_time
>> 2025-05-19 10:30:07.467048+02
>> 2025-05-19 10:30:14 CEST \[1348336-466] remote\_production\_user\@blue LOCATION:
>> ProcessStandbyReplyMessage, walsender.c:2431
>> 2025-05-19 10:30:14 CEST \[1348336-467] remote\_production\_user\@blue DEBUG:
>> 00000: skipped replication of an empty transaction with XID: 207637565
>> 2025-05-19 10:30:14 CEST \[1348336-468] remote\_production\_user\@blue CONTEXT:
>> slot "jnb\_production", output plugin "pgoutput", in the commit callback,
>> associated LSN FB03/349FF938
>> 2025-05-19 10:30:14 CEST \[1348336-469] remote\_production\_user\@blue LOCATION:
>> pgoutput\_commit\_txn, pgoutput.c:629
>> 2025-05-19 10:30:14 CEST \[1348336-470] remote\_production\_user\@blue DEBUG:
>> 00000: UpdateDecodingStats: updating stats 0x5ae1616c17a8 0 0 0 0 1 0 1 191
>> 2025-05-19 10:30:14 CEST \[1348336-471] remote\_production\_user\@blue LOCATION:
>> UpdateDecodingStats, logical.c:1943
>> 2025-05-19 10:30:14 CEST \[1348336-472] remote\_production\_user\@blue DEBUG:
>> 00000: found top level transaction 207637519, with catalog changes
>> 2025-05-19 10:30:14 CEST \[1348336-473] remote\_production\_user\@blue LOCATION:
>> SnapBuildCommitTxn, snapbuild.c:1150
>> 2025-05-19 10:30:14 CEST \[1348336-474] remote\_production\_user\@blue DEBUG:
>> 00000: adding a new snapshot and invalidations to 207616976 at FB03/34A1AAE0
>> 2025-05-19 10:30:14 CEST \[1348336-475] remote\_production\_user\@blue LOCATION:
>> SnapBuildDistributeSnapshotAndInval, snapbuild.c:915
>> 2025-05-19 10:30:14 CEST \[1348336-476] remote\_production\_user\@blue ERROR:
>> XX000: invalid memory alloc request size 1196493216
>>
>> If I'm reading it right, things go wrong on the publisher while preparing the
>> message, i.e. it's not a subscriber problem.
>>
>
> Right, I also think so.
>
>> This particular instance was triggered by a large number of catalog
>> invalidations: I dumped what I think is the relevant WAL with "pg_waldump -s
>> FB03/34A1AAE0 -p 17/main/ --xid=207637519" and the output was a single long line:
>>
> ...
> ...
>>
>> While it is long, it doesn't seem to merit allocating anything like 1GB of
>> memory. So I'm guessing that postgres is miscalculating the required size somehow.
>>
>
> We fixed a bug in commit 4909b38af0 to distribute invalidation at the
> transaction end to avoid data loss in certain cases, which could cause
> such a problem. I am wondering that even prior to that commit, we
> would eventually end up allocating the required memory for a
> transaction for all the invalidations because of repalloc in
> ReorderBufferAddInvalidations, so why it matter with this commit? One
> possibility is that we need allocations for multiple in-progress
> transactions now. I'll think more about this. It would be helpful if
> you could share more details about the workload, or if possible, a
> testcase or script using which we can reproduce this problem.
>
^ permalink raw reply [nested|flat] 94+ messages in thread
* RE: Logical replication 'invalid memory alloc request size 1585837200' after upgrading to 17.5
@ 2025-05-21 11:48 Hayato Kuroda (Fujitsu) <kuroda.hayato@fujitsu.com>
parent: Amit Kapila <amit.kapila16@gmail.com>
1 sibling, 3 replies; 94+ messages in thread
From: Hayato Kuroda (Fujitsu) @ 2025-05-21 11:48 UTC (permalink / raw)
To: Amit Kapila <amit.kapila16@gmail.com>; Duncan Sands <duncan.sands@deepbluecap.com>; +Cc: pgsql-bugs@lists.postgresql.org <pgsql-bugs@lists.postgresql.org>; Sawada Masahiko <sawada.mshk@gmail.com>
Dear hackers,
> I think the problem here is that when we are distributing
> invalidations to a concurrent transaction, in addition to queuing the
> invalidations as a change, we also copy the distributed invalidations
> along with the original transaction's invalidations via repalloc in
> ReorderBufferAddInvalidations. So, when there are many in-progress
> transactions, each would try to copy all its accumulated invalidations
> to the remaining in-progress transactions. This could lead to such an
> increase in allocation request size. However, after queuing the
> change, we don't need to copy it along with the original transaction's
> invalidations. This is because the copy is only required when we don't
> process any changes in cases like ReorderBufferForget(). I have
> analyzed all such cases, and my analysis is as follows:
Based on the analysis, I created a PoC which avoids the repalloc().
Invalidation messages distributed by SnapBuildDistributeSnapshotAndInval() are
skipped to add in the list, just queued - repalloc can be skipped. Also, the function
distributes messages only in the list, so received messages won't be sent again.
Now a patch for PG17 is created for testing purpose. Duncan, can you apply this and
confirms whether the issue can be solved?
Best regards,
Hayato Kuroda
FUJITSU LIMITED
Attachments:
[application/octet-stream] PG17-0001-Avoid-distributing-invalidation-messages-several-tim.patch (5.0K, ../../OSCPR01MB14966139B6C89C27647712A90F59EA@OSCPR01MB14966.jpnprd01.prod.outlook.com/2-PG17-0001-Avoid-distributing-invalidation-messages-several-tim.patch)
download | inline diff:
From 259a1f79511bb5acb03167e44eadb2d3fd5ab644 Mon Sep 17 00:00:00 2001
From: Hayato Kuroda <kuroda.hayato@fujitsu.com>
Date: Wed, 21 May 2025 18:02:36 +0900
Subject: [PATCH] Avoid distributing invalidation messages several times
---
.../replication/logical/reorderbuffer.c | 71 ++++++++++++-------
src/backend/replication/logical/snapbuild.c | 5 +-
src/include/replication/reorderbuffer.h | 3 +
3 files changed, 51 insertions(+), 28 deletions(-)
diff --git a/src/backend/replication/logical/reorderbuffer.c b/src/backend/replication/logical/reorderbuffer.c
index 03eb005c39d..868fcd8a344 100644
--- a/src/backend/replication/logical/reorderbuffer.c
+++ b/src/backend/replication/logical/reorderbuffer.c
@@ -3330,41 +3330,60 @@ ReorderBufferAddInvalidations(ReorderBuffer *rb, TransactionId xid,
XLogRecPtr lsn, Size nmsgs,
SharedInvalidationMessage *msgs)
{
- ReorderBufferTXN *txn;
+ ReorderBufferAddInvalidationsExtended(rb, xid, lsn, nmsgs, msgs, true);
+}
+
+/*
+ * An extended workhorse version of ReorderBufferAddInvalidations(). This can
+ * control whether invalidation messages can be accumulated in reorderbuffer
+ * transactions, which would be distributed to all concurrent transactions at
+ * commit.
+ */
+void
+ReorderBufferAddInvalidationsExtended(ReorderBuffer *rb, TransactionId xid,
+ XLogRecPtr lsn, Size nmsgs,
+ SharedInvalidationMessage *msgs,
+ bool needs_distribute)
+{
MemoryContext oldcontext;
ReorderBufferChange *change;
- txn = ReorderBufferTXNByXid(rb, xid, true, NULL, lsn, true);
-
oldcontext = MemoryContextSwitchTo(rb->context);
- /*
- * Collect all the invalidations under the top transaction, if available,
- * so that we can execute them all together. See comments atop this
- * function.
- */
- txn = rbtxn_get_toptxn(txn);
-
Assert(nmsgs > 0);
- /* Accumulate invalidations. */
- if (txn->ninvalidations == 0)
- {
- txn->ninvalidations = nmsgs;
- txn->invalidations = (SharedInvalidationMessage *)
- palloc(sizeof(SharedInvalidationMessage) * nmsgs);
- memcpy(txn->invalidations, msgs,
- sizeof(SharedInvalidationMessage) * nmsgs);
- }
- else
+ if (needs_distribute)
{
- txn->invalidations = (SharedInvalidationMessage *)
- repalloc(txn->invalidations, sizeof(SharedInvalidationMessage) *
- (txn->ninvalidations + nmsgs));
+ ReorderBufferTXN *txn;
+
+ txn = ReorderBufferTXNByXid(rb, xid, true, NULL, lsn, true);
+
+ /*
+ * Collect all the invalidations under the top transaction, if
+ * available, so that we can execute them all together. See comments
+ * atop ReorderBufferAddInvalidations.
+ */
+ txn = rbtxn_get_toptxn(txn);
- memcpy(txn->invalidations + txn->ninvalidations, msgs,
- nmsgs * sizeof(SharedInvalidationMessage));
- txn->ninvalidations += nmsgs;
+ /* Accumulate invalidations. */
+ if (txn->ninvalidations == 0)
+ {
+ txn->ninvalidations = nmsgs;
+ txn->invalidations = (SharedInvalidationMessage *)
+ palloc(sizeof(SharedInvalidationMessage) * nmsgs);
+ memcpy(txn->invalidations, msgs,
+ sizeof(SharedInvalidationMessage) * nmsgs);
+ }
+ else
+ {
+ txn->invalidations = (SharedInvalidationMessage *)
+ repalloc(txn->invalidations,
+ sizeof(SharedInvalidationMessage) * (txn->ninvalidations + nmsgs));
+
+ memcpy(txn->invalidations + txn->ninvalidations, msgs,
+ nmsgs * sizeof(SharedInvalidationMessage));
+ txn->ninvalidations += nmsgs;
+ }
}
change = ReorderBufferGetChange(rb);
diff --git a/src/backend/replication/logical/snapbuild.c b/src/backend/replication/logical/snapbuild.c
index 110e0b0a044..a8225c1d905 100644
--- a/src/backend/replication/logical/snapbuild.c
+++ b/src/backend/replication/logical/snapbuild.c
@@ -945,8 +945,9 @@ SnapBuildDistributeSnapshotAndInval(SnapBuild *builder, XLogRecPtr lsn, Transact
{
Assert(msgs != NULL);
- ReorderBufferAddInvalidations(builder->reorder, txn->xid, lsn,
- ninvalidations, msgs);
+ ReorderBufferAddInvalidationsExtended(builder->reorder, txn->xid, lsn,
+ ninvalidations, msgs,
+ false);
}
}
}
diff --git a/src/include/replication/reorderbuffer.h b/src/include/replication/reorderbuffer.h
index 4c56f219fd8..001d0e716aa 100644
--- a/src/include/replication/reorderbuffer.h
+++ b/src/include/replication/reorderbuffer.h
@@ -709,6 +709,9 @@ extern void ReorderBufferAddNewTupleCids(ReorderBuffer *rb, TransactionId xid,
CommandId cmin, CommandId cmax, CommandId combocid);
extern void ReorderBufferAddInvalidations(ReorderBuffer *rb, TransactionId xid, XLogRecPtr lsn,
Size nmsgs, SharedInvalidationMessage *msgs);
+extern void ReorderBufferAddInvalidationsExtended(ReorderBuffer *rb, TransactionId xid, XLogRecPtr lsn,
+ Size nmsgs, SharedInvalidationMessage *msgs,
+ bool needs_distribute);
extern void ReorderBufferImmediateInvalidation(ReorderBuffer *rb, uint32 ninvalidations,
SharedInvalidationMessage *invalidations);
extern void ReorderBufferProcessXid(ReorderBuffer *rb, TransactionId xid, XLogRecPtr lsn);
--
2.43.5
^ permalink raw reply [nested|flat] 94+ messages in thread
* Re: Logical replication 'invalid memory alloc request size 1585837200' after upgrading to 17.5
@ 2025-05-21 13:27 Duncan Sands <duncan.sands@deepbluecap.com>
parent: Hayato Kuroda (Fujitsu) <kuroda.hayato@fujitsu.com>
2 siblings, 0 replies; 94+ messages in thread
From: Duncan Sands @ 2025-05-21 13:27 UTC (permalink / raw)
To: Hayato Kuroda (Fujitsu) <kuroda.hayato@fujitsu.com>; Amit Kapila <amit.kapila16@gmail.com>; +Cc: pgsql-bugs@lists.postgresql.org <pgsql-bugs@lists.postgresql.org>; Sawada Masahiko <sawada.mshk@gmail.com>
> Based on the analysis, I created a PoC which avoids the repalloc().
> Invalidation messages distributed by SnapBuildDistributeSnapshotAndInval() are
> skipped to add in the list, just queued - repalloc can be skipped. Also, the function
> distributes messages only in the list, so received messages won't be sent again.
>
> Now a patch for PG17 is created for testing purpose. Duncan, can you apply this and
> confirms whether the issue can be solved?
Thanks Hayato Kuroda, will do, however it may take a few days.
Best wishes, Duncan.
^ permalink raw reply [nested|flat] 94+ messages in thread
* Re: Logical replication 'invalid memory alloc request size 1585837200' after upgrading to 17.5
@ 2025-05-21 18:24 Masahiko Sawada <sawada.mshk@gmail.com>
parent: Amit Kapila <amit.kapila16@gmail.com>
1 sibling, 1 reply; 94+ messages in thread
From: Masahiko Sawada @ 2025-05-21 18:24 UTC (permalink / raw)
To: Amit Kapila <amit.kapila16@gmail.com>; +Cc: Duncan Sands <duncan.sands@deepbluecap.com>; pgsql-bugs@lists.postgresql.org
On Wed, May 21, 2025 at 4:12 AM Amit Kapila <amit.kapila16@gmail.com> wrote:
>
> On Wed, May 21, 2025 at 11:18 AM Amit Kapila <amit.kapila16@gmail.com> wrote:
> >
> > On Mon, May 19, 2025 at 8:08 PM Duncan Sands
> > <duncan.sands@deepbluecap.com> wrote:
> > >
> > > While it is long, it doesn't seem to merit allocating anything like 1GB of
> > > memory. So I'm guessing that postgres is miscalculating the required size somehow.
> > >
> >
> > We fixed a bug in commit 4909b38af0 to distribute invalidation at the
> > transaction end to avoid data loss in certain cases, which could cause
> > such a problem. I am wondering that even prior to that commit, we
> > would eventually end up allocating the required memory for a
> > transaction for all the invalidations because of repalloc in
> > ReorderBufferAddInvalidations, so why it matter with this commit? One
> > possibility is that we need allocations for multiple in-progress
> > transactions now.
> >
>
> I think the problem here is that when we are distributing
> invalidations to a concurrent transaction, in addition to queuing the
> invalidations as a change, we also copy the distributed invalidations
> along with the original transaction's invalidations via repalloc in
> ReorderBufferAddInvalidations. So, when there are many in-progress
> transactions, each would try to copy all its accumulated invalidations
> to the remaining in-progress transactions. This could lead to such an
> increase in allocation request size.
I agree with this analysis.
> However, after queuing the
> change, we don't need to copy it along with the original transaction's
> invalidations. This is because the copy is only required when we don't
> process any changes in cases like ReorderBufferForget().
It seems that we use the accumulated invalidation message also after
replaying or concurrently aborting a transaction via
ReorderBufferExecuteInvalidations(). Do we need to consider such cases
too?
Regards,
--
Masahiko Sawada
Amazon Web Services: https://aws.amazon.com
^ permalink raw reply [nested|flat] 94+ messages in thread
* Re: Logical replication 'invalid memory alloc request size 1585837200' after upgrading to 17.5
@ 2025-05-22 10:56 Amit Kapila <amit.kapila16@gmail.com>
parent: Masahiko Sawada <sawada.mshk@gmail.com>
0 siblings, 2 replies; 94+ messages in thread
From: Amit Kapila @ 2025-05-22 10:56 UTC (permalink / raw)
To: Masahiko Sawada <sawada.mshk@gmail.com>; +Cc: Duncan Sands <duncan.sands@deepbluecap.com>; pgsql-bugs@lists.postgresql.org
On Wed, May 21, 2025 at 11:54 PM Masahiko Sawada <sawada.mshk@gmail.com> wrote:
>
> On Wed, May 21, 2025 at 4:12 AM Amit Kapila <amit.kapila16@gmail.com> wrote:
> >
>
> > However, after queuing the
> > change, we don't need to copy it along with the original transaction's
> > invalidations. This is because the copy is only required when we don't
> > process any changes in cases like ReorderBufferForget().
>
> It seems that we use the accumulated invalidation message also after
> replaying or concurrently aborting a transaction via
> ReorderBufferExecuteInvalidations(). Do we need to consider such cases
> too?
>
Good point. After replaying the transaction, it doesn't matter because
we would have already relayed the required invalidation while
processing REORDER_BUFFER_CHANGE_INVALIDATION messages. However for
concurrent abort case it could matter. See my analysis for the same
below:
Simulation of concurrent abort
------------------------------------------
1) S1: CREATE TABLE d(data text not null);
2) S1: INSERT INTO d VALUES('d1');
3) S2: BEGIN; INSERT INTO d VALUES('d2');
4) S2: INSERT INTO unrelated_tab VALUES(1);
5) S1: ALTER PUBLICATION pb ADD TABLE d;
6) S2: INSERT INTO unrelated_tab VALUES(2);
7) S2: ROLLBACK;
8) S2: INSERT INTO d VALUES('d3');
9) S1: INSERT INTO d VALUES('d4');
The problem with the sequence is that the insert from 3) could be
decoded *after* 5) in step 6) due to streaming and that to decode the
insert (which happened before the ALTER) the catalog snapshot and
cache state is from *before* the ALTER TABLE. Because the transaction
started in 3) doesn't actually modify any catalogs, no invalidations
are executed after decoding it. Now, assume, while decoding Insert
from 4), we detected a concurrent abort, then the distributed
invalidation won't be executed, and if we don't have accumulated
messages in txn->invalidations, then the invalidation from step 5)
won't be performed. The data loss can occur in steps 8 and 9. This is
just a theory, so I could be missing something.
If the above turns out to be a problem, one idea for fixing it is that
for the concurrent abort case (both during streaming and for prepared
transaction's processing), we still check all the remaining changes
and process only the changes related to invalidations. This has to be
done before the current txn changes are freed via
ReorderBufferResetTXN->ReorderBufferTruncateTXN.
Thoughts?
--
With Regards,
Amit Kapila.
^ permalink raw reply [nested|flat] 94+ messages in thread
* Re: Logical replication 'invalid memory alloc request size 1585837200' after upgrading to 17.5
@ 2025-05-22 12:23 Shlok Kyal <shlok.kyal.oss@gmail.com>
parent: Hayato Kuroda (Fujitsu) <kuroda.hayato@fujitsu.com>
2 siblings, 0 replies; 94+ messages in thread
From: Shlok Kyal @ 2025-05-22 12:23 UTC (permalink / raw)
To: Hayato Kuroda (Fujitsu) <kuroda.hayato@fujitsu.com>; +Cc: Amit Kapila <amit.kapila16@gmail.com>; Duncan Sands <duncan.sands@deepbluecap.com>; pgsql-bugs@lists.postgresql.org <pgsql-bugs@lists.postgresql.org>; Sawada Masahiko <sawada.mshk@gmail.com>
On Wed, 21 May 2025 at 17:18, Hayato Kuroda (Fujitsu)
<kuroda.hayato@fujitsu.com> wrote:
>
> Dear hackers,
>
> > I think the problem here is that when we are distributing
> > invalidations to a concurrent transaction, in addition to queuing the
> > invalidations as a change, we also copy the distributed invalidations
> > along with the original transaction's invalidations via repalloc in
> > ReorderBufferAddInvalidations. So, when there are many in-progress
> > transactions, each would try to copy all its accumulated invalidations
> > to the remaining in-progress transactions. This could lead to such an
> > increase in allocation request size. However, after queuing the
> > change, we don't need to copy it along with the original transaction's
> > invalidations. This is because the copy is only required when we don't
> > process any changes in cases like ReorderBufferForget(). I have
> > analyzed all such cases, and my analysis is as follows:
>
> Based on the analysis, I created a PoC which avoids the repalloc().
> Invalidation messages distributed by SnapBuildDistributeSnapshotAndInval() are
> skipped to add in the list, just queued - repalloc can be skipped. Also, the function
> distributes messages only in the list, so received messages won't be sent again.
>
> Now a patch for PG17 is created for testing purpose. Duncan, can you apply this and
> confirms whether the issue can be solved?
>
Hi,
I was able to reproduce the issue with following test:
1. First begin 9 concurrent txn. (BEGIN; INSERT into t1 values(11);)
2. In 10th concurrent txn : perform 1000 DDL (ALTER PUBLICATION ADD/DROP TABLE)
3. For each concurrent 9 txn. Perform:
i. Add 1000 DDL
ii. COMMIT;
iii. BEGIN; INSERT into t1 values(11);
4. Perform step (2 and 3) in loop
This steps reproduced the error:
2025-05-22 19:03:35.111 JST [63150] sub1 ERROR: invalid memory alloc
request size 1555752832
2025-05-22 19:03:35.111 JST [63150] sub1 STATEMENT: START_REPLICATION
SLOT "sub1" LOGICAL 0/0 (proto_version '4', streaming 'parallel',
origin 'any', publication_names '"pub1"')
I have also attached the test script for the same.
Also, I tried to run the test with Kuroda-san's patch and it did not
reproduce the issue.
Thanks and Regards,
Shlok Kyal
Attachments:
[application/octet-stream] 036_test.pl (6.1K, ../../CANhcyEW8UyMr_7idB580DT3bjtB=EKiHwecTx5KC3ggiVs9c+A@mail.gmail.com/2-036_test.pl)
download
^ permalink raw reply [nested|flat] 94+ messages in thread
* RE: Logical replication 'invalid memory alloc request size 1585837200' after upgrading to 17.5
@ 2025-05-22 12:59 Hayato Kuroda (Fujitsu) <kuroda.hayato@fujitsu.com>
parent: Amit Kapila <amit.kapila16@gmail.com>
1 sibling, 1 reply; 94+ messages in thread
From: Hayato Kuroda (Fujitsu) @ 2025-05-22 12:59 UTC (permalink / raw)
To: Amit Kapila <amit.kapila16@gmail.com>; Masahiko Sawada <sawada.mshk@gmail.com>; +Cc: Duncan Sands <duncan.sands@deepbluecap.com>; pgsql-bugs@lists.postgresql.org <pgsql-bugs@lists.postgresql.org>
Dear Amit, Sawada-san,
> Good point. After replaying the transaction, it doesn't matter because
> we would have already relayed the required invalidation while
> processing REORDER_BUFFER_CHANGE_INVALIDATION messages. However
> for
> concurrent abort case it could matter. See my analysis for the same
> below:
>
> Simulation of concurrent abort
> ------------------------------------------
> 1) S1: CREATE TABLE d(data text not null);
> 2) S1: INSERT INTO d VALUES('d1');
> 3) S2: BEGIN; INSERT INTO d VALUES('d2');
> 4) S2: INSERT INTO unrelated_tab VALUES(1);
> 5) S1: ALTER PUBLICATION pb ADD TABLE d;
> 6) S2: INSERT INTO unrelated_tab VALUES(2);
> 7) S2: ROLLBACK;
> 8) S2: INSERT INTO d VALUES('d3');
> 9) S1: INSERT INTO d VALUES('d4');
> The problem with the sequence is that the insert from 3) could be
> decoded *after* 5) in step 6) due to streaming and that to decode the
> insert (which happened before the ALTER) the catalog snapshot and
> cache state is from *before* the ALTER TABLE. Because the transaction
> started in 3) doesn't actually modify any catalogs, no invalidations
> are executed after decoding it. Now, assume, while decoding Insert
> from 4), we detected a concurrent abort, then the distributed
> invalidation won't be executed, and if we don't have accumulated
> messages in txn->invalidations, then the invalidation from step 5)
> won't be performed. The data loss can occur in steps 8 and 9. This is
> just a theory, so I could be missing something.
I verified this is real or not, and succeeded to reproduce. See appendix the
detailed steps.
> If the above turns out to be a problem, one idea for fixing it is that
> for the concurrent abort case (both during streaming and for prepared
> transaction's processing), we still check all the remaining changes
> and process only the changes related to invalidations. This has to be
> done before the current txn changes are freed via
> ReorderBufferResetTXN->ReorderBufferTruncateTXN.
I roughly implemented the part, PSA the updated version. One concern is whether we
should consider the case that invalidations can cause ereport(ERROR). If happens,
the walsender will exit at that time.
Appendix - reproducer
==============
Only an instance was used in the test. Defined objects were:
```
CREATE TABLE d(data text not null);
CREATE TABLE unrelated_tab(data text not null);
CREATE PUBLICATION pb;
```
Then, pg_recvlogical was used to allow replicating changes. Actual command:
```
$ pg_recvlogical --plugin=pgoutput --create-slot --start --slot test -U postgres
-d postgres -o proto_version=4 -o publication_names=pb -o messages=true
-o streaming=true -f -
```
Below are the actual steps. Gdb debugger was used to synchronize tests.
0. Prepare two sessions S1, and S2, and one replication connection
1. Ran "INSERT INTO d VALUES('d1');" on S1
2. Ran "BEGIN; INSERT INTO d VALUES('d2');"
3. Ran "INSERT INTO unrelated_tab VALUES('d2');"
4. Ran "ALTER PUBLICATION pb ADD TABLE d;"
5. Attached the walsender process via gdb
6. set a breakpoint at HandleConcurrentAbort
7. Ran INSERT INTO unrelated_tab VALUES(generate_series(1, 5000));
This allows to stream changes in S2.
8. Confrimed that gdb stopped the walsender process.
9. Ran continue comamnd in gdb several times, to ensure the process
accesses to "unrelated_tab". On my env, backtrace at that time was [1].
10. Ran "ROLLBACK" in S2
11. On gdb session, moved forward the program and ensured that the concurrent_abort
error was raised.
12. gdb detached from the walsender
13. Ran "INSERT INTO d VALUES('d3');" on S2
14. Ran "INSERT INTO d VALUES('d4');" on S1.
15. Checked the output from pg_recvlogical, and confirmed d3 and d4 were not output [2]
[1]
```
Breakpoint 1, HandleConcurrentAbort () at ../postgres/src/backend/access/index/genam.c:484
484 if (TransactionIdIsValid(CheckXidAlive) &&
(gdb) bt
#0 HandleConcurrentAbort () at ../postgres/src/backend/access/index/genam.c:484
#1 0x000000000052628a in systable_getnext (sysscan=0x31bcbd0)
at ../postgres/src/backend/access/index/genam.c:545
#2 0x0000000000b37afa in SearchCatCacheMiss (cache=0x3107180, nkeys=1, hashValue=2617776010,
hashIndex=10, v1=16389, v2=0, v3=0, v4=0)
at ../postgres/src/backend/utils/cache/catcache.c:1544
#3 0x0000000000b379a3 in SearchCatCacheInternal (cache=0x3107180, nkeys=1, v1=16389, v2=0, v3=0,
v4=0) at ../postgres/src/backend/utils/cache/catcache.c:1464
#4 0x0000000000b3769a in SearchCatCache1 (cache=0x3107180, v1=16389)
at ../postgres/src/backend/utils/cache/catcache.c:1332
#5 0x0000000000b544d5 in SearchSysCache1 (cacheId=55, key1=16389)
at ../postgres/src/backend/utils/cache/syscache.c:228
#6 0x0000000000b3e62a in get_rel_namespace (relid=16389)
at ../postgres/src/backend/utils/cache/lsyscache.c:1956
#7 0x00007fa3fdb1e0ec in get_rel_sync_entry (data=0x3160108, relation=0x7fa3fd06f398)
at ../postgres/src/backend/replication/pgoutput/pgoutput.c:2037
#8 0x00007fa3fdb1d126 in pgoutput_change (ctx=0x315fd90, txn=0x31b8aa0, relation=0x7fa3fd06f398,
change=0x31bab18) at ../postgres/src/backend/replication/pgoutput/pgoutput.c:1455
--Type <RET> for more, q to quit, c to continue without paging--q
Quit
(gdb) f 8
#8 0x00007fa3fdb1d126 in pgoutput_change (ctx=0x315fd90, txn=0x31b8aa0, relation=0x7fa3fd06f398,
change=0x31bab18) at ../postgres/src/backend/replication/pgoutput/pgoutput.c:1455
1455 relentry = get_rel_sync_entry(data, relation);
(gdb) p relation->rd_rel.relname
$2 = {data = "unrelated_tab", '\000' <repeats 50 times>}
```
[2]:
```
$ pg_recvlogical --plugin=pgoutput --create-slot --start --slot test -U postgres
-d postgres -o proto_version=4 -o publication_names=pb -o messages=true
-o streaming=true -f -
S
E
A
```
Best regards,
Hayato Kuroda
FUJITSU LIMITED
Attachments:
[application/octet-stream] v2-PG17-0001-Avoid-distributing-invalidation-messages-sev.patch (7.1K, ../../OSCPR01MB149669E1CAFE63051244F8E35F599A@OSCPR01MB14966.jpnprd01.prod.outlook.com/2-v2-PG17-0001-Avoid-distributing-invalidation-messages-sev.patch)
download | inline diff:
From 277a3ae7a8ed9f08cace293d9b6a06377e3e6ed1 Mon Sep 17 00:00:00 2001
From: Hayato Kuroda <kuroda.hayato@fujitsu.com>
Date: Wed, 21 May 2025 18:02:36 +0900
Subject: [PATCH v2-PG17] Avoid distributing invalidation messages several
times
---
.../replication/logical/reorderbuffer.c | 120 ++++++++++++++----
src/backend/replication/logical/snapbuild.c | 5 +-
src/include/replication/reorderbuffer.h | 3 +
3 files changed, 100 insertions(+), 28 deletions(-)
diff --git a/src/backend/replication/logical/reorderbuffer.c b/src/backend/replication/logical/reorderbuffer.c
index 03eb005c39d..d92247ca41b 100644
--- a/src/backend/replication/logical/reorderbuffer.c
+++ b/src/backend/replication/logical/reorderbuffer.c
@@ -2093,6 +2093,53 @@ ReorderBufferResetTXN(ReorderBuffer *rb, ReorderBufferTXN *txn,
Assert(txn->size == 0);
}
+/*
+ * Lookup changes in the transaction and apply only invalidation messages
+ */
+static void
+ReorderBufferProcessInvalidations(ReorderBuffer *rb, ReorderBufferTXN *txn)
+{
+ ReorderBufferChange *change;
+ ReorderBufferIterTXNState *volatile iterstate = NULL;
+
+ ReorderBufferIterTXNInit(rb, txn, &iterstate);
+ while ((change = ReorderBufferIterTXNNext(rb, iterstate)) != NULL)
+ {
+ CHECK_FOR_INTERRUPTS();
+
+ switch (change->action)
+ {
+ /* Only interested in invalidation messages */
+ case REORDER_BUFFER_CHANGE_INVALIDATION:
+ /*
+ * Execute the invalidation messages locally.
+ *
+ * XXX: what if ereport(ERROR) happens here?
+ */
+ ReorderBufferExecuteInvalidations(change->data.inval.ninvalidations,
+ change->data.inval.invalidations);
+ break;
+
+ /* Skip other changes because the transaction was aborted */
+ case REORDER_BUFFER_CHANGE_INSERT:
+ case REORDER_BUFFER_CHANGE_UPDATE:
+ case REORDER_BUFFER_CHANGE_DELETE:
+ case REORDER_BUFFER_CHANGE_MESSAGE:
+ case REORDER_BUFFER_CHANGE_INTERNAL_SNAPSHOT:
+ case REORDER_BUFFER_CHANGE_INTERNAL_COMMAND_ID:
+ case REORDER_BUFFER_CHANGE_INTERNAL_TUPLECID:
+ case REORDER_BUFFER_CHANGE_INTERNAL_SPEC_INSERT:
+ case REORDER_BUFFER_CHANGE_INTERNAL_SPEC_CONFIRM:
+ case REORDER_BUFFER_CHANGE_INTERNAL_SPEC_ABORT:
+ case REORDER_BUFFER_CHANGE_TRUNCATE:
+ break;
+ }
+ }
+
+ /* clean up the iterator */
+ ReorderBufferIterTXNFinish(rb, iterstate);
+}
+
/*
* Helper function for ReorderBufferReplay and ReorderBufferStreamTXN.
*
@@ -2650,6 +2697,8 @@ ReorderBufferProcessTXN(ReorderBuffer *rb, ReorderBufferTXN *txn,
errdata = NULL;
curtxn->concurrent_abort = true;
+ ReorderBufferProcessInvalidations(rb, txn);
+
/* Reset the TXN so that it is allowed to stream remaining data. */
ReorderBufferResetTXN(rb, txn, snapshot_now,
command_id, prev_lsn,
@@ -3330,41 +3379,60 @@ ReorderBufferAddInvalidations(ReorderBuffer *rb, TransactionId xid,
XLogRecPtr lsn, Size nmsgs,
SharedInvalidationMessage *msgs)
{
- ReorderBufferTXN *txn;
+ ReorderBufferAddInvalidationsExtended(rb, xid, lsn, nmsgs, msgs, true);
+}
+
+/*
+ * An extended workhorse version of ReorderBufferAddInvalidations(). This can
+ * control whether invalidation messages can be accumulated in reorderbuffer
+ * transactions, which would be distributed to all concurrent transactions at
+ * commit.
+ */
+void
+ReorderBufferAddInvalidationsExtended(ReorderBuffer *rb, TransactionId xid,
+ XLogRecPtr lsn, Size nmsgs,
+ SharedInvalidationMessage *msgs,
+ bool needs_distribute)
+{
MemoryContext oldcontext;
ReorderBufferChange *change;
- txn = ReorderBufferTXNByXid(rb, xid, true, NULL, lsn, true);
-
oldcontext = MemoryContextSwitchTo(rb->context);
- /*
- * Collect all the invalidations under the top transaction, if available,
- * so that we can execute them all together. See comments atop this
- * function.
- */
- txn = rbtxn_get_toptxn(txn);
-
Assert(nmsgs > 0);
- /* Accumulate invalidations. */
- if (txn->ninvalidations == 0)
- {
- txn->ninvalidations = nmsgs;
- txn->invalidations = (SharedInvalidationMessage *)
- palloc(sizeof(SharedInvalidationMessage) * nmsgs);
- memcpy(txn->invalidations, msgs,
- sizeof(SharedInvalidationMessage) * nmsgs);
- }
- else
+ if (needs_distribute)
{
- txn->invalidations = (SharedInvalidationMessage *)
- repalloc(txn->invalidations, sizeof(SharedInvalidationMessage) *
- (txn->ninvalidations + nmsgs));
+ ReorderBufferTXN *txn;
+
+ txn = ReorderBufferTXNByXid(rb, xid, true, NULL, lsn, true);
+
+ /*
+ * Collect all the invalidations under the top transaction, if
+ * available, so that we can execute them all together. See comments
+ * atop ReorderBufferAddInvalidations.
+ */
+ txn = rbtxn_get_toptxn(txn);
+
+ /* Accumulate invalidations. */
+ if (txn->ninvalidations == 0)
+ {
+ txn->ninvalidations = nmsgs;
+ txn->invalidations = (SharedInvalidationMessage *)
+ palloc(sizeof(SharedInvalidationMessage) * nmsgs);
+ memcpy(txn->invalidations, msgs,
+ sizeof(SharedInvalidationMessage) * nmsgs);
+ }
+ else
+ {
+ txn->invalidations = (SharedInvalidationMessage *)
+ repalloc(txn->invalidations,
+ sizeof(SharedInvalidationMessage) * (txn->ninvalidations + nmsgs));
- memcpy(txn->invalidations + txn->ninvalidations, msgs,
- nmsgs * sizeof(SharedInvalidationMessage));
- txn->ninvalidations += nmsgs;
+ memcpy(txn->invalidations + txn->ninvalidations, msgs,
+ nmsgs * sizeof(SharedInvalidationMessage));
+ txn->ninvalidations += nmsgs;
+ }
}
change = ReorderBufferGetChange(rb);
diff --git a/src/backend/replication/logical/snapbuild.c b/src/backend/replication/logical/snapbuild.c
index 110e0b0a044..a8225c1d905 100644
--- a/src/backend/replication/logical/snapbuild.c
+++ b/src/backend/replication/logical/snapbuild.c
@@ -945,8 +945,9 @@ SnapBuildDistributeSnapshotAndInval(SnapBuild *builder, XLogRecPtr lsn, Transact
{
Assert(msgs != NULL);
- ReorderBufferAddInvalidations(builder->reorder, txn->xid, lsn,
- ninvalidations, msgs);
+ ReorderBufferAddInvalidationsExtended(builder->reorder, txn->xid, lsn,
+ ninvalidations, msgs,
+ false);
}
}
}
diff --git a/src/include/replication/reorderbuffer.h b/src/include/replication/reorderbuffer.h
index 4c56f219fd8..001d0e716aa 100644
--- a/src/include/replication/reorderbuffer.h
+++ b/src/include/replication/reorderbuffer.h
@@ -709,6 +709,9 @@ extern void ReorderBufferAddNewTupleCids(ReorderBuffer *rb, TransactionId xid,
CommandId cmin, CommandId cmax, CommandId combocid);
extern void ReorderBufferAddInvalidations(ReorderBuffer *rb, TransactionId xid, XLogRecPtr lsn,
Size nmsgs, SharedInvalidationMessage *msgs);
+extern void ReorderBufferAddInvalidationsExtended(ReorderBuffer *rb, TransactionId xid, XLogRecPtr lsn,
+ Size nmsgs, SharedInvalidationMessage *msgs,
+ bool needs_distribute);
extern void ReorderBufferImmediateInvalidation(ReorderBuffer *rb, uint32 ninvalidations,
SharedInvalidationMessage *invalidations);
extern void ReorderBufferProcessXid(ReorderBuffer *rb, TransactionId xid, XLogRecPtr lsn);
--
2.43.5
^ permalink raw reply [nested|flat] 94+ messages in thread
* Re: Logical replication 'invalid memory alloc request size 1585837200' after upgrading to 17.5
@ 2025-05-23 03:00 Amit Kapila <amit.kapila16@gmail.com>
parent: Hayato Kuroda (Fujitsu) <kuroda.hayato@fujitsu.com>
0 siblings, 0 replies; 94+ messages in thread
From: Amit Kapila @ 2025-05-23 03:00 UTC (permalink / raw)
To: Hayato Kuroda (Fujitsu) <kuroda.hayato@fujitsu.com>; +Cc: Masahiko Sawada <sawada.mshk@gmail.com>; Duncan Sands <duncan.sands@deepbluecap.com>; pgsql-bugs@lists.postgresql.org <pgsql-bugs@lists.postgresql.org>
On Thu, May 22, 2025 at 6:29 PM Hayato Kuroda (Fujitsu)
<kuroda.hayato@fujitsu.com> wrote:
>
> Dear Amit, Sawada-san,
>
> > Good point. After replaying the transaction, it doesn't matter because
> > we would have already relayed the required invalidation while
> > processing REORDER_BUFFER_CHANGE_INVALIDATION messages. However
> > for
> > concurrent abort case it could matter. See my analysis for the same
> > below:
> >
> > Simulation of concurrent abort
> > ------------------------------------------
> > 1) S1: CREATE TABLE d(data text not null);
> > 2) S1: INSERT INTO d VALUES('d1');
> > 3) S2: BEGIN; INSERT INTO d VALUES('d2');
> > 4) S2: INSERT INTO unrelated_tab VALUES(1);
> > 5) S1: ALTER PUBLICATION pb ADD TABLE d;
> > 6) S2: INSERT INTO unrelated_tab VALUES(2);
> > 7) S2: ROLLBACK;
> > 8) S2: INSERT INTO d VALUES('d3');
> > 9) S1: INSERT INTO d VALUES('d4');
>
> > The problem with the sequence is that the insert from 3) could be
> > decoded *after* 5) in step 6) due to streaming and that to decode the
> > insert (which happened before the ALTER) the catalog snapshot and
> > cache state is from *before* the ALTER TABLE. Because the transaction
> > started in 3) doesn't actually modify any catalogs, no invalidations
> > are executed after decoding it. Now, assume, while decoding Insert
> > from 4), we detected a concurrent abort, then the distributed
> > invalidation won't be executed, and if we don't have accumulated
> > messages in txn->invalidations, then the invalidation from step 5)
> > won't be performed. The data loss can occur in steps 8 and 9. This is
> > just a theory, so I could be missing something.
>
> I verified this is real or not, and succeeded to reproduce. See appendix the
> detailed steps.
>
> > If the above turns out to be a problem, one idea for fixing it is that
> > for the concurrent abort case (both during streaming and for prepared
> > transaction's processing), we still check all the remaining changes
> > and process only the changes related to invalidations. This has to be
> > done before the current txn changes are freed via
> > ReorderBufferResetTXN->ReorderBufferTruncateTXN.
>
> I roughly implemented the part, PSA the updated version. One concern is whether we
> should consider the case that invalidations can cause ereport(ERROR). If happens,
> the walsender will exit at that time.
>
But, in the catch part, we are already executing invalidations:
...
/* make sure there's no cache pollution */
ReorderBufferExecuteInvalidations(txn->ninvalidations, txn->invalidations);
...
So, the behaviour should be the same.
--
With Regards,
Amit Kapila.
^ permalink raw reply [nested|flat] 94+ messages in thread
* Re: Logical replication 'invalid memory alloc request size 1585837200' after upgrading to 17.5
@ 2025-05-23 17:31 Masahiko Sawada <sawada.mshk@gmail.com>
parent: Amit Kapila <amit.kapila16@gmail.com>
1 sibling, 2 replies; 94+ messages in thread
From: Masahiko Sawada @ 2025-05-23 17:31 UTC (permalink / raw)
To: Amit Kapila <amit.kapila16@gmail.com>; +Cc: Duncan Sands <duncan.sands@deepbluecap.com>; pgsql-bugs@lists.postgresql.org
On Thu, May 22, 2025 at 3:57 AM Amit Kapila <amit.kapila16@gmail.com> wrote:
>
> On Wed, May 21, 2025 at 11:54 PM Masahiko Sawada <sawada.mshk@gmail.com> wrote:
> >
> > On Wed, May 21, 2025 at 4:12 AM Amit Kapila <amit.kapila16@gmail.com> wrote:
> > >
> >
> > > However, after queuing the
> > > change, we don't need to copy it along with the original transaction's
> > > invalidations. This is because the copy is only required when we don't
> > > process any changes in cases like ReorderBufferForget().
> >
> > It seems that we use the accumulated invalidation message also after
> > replaying or concurrently aborting a transaction via
> > ReorderBufferExecuteInvalidations(). Do we need to consider such cases
> > too?
> >
>
> Good point. After replaying the transaction, it doesn't matter because
> we would have already relayed the required invalidation while
> processing REORDER_BUFFER_CHANGE_INVALIDATION messages.
I think the reason why we execute all invalidation messages even in
non concurrent abort cases is that we need to invalidate all caches as
well that are loaded during the replay. Consider the following
sequences:
1) S1: CREATE TABLE d (data text not null);
2) S1: INSERT INTO d VALUES ('d1');
3) S2: BEGIN; INSERT INTO d VALUES ('d2');
4) S3: BEGIN; INSERT INTO d VALUES ('d3');
5) S1: ALTER PUBLICATION pb ADD TABLE d;
6) S2: INSERT INTO d VALUES ('d4');
7) S2: COMMIT;
8) S3: COMMIT;
9) S2: INSERT INTO d VALUES('d5');
10) S1: INSERT INTO d VALUES ('d6');
When replaying S2's first transaction at 7), we decode the insert from
3) using the snapshot which is from before the ALTER, creating the
cache for table 'd'. Then we invalidate the cache by the inval message
distributed from S1's the ALTER and then build the relcache again when
decoding the insert from 6). The cache is the state after the ALTER.
When replaying S3's transaction at 8), we should decode the insert
from 4) using the snapshot which is from before the ALTER. Since we
call ReorderBufferExecuteInvalidations() also in non concurrent abort
paths, we can invalidate the relcache built when decoding the insert
from 6). If we don't include the inval message distributed from 5) to
txn->invalidations, we don't invalidate the relcache and end up
sending the insert from 4) even though it happened before the ALTER.
> However for
> concurrent abort case it could matter. See my analysis for the same
> below:
>
> Simulation of concurrent abort
> ------------------------------------------
> 1) S1: CREATE TABLE d(data text not null);
> 2) S1: INSERT INTO d VALUES('d1');
> 3) S2: BEGIN; INSERT INTO d VALUES('d2');
> 4) S2: INSERT INTO unrelated_tab VALUES(1);
> 5) S1: ALTER PUBLICATION pb ADD TABLE d;
> 6) S2: INSERT INTO unrelated_tab VALUES(2);
> 7) S2: ROLLBACK;
> 8) S2: INSERT INTO d VALUES('d3');
> 9) S1: INSERT INTO d VALUES('d4');
>
> The problem with the sequence is that the insert from 3) could be
> decoded *after* 5) in step 6) due to streaming and that to decode the
> insert (which happened before the ALTER) the catalog snapshot and
> cache state is from *before* the ALTER TABLE. Because the transaction
> started in 3) doesn't actually modify any catalogs, no invalidations
> are executed after decoding it. Now, assume, while decoding Insert
> from 4), we detected a concurrent abort, then the distributed
> invalidation won't be executed, and if we don't have accumulated
> messages in txn->invalidations, then the invalidation from step 5)
> won't be performed. The data loss can occur in steps 8 and 9. This is
> just a theory, so I could be missing something.
This scenario makes sense to me. I agree that this turns out to be a problem.
>
> If the above turns out to be a problem, one idea for fixing it is that
> for the concurrent abort case (both during streaming and for prepared
> transaction's processing), we still check all the remaining changes
> and process only the changes related to invalidations. This has to be
> done before the current txn changes are freed via
> ReorderBufferResetTXN->ReorderBufferTruncateTXN.
>
> Thoughts?
If the above hypothesis is true, we need to consider another idea so
that we can execute invalidation messages in both cases.
Regards,
--
Masahiko Sawada
Amazon Web Services: https://aws.amazon.com
^ permalink raw reply [nested|flat] 94+ messages in thread
* Re: Logical replication 'invalid memory alloc request size 1585837200' after upgrading to 17.5
@ 2025-05-24 15:42 Duncan Sands <duncan.sands@deepbluecap.com>
parent: Hayato Kuroda (Fujitsu) <kuroda.hayato@fujitsu.com>
2 siblings, 0 replies; 94+ messages in thread
From: Duncan Sands @ 2025-05-24 15:42 UTC (permalink / raw)
To: Hayato Kuroda (Fujitsu) <kuroda.hayato@fujitsu.com>; Amit Kapila <amit.kapila16@gmail.com>; +Cc: pgsql-bugs@lists.postgresql.org <pgsql-bugs@lists.postgresql.org>; Sawada Masahiko <sawada.mshk@gmail.com>
Dear Hayato Kuroda, thank you so much for working on this problem. Your patch
PG17-0001-Avoid-distributing-invalidation-messages-several-tim.patch solves the
issue for me. Without it I get an invalid memory alloc request error within
about twenty minutes. With your patch, 24 hours have passed with no errors.
Best wishes, Duncan.
On 21/05/2025 13:48, Hayato Kuroda (Fujitsu) wrote:
> Dear hackers,
>
>> I think the problem here is that when we are distributing
>> invalidations to a concurrent transaction, in addition to queuing the
>> invalidations as a change, we also copy the distributed invalidations
>> along with the original transaction's invalidations via repalloc in
>> ReorderBufferAddInvalidations. So, when there are many in-progress
>> transactions, each would try to copy all its accumulated invalidations
>> to the remaining in-progress transactions. This could lead to such an
>> increase in allocation request size. However, after queuing the
>> change, we don't need to copy it along with the original transaction's
>> invalidations. This is because the copy is only required when we don't
>> process any changes in cases like ReorderBufferForget(). I have
>> analyzed all such cases, and my analysis is as follows:
>
> Based on the analysis, I created a PoC which avoids the repalloc().
> Invalidation messages distributed by SnapBuildDistributeSnapshotAndInval() are
> skipped to add in the list, just queued - repalloc can be skipped. Also, the function
> distributes messages only in the list, so received messages won't be sent again.
>
> Now a patch for PG17 is created for testing purpose. Duncan, can you apply this and
> confirms whether the issue can be solved?
>
> Best regards,
> Hayato Kuroda
> FUJITSU LIMITED
>
^ permalink raw reply [nested|flat] 94+ messages in thread
* RE: Logical replication 'invalid memory alloc request size 1585837200' after upgrading to 17.5
@ 2025-05-25 04:55 Hayato Kuroda (Fujitsu) <kuroda.hayato@fujitsu.com>
parent: Masahiko Sawada <sawada.mshk@gmail.com>
1 sibling, 0 replies; 94+ messages in thread
From: Hayato Kuroda (Fujitsu) @ 2025-05-25 04:55 UTC (permalink / raw)
To: Masahiko Sawada <sawada.mshk@gmail.com>; Amit Kapila <amit.kapila16@gmail.com>; +Cc: Duncan Sands <duncan.sands@deepbluecap.com>; pgsql-bugs@lists.postgresql.org <pgsql-bugs@lists.postgresql.org>
Dear Sawada-san,
> I think the reason why we execute all invalidation messages even in
> non concurrent abort cases is that we need to invalidate all caches as
> well that are loaded during the replay. Consider the following
> sequences:
>
> 1) S1: CREATE TABLE d (data text not null);
> 2) S1: INSERT INTO d VALUES ('d1');
> 3) S2: BEGIN; INSERT INTO d VALUES ('d2');
> 4) S3: BEGIN; INSERT INTO d VALUES ('d3');
> 5) S1: ALTER PUBLICATION pb ADD TABLE d;
> 6) S2: INSERT INTO d VALUES ('d4');
> 7) S2: COMMIT;
> 8) S3: COMMIT;
> 9) S2: INSERT INTO d VALUES('d5');
> 10) S1: INSERT INTO d VALUES ('d6');
>
> When replaying S2's first transaction at 7), we decode the insert from
> 3) using the snapshot which is from before the ALTER, creating the
> cache for table 'd'. Then we invalidate the cache by the inval message
> distributed from S1's the ALTER and then build the relcache again when
> decoding the insert from 6). The cache is the state after the ALTER.
> When replaying S3's transaction at 8), we should decode the insert
> from 4) using the snapshot which is from before the ALTER. Since we
> call ReorderBufferExecuteInvalidations() also in non concurrent abort
> paths, we can invalidate the relcache built when decoding the insert
> from 6). If we don't include the inval message distributed from 5) to
> txn->invalidations, we don't invalidate the relcache and end up
> sending the insert from 4) even though it happened before the ALTER.
Thanks for giving another scenario. Let me test this workload and share result later.
Best regards,
Hayato Kuroda
FUJITSU LIMITED
^ permalink raw reply [nested|flat] 94+ messages in thread
* RE: Logical replication 'invalid memory alloc request size 1585837200' after upgrading to 17.5
@ 2025-05-26 09:22 Hayato Kuroda (Fujitsu) <kuroda.hayato@fujitsu.com>
parent: Masahiko Sawada <sawada.mshk@gmail.com>
1 sibling, 1 reply; 94+ messages in thread
From: Hayato Kuroda (Fujitsu) @ 2025-05-26 09:22 UTC (permalink / raw)
To: Masahiko Sawada <sawada.mshk@gmail.com>; Amit Kapila <amit.kapila16@gmail.com>; +Cc: Duncan Sands <duncan.sands@deepbluecap.com>; pgsql-bugs@lists.postgresql.org <pgsql-bugs@lists.postgresql.org>
Dear Sawada-san,
> I think the reason why we execute all invalidation messages even in
> non concurrent abort cases is that we need to invalidate all caches as
> well that are loaded during the replay. Consider the following
> sequences:
>
> 1) S1: CREATE TABLE d (data text not null);
> 2) S1: INSERT INTO d VALUES ('d1');
> 3) S2: BEGIN; INSERT INTO d VALUES ('d2');
> 4) S3: BEGIN; INSERT INTO d VALUES ('d3');
> 5) S1: ALTER PUBLICATION pb ADD TABLE d;
> 6) S2: INSERT INTO d VALUES ('d4');
> 7) S2: COMMIT;
> 8) S3: COMMIT;
> 9) S2: INSERT INTO d VALUES('d5');
> 10) S1: INSERT INTO d VALUES ('d6');
>
> When replaying S2's first transaction at 7), we decode the insert from
> 3) using the snapshot which is from before the ALTER, creating the
> cache for table 'd'. Then we invalidate the cache by the inval message
> distributed from S1's the ALTER and then build the relcache again when
> decoding the insert from 6). The cache is the state after the ALTER.
> When replaying S3's transaction at 8), we should decode the insert
> from 4) using the snapshot which is from before the ALTER. Since we
> call ReorderBufferExecuteInvalidations() also in non concurrent abort
> paths, we can invalidate the relcache built when decoding the insert
> from 6). If we don't include the inval message distributed from 5) to
> txn->invalidations, we don't invalidate the relcache and end up
> sending the insert from 4) even though it happened before the ALTER.
You're right. I tested the workload on the latest PG17 and PoC, and confirmed that
PoC replicated d3 tuple, which is not good.
> If the above hypothesis is true, we need to consider another idea so
> that we can execute invalidation messages in both cases.
The straightforward fix is to check the change queue as well when the transaction
has invalidation messages. 0003 implemented that. One downside is that traversing
changes can affect performance. Currently we iterates all of changes even a
single REORDER_BUFFER_CHANGE_INVALIDATION. I cannot find better solutions for now.
Thought?
Best regards,
Hayato Kuroda
FUJITSU LIMITED
Attachments:
[application/octet-stream] v3-PG17-0001-Avoid-distributing-invalidation-messages-sev.patch (8.5K, ../../OS7PR01MB14968B3C263074A2DEB77DB58F565A@OS7PR01MB14968.jpnprd01.prod.outlook.com/2-v3-PG17-0001-Avoid-distributing-invalidation-messages-sev.patch)
download | inline diff:
From 31503be386bd5bea52bdc7248b76c7a4355fd142 Mon Sep 17 00:00:00 2001
From: Hayato Kuroda <kuroda.hayato@fujitsu.com>
Date: Wed, 21 May 2025 18:02:36 +0900
Subject: [PATCH v3-PG17] Avoid distributing invalidation messages several
times
---
.../replication/logical/reorderbuffer.c | 134 ++++++++++++++----
src/backend/replication/logical/snapbuild.c | 5 +-
src/include/replication/reorderbuffer.h | 3 +
3 files changed, 114 insertions(+), 28 deletions(-)
diff --git a/src/backend/replication/logical/reorderbuffer.c b/src/backend/replication/logical/reorderbuffer.c
index 03eb005c39d..47e5cdc4de6 100644
--- a/src/backend/replication/logical/reorderbuffer.c
+++ b/src/backend/replication/logical/reorderbuffer.c
@@ -2093,6 +2093,49 @@ ReorderBufferResetTXN(ReorderBuffer *rb, ReorderBufferTXN *txn,
Assert(txn->size == 0);
}
+/*
+ * Lookup changes in the transaction and apply only invalidation messages
+ */
+static void
+ReorderBufferProcessInvalidations(ReorderBuffer *rb, ReorderBufferTXN *txn)
+{
+ ReorderBufferChange *change;
+ ReorderBufferIterTXNState *volatile iterstate = NULL;
+
+ ReorderBufferIterTXNInit(rb, txn, &iterstate);
+ while ((change = ReorderBufferIterTXNNext(rb, iterstate)) != NULL)
+ {
+ CHECK_FOR_INTERRUPTS();
+
+ switch (change->action)
+ {
+ /* Only interested in invalidation messages */
+ case REORDER_BUFFER_CHANGE_INVALIDATION:
+ /* Execute the invalidation messages locally */
+ ReorderBufferExecuteInvalidations(change->data.inval.ninvalidations,
+ change->data.inval.invalidations);
+ break;
+
+ /* Skip other changes because the transaction was aborted */
+ case REORDER_BUFFER_CHANGE_INSERT:
+ case REORDER_BUFFER_CHANGE_UPDATE:
+ case REORDER_BUFFER_CHANGE_DELETE:
+ case REORDER_BUFFER_CHANGE_MESSAGE:
+ case REORDER_BUFFER_CHANGE_INTERNAL_SNAPSHOT:
+ case REORDER_BUFFER_CHANGE_INTERNAL_COMMAND_ID:
+ case REORDER_BUFFER_CHANGE_INTERNAL_TUPLECID:
+ case REORDER_BUFFER_CHANGE_INTERNAL_SPEC_INSERT:
+ case REORDER_BUFFER_CHANGE_INTERNAL_SPEC_CONFIRM:
+ case REORDER_BUFFER_CHANGE_INTERNAL_SPEC_ABORT:
+ case REORDER_BUFFER_CHANGE_TRUNCATE:
+ break;
+ }
+ }
+
+ /* clean up the iterator */
+ ReorderBufferIterTXNFinish(rb, iterstate);
+}
+
/*
* Helper function for ReorderBufferReplay and ReorderBufferStreamTXN.
*
@@ -2119,6 +2162,7 @@ ReorderBufferProcessTXN(ReorderBuffer *rb, ReorderBufferTXN *txn,
ReorderBufferChange *volatile specinsert = NULL;
volatile bool stream_started = false;
ReorderBufferTXN *volatile curtxn = NULL;
+ bool contains_inval = false;
/* build data to be able to lookup the CommandIds of catalog tuples */
ReorderBufferBuildTupleCidHash(rb, txn);
@@ -2423,6 +2467,9 @@ ReorderBufferProcessTXN(ReorderBuffer *rb, ReorderBufferTXN *txn,
/* Execute the invalidation messages locally */
ReorderBufferExecuteInvalidations(change->data.inval.ninvalidations,
change->data.inval.invalidations);
+
+ /* Also tracks the existence to iterate changes again */
+ contains_inval = true;
break;
case REORDER_BUFFER_CHANGE_INTERNAL_SNAPSHOT:
@@ -2576,6 +2623,14 @@ ReorderBufferProcessTXN(ReorderBuffer *rb, ReorderBufferTXN *txn,
/* make sure there's no cache pollution */
ReorderBufferExecuteInvalidations(txn->ninvalidations, txn->invalidations);
+ /*
+ * The change queue must be checked again if the transaction contains
+ * the inval messages because some them are not stored in
+ * txn->invalidations.
+ */
+ if (contains_inval)
+ ReorderBufferProcessInvalidations(rb, txn);
+
if (using_subtxn)
RollbackAndReleaseCurrentSubTransaction();
@@ -2650,6 +2705,14 @@ ReorderBufferProcessTXN(ReorderBuffer *rb, ReorderBufferTXN *txn,
errdata = NULL;
curtxn->concurrent_abort = true;
+ /*
+ * Iterate changes in the transaction to execute invalidation
+ * messages. Unlike the PG_TRY part, this must be done
+ * unconditionally because the processing might fail before we
+ * reach invalidation messages.
+ */
+ ReorderBufferProcessInvalidations(rb, txn);
+
/* Reset the TXN so that it is allowed to stream remaining data. */
ReorderBufferResetTXN(rb, txn, snapshot_now,
command_id, prev_lsn,
@@ -3330,41 +3393,60 @@ ReorderBufferAddInvalidations(ReorderBuffer *rb, TransactionId xid,
XLogRecPtr lsn, Size nmsgs,
SharedInvalidationMessage *msgs)
{
- ReorderBufferTXN *txn;
+ ReorderBufferAddInvalidationsExtended(rb, xid, lsn, nmsgs, msgs, true);
+}
+
+/*
+ * An extended workhorse version of ReorderBufferAddInvalidations(). This can
+ * control whether invalidation messages can be accumulated in reorderbuffer
+ * transactions, which would be distributed to all concurrent transactions at
+ * commit.
+ */
+void
+ReorderBufferAddInvalidationsExtended(ReorderBuffer *rb, TransactionId xid,
+ XLogRecPtr lsn, Size nmsgs,
+ SharedInvalidationMessage *msgs,
+ bool needs_distribute)
+{
MemoryContext oldcontext;
ReorderBufferChange *change;
- txn = ReorderBufferTXNByXid(rb, xid, true, NULL, lsn, true);
-
oldcontext = MemoryContextSwitchTo(rb->context);
- /*
- * Collect all the invalidations under the top transaction, if available,
- * so that we can execute them all together. See comments atop this
- * function.
- */
- txn = rbtxn_get_toptxn(txn);
-
Assert(nmsgs > 0);
- /* Accumulate invalidations. */
- if (txn->ninvalidations == 0)
- {
- txn->ninvalidations = nmsgs;
- txn->invalidations = (SharedInvalidationMessage *)
- palloc(sizeof(SharedInvalidationMessage) * nmsgs);
- memcpy(txn->invalidations, msgs,
- sizeof(SharedInvalidationMessage) * nmsgs);
- }
- else
+ if (needs_distribute)
{
- txn->invalidations = (SharedInvalidationMessage *)
- repalloc(txn->invalidations, sizeof(SharedInvalidationMessage) *
- (txn->ninvalidations + nmsgs));
+ ReorderBufferTXN *txn;
+
+ txn = ReorderBufferTXNByXid(rb, xid, true, NULL, lsn, true);
+
+ /*
+ * Collect all the invalidations under the top transaction, if
+ * available, so that we can execute them all together. See comments
+ * atop ReorderBufferAddInvalidations.
+ */
+ txn = rbtxn_get_toptxn(txn);
- memcpy(txn->invalidations + txn->ninvalidations, msgs,
- nmsgs * sizeof(SharedInvalidationMessage));
- txn->ninvalidations += nmsgs;
+ /* Accumulate invalidations. */
+ if (txn->ninvalidations == 0)
+ {
+ txn->ninvalidations = nmsgs;
+ txn->invalidations = (SharedInvalidationMessage *)
+ palloc(sizeof(SharedInvalidationMessage) * nmsgs);
+ memcpy(txn->invalidations, msgs,
+ sizeof(SharedInvalidationMessage) * nmsgs);
+ }
+ else
+ {
+ txn->invalidations = (SharedInvalidationMessage *)
+ repalloc(txn->invalidations,
+ sizeof(SharedInvalidationMessage) * (txn->ninvalidations + nmsgs));
+
+ memcpy(txn->invalidations + txn->ninvalidations, msgs,
+ nmsgs * sizeof(SharedInvalidationMessage));
+ txn->ninvalidations += nmsgs;
+ }
}
change = ReorderBufferGetChange(rb);
diff --git a/src/backend/replication/logical/snapbuild.c b/src/backend/replication/logical/snapbuild.c
index 110e0b0a044..a8225c1d905 100644
--- a/src/backend/replication/logical/snapbuild.c
+++ b/src/backend/replication/logical/snapbuild.c
@@ -945,8 +945,9 @@ SnapBuildDistributeSnapshotAndInval(SnapBuild *builder, XLogRecPtr lsn, Transact
{
Assert(msgs != NULL);
- ReorderBufferAddInvalidations(builder->reorder, txn->xid, lsn,
- ninvalidations, msgs);
+ ReorderBufferAddInvalidationsExtended(builder->reorder, txn->xid, lsn,
+ ninvalidations, msgs,
+ false);
}
}
}
diff --git a/src/include/replication/reorderbuffer.h b/src/include/replication/reorderbuffer.h
index 4c56f219fd8..001d0e716aa 100644
--- a/src/include/replication/reorderbuffer.h
+++ b/src/include/replication/reorderbuffer.h
@@ -709,6 +709,9 @@ extern void ReorderBufferAddNewTupleCids(ReorderBuffer *rb, TransactionId xid,
CommandId cmin, CommandId cmax, CommandId combocid);
extern void ReorderBufferAddInvalidations(ReorderBuffer *rb, TransactionId xid, XLogRecPtr lsn,
Size nmsgs, SharedInvalidationMessage *msgs);
+extern void ReorderBufferAddInvalidationsExtended(ReorderBuffer *rb, TransactionId xid, XLogRecPtr lsn,
+ Size nmsgs, SharedInvalidationMessage *msgs,
+ bool needs_distribute);
extern void ReorderBufferImmediateInvalidation(ReorderBuffer *rb, uint32 ninvalidations,
SharedInvalidationMessage *invalidations);
extern void ReorderBufferProcessXid(ReorderBuffer *rb, TransactionId xid, XLogRecPtr lsn);
--
2.47.1
^ permalink raw reply [nested|flat] 94+ messages in thread
* Re: Logical replication 'invalid memory alloc request size 1585837200' after upgrading to 17.5
@ 2025-05-26 11:18 Amit Kapila <amit.kapila16@gmail.com>
parent: Hayato Kuroda (Fujitsu) <kuroda.hayato@fujitsu.com>
0 siblings, 2 replies; 94+ messages in thread
From: Amit Kapila @ 2025-05-26 11:18 UTC (permalink / raw)
To: Hayato Kuroda (Fujitsu) <kuroda.hayato@fujitsu.com>; +Cc: Masahiko Sawada <sawada.mshk@gmail.com>; Duncan Sands <duncan.sands@deepbluecap.com>; pgsql-bugs@lists.postgresql.org <pgsql-bugs@lists.postgresql.org>
On Mon, May 26, 2025 at 2:52 PM Hayato Kuroda (Fujitsu)
<kuroda.hayato@fujitsu.com> wrote:
>
> > If the above hypothesis is true, we need to consider another idea so
> > that we can execute invalidation messages in both cases.
>
> The straightforward fix is to check the change queue as well when the transaction
> has invalidation messages. 0003 implemented that. One downside is that traversing
> changes can affect performance. Currently we iterates all of changes even a
> single REORDER_BUFFER_CHANGE_INVALIDATION. I cannot find better solutions for now.
>
It can impact the performance for large transactions with fewer
invalidations, especially the ones which has spilled changes because
it needs to traverse the entire list of changes again at the end. The
other idea would be to add new member(s) in ReorderBufferTXN to
receive distributed invalidations. For adding the new member in
ReorderBufferTXN: (a) in HEAD, it should be okay, (b) for
backbranches, we may be able to add at the end, but we should check if
there are any extensions using sizeof(ReorderBufferTxn) and if they
are using what we need to do.
I think the new member could be similar to existing members (uint32
ninvalidations; and SharedInvalidationMessage *invalidations;) or a
separate change queue of only REORDER_BUFFER_CHANGE_INVALIDATION
messages. The second one is worth considering because multiple
transactions can distribute their invalidations to a single txn in
chunks, which can be stored as separate changes, and the other benefit
of the second one is lower risk of the need for a larger chunk of
memory allocation due to repalloc. Also, it would be easy to consider
its size via ReorderBufferChangeMemoryUpdate.
--
With Regards,
Amit Kapila.
^ permalink raw reply [nested|flat] 94+ messages in thread
* Re: Logical replication 'invalid memory alloc request size 1585837200' after upgrading to 17.5
@ 2025-05-26 11:47 Christoph Berg <myon@debian.org>
parent: Duncan Sands <duncan.sands@deepbluecap.com>
2 siblings, 0 replies; 94+ messages in thread
From: Christoph Berg @ 2025-05-26 11:47 UTC (permalink / raw)
To: pgsql-bugs@lists.postgresql.org
Re: Duncan Sands
> PostgreSQL v17.5 (Ubuntu 17.5-1.pgdg24.04+1); Ubuntu 24.04.2 LTS (kernel
> 6.8.0); x86-64
Fwiw, one more field report from Debian:
> We had a severe issue after upgrading postgresql-16 from 16.8-1.pgdg110+1 to 16.9-1.pgdg110+1. The following error happened quickly on a logical replication (sorry for the French).
>
> ERREUR: n'a pas pu recevoir des données du flux de WAL : ERROR: invalid memory alloc request size 1196912896
>
> After trying several things, we finally managed to restart the replication after downgrading the pg-related packages on this host alone (the other servers were not impacted).
https://salsa.debian.org/postgresql/postgresql/-/issues/4
(no extra details in there, though)
Christoph
^ permalink raw reply [nested|flat] 94+ messages in thread
* Re: Logical replication 'invalid memory alloc request size 1585837200' after upgrading to 17.5
@ 2025-05-27 21:05 Masahiko Sawada <sawada.mshk@gmail.com>
parent: Amit Kapila <amit.kapila16@gmail.com>
1 sibling, 1 reply; 94+ messages in thread
From: Masahiko Sawada @ 2025-05-27 21:05 UTC (permalink / raw)
To: Amit Kapila <amit.kapila16@gmail.com>; +Cc: Hayato Kuroda (Fujitsu) <kuroda.hayato@fujitsu.com>; Duncan Sands <duncan.sands@deepbluecap.com>; pgsql-bugs@lists.postgresql.org <pgsql-bugs@lists.postgresql.org>
On Mon, May 26, 2025 at 4:19 AM Amit Kapila <amit.kapila16@gmail.com> wrote:
>
> On Mon, May 26, 2025 at 2:52 PM Hayato Kuroda (Fujitsu)
> <kuroda.hayato@fujitsu.com> wrote:
> >
> > > If the above hypothesis is true, we need to consider another idea so
> > > that we can execute invalidation messages in both cases.
> >
> > The straightforward fix is to check the change queue as well when the transaction
> > has invalidation messages. 0003 implemented that. One downside is that traversing
> > changes can affect performance. Currently we iterates all of changes even a
> > single REORDER_BUFFER_CHANGE_INVALIDATION. I cannot find better solutions for now.
> >
>
> It can impact the performance for large transactions with fewer
> invalidations, especially the ones which has spilled changes because
> it needs to traverse the entire list of changes again at the end.
Agreed.
> The
> other idea would be to add new member(s) in ReorderBufferTXN to
> receive distributed invalidations. For adding the new member in
> ReorderBufferTXN: (a) in HEAD, it should be okay, (b) for
> backbranches, we may be able to add at the end, but we should check if
> there are any extensions using sizeof(ReorderBufferTxn) and if they
> are using what we need to do.
If we can make sure that that change won't break the existing
extensions, I think this would be the most reasonable solution.
Regards,
--
Masahiko Sawada
Amazon Web Services: https://aws.amazon.com
^ permalink raw reply [nested|flat] 94+ messages in thread
* RE: Logical replication 'invalid memory alloc request size 1585837200' after upgrading to 17.5
@ 2025-05-28 12:27 Hayato Kuroda (Fujitsu) <kuroda.hayato@fujitsu.com>
parent: Masahiko Sawada <sawada.mshk@gmail.com>
0 siblings, 1 reply; 94+ messages in thread
From: Hayato Kuroda (Fujitsu) @ 2025-05-28 12:27 UTC (permalink / raw)
To: Masahiko Sawada <sawada.mshk@gmail.com>; Amit Kapila <amit.kapila16@gmail.com>; +Cc: Duncan Sands <duncan.sands@deepbluecap.com>; pgsql-bugs@lists.postgresql.org <pgsql-bugs@lists.postgresql.org>
Dear Sawada-san, Amit,
> > It can impact the performance for large transactions with fewer
> > invalidations, especially the ones which has spilled changes because
> > it needs to traverse the entire list of changes again at the end.
>
> Agreed.
>
> > The
> > other idea would be to add new member(s) in ReorderBufferTXN to
> > receive distributed invalidations. For adding the new member in
> > ReorderBufferTXN: (a) in HEAD, it should be okay, (b) for
> > backbranches, we may be able to add at the end, but we should check if
> > there are any extensions using sizeof(ReorderBufferTxn) and if they
> > are using what we need to do.
>
> If we can make sure that that change won't break the existing
> extensions, I think this would be the most reasonable solution.
Based on the discussion, I created PoC for master/PG17. Please see attached.
The basic idea is to introduce the new queue which only contains distributed inval
messages. Contents are consumed at end of transactions. I feel some of codes can
be re-used so that internal functions are introduced. At least, it could pass
regression tests and workloads discussed here.
Best regards,
Hayato Kuroda
FUJITSU LIMITED
Attachments:
[application/octet-stream] v4-PG17-0001-Avoid-distributing-invalidation-messages-sev.patch (21.2K, ../../OSCPR01MB149667B316377CE0615E15138F567A@OSCPR01MB14966.jpnprd01.prod.outlook.com/2-v4-PG17-0001-Avoid-distributing-invalidation-messages-sev.patch)
download | inline diff:
From 82e688a1734cda5076ab1d6fdd5997f1e8cbe792 Mon Sep 17 00:00:00 2001
From: Hayato Kuroda <kuroda.hayato@fujitsu.com>
Date: Wed, 21 May 2025 18:02:36 +0900
Subject: [PATCH v4-PG17] Avoid distributing invalidation messages several
times
---
.../replication/logical/reorderbuffer.c | 367 ++++++++++++++----
src/backend/replication/logical/snapbuild.c | 5 +-
src/include/replication/reorderbuffer.h | 6 +
3 files changed, 307 insertions(+), 71 deletions(-)
diff --git a/src/backend/replication/logical/reorderbuffer.c b/src/backend/replication/logical/reorderbuffer.c
index 03eb005c39d..34c5dd88335 100644
--- a/src/backend/replication/logical/reorderbuffer.c
+++ b/src/backend/replication/logical/reorderbuffer.c
@@ -239,11 +239,30 @@ static void AssertTXNLsnOrder(ReorderBuffer *rb);
*/
static void ReorderBufferIterTXNInit(ReorderBuffer *rb, ReorderBufferTXN *txn,
ReorderBufferIterTXNState *volatile *iter_state);
+static void ReorderBufferIterTXNInitForInval(ReorderBuffer *rb,
+ ReorderBufferTXN *txn,
+ ReorderBufferIterTXNState *volatile *iter_state);
+static void ReorderBufferIterTXNInitInternal(ReorderBuffer *rb,
+ ReorderBufferTXN *txn,
+ ReorderBufferIterTXNState *volatile *iter_state,
+ bool iterate_inval);
+
static ReorderBufferChange *ReorderBufferIterTXNNext(ReorderBuffer *rb, ReorderBufferIterTXNState *state);
+static ReorderBufferChange *ReorderBufferIterTXNNextForInval(ReorderBuffer *rb, ReorderBufferIterTXNState *state);
+static ReorderBufferChange *ReorderBufferIterTXNNextInternal(ReorderBuffer *rb, ReorderBufferIterTXNState *state,
+ bool iterate_inval);
static void ReorderBufferIterTXNFinish(ReorderBuffer *rb,
ReorderBufferIterTXNState *state);
static void ReorderBufferExecuteInvalidations(uint32 nmsgs, SharedInvalidationMessage *msgs);
+static void ReorderBufferQueueChangeInternal(ReorderBuffer *rb, TransactionId xid, XLogRecPtr lsn,
+ ReorderBufferChange *change, bool toast_insert,
+ bool for_inval);
+static void ReorderBufferExecuteInvalidationsInQueue(ReorderBuffer *rb, ReorderBufferTXN *txn);
+
+static void ReorderBufferQueueInval(ReorderBuffer *rb, TransactionId xid,
+ XLogRecPtr lsn, ReorderBufferChange *change);
+
/*
* ---------------------------------------
* Disk serialization support functions
@@ -428,6 +447,7 @@ ReorderBufferGetTXN(ReorderBuffer *rb)
dlist_init(&txn->changes);
dlist_init(&txn->tuplecids);
dlist_init(&txn->subtxns);
+ dlist_init(&txn->distributed_inval);
/* InvalidCommandId is not zero, so set it explicitly */
txn->command_id = InvalidCommandId;
@@ -787,6 +807,24 @@ ReorderBufferProcessPartialChange(ReorderBuffer *rb, ReorderBufferTXN *txn,
void
ReorderBufferQueueChange(ReorderBuffer *rb, TransactionId xid, XLogRecPtr lsn,
ReorderBufferChange *change, bool toast_insert)
+{
+ ReorderBufferQueueChangeInternal(rb, xid, lsn, change, toast_insert, false);
+}
+
+static void
+ReorderBufferQueueInval(ReorderBuffer *rb, TransactionId xid,
+ XLogRecPtr lsn, ReorderBufferChange *change)
+{
+ ReorderBufferQueueChangeInternal(rb, xid, lsn, change, false, true);
+}
+
+/*
+ * Actual workhorse for ReorderBufferQueueChange and ReorderBufferQueueInval
+ */
+static void
+ReorderBufferQueueChangeInternal(ReorderBuffer *rb, TransactionId xid,
+ XLogRecPtr lsn, ReorderBufferChange *change,
+ bool toast_insert, bool for_inval)
{
ReorderBufferTXN *txn;
@@ -797,7 +835,7 @@ ReorderBufferQueueChange(ReorderBuffer *rb, TransactionId xid, XLogRecPtr lsn,
* transaction is aborted. So there is no point in collecting further
* changes for it.
*/
- if (txn->concurrent_abort)
+ if (!for_inval && txn->concurrent_abort)
{
/*
* We don't need to update memory accounting for this change as we
@@ -825,22 +863,84 @@ ReorderBufferQueueChange(ReorderBuffer *rb, TransactionId xid, XLogRecPtr lsn,
}
change->lsn = lsn;
- change->txn = txn;
+
+ /*
+ * Associate the change with current transaction only if we are not
+ * distributing invalidation messages. Associated one has shorter
+ * lifetime than the target transaction, which may cause dangling
+ * link issues.
+ */
+ change->txn = for_inval ? NULL : txn;
Assert(InvalidXLogRecPtr != lsn);
- dlist_push_tail(&txn->changes, &change->node);
- txn->nentries++;
- txn->nentries_mem++;
- /* update memory accounting information */
- ReorderBufferChangeMemoryUpdate(rb, change, NULL, true,
- ReorderBufferChangeSize(change));
+ if (for_inval)
+ {
+ dlist_push_tail(&txn->distributed_inval, &change->node);
+ txn->nentries_distr++;
+ }
+ else
+ {
+ dlist_push_tail(&txn->changes, &change->node);
+ txn->nentries++;
+ txn->nentries_mem++;
+
+ /*
+ * For normal changes, we must update memory usage and process pertial
+ * changes. This is not needed for distributed ones, because they
+ * won't be added to the total amount of memory.
+ */
+
+ /* update memory accounting information */
+ ReorderBufferChangeMemoryUpdate(rb, change, NULL, true,
+ ReorderBufferChangeSize(change));
+
+ /* process partial change */
+ ReorderBufferProcessPartialChange(rb, txn, change, toast_insert);
- /* process partial change */
- ReorderBufferProcessPartialChange(rb, txn, change, toast_insert);
+ /* check the memory limits and evict something if needed */
+ ReorderBufferCheckMemoryLimit(rb);
+ }
+}
+
+static void
+ReorderBufferExecuteInvalidationsInQueue(ReorderBuffer *rb, ReorderBufferTXN *txn)
+{
+ ReorderBufferChange *change;
+ ReorderBufferIterTXNState *volatile iterstate = NULL;
+
+ ReorderBufferIterTXNInitForInval(rb, txn, &iterstate);
+ while ((change = ReorderBufferIterTXNNextForInval(rb, iterstate)) != NULL)
+ {
+ /* XXX do we have to do something here? */
+
+ switch (change->action)
+ {
+ /* Only interested in invalidation */
+ case REORDER_BUFFER_CHANGE_INVALIDATION:
+ /* Execute the invalidation messages locally */
+ ReorderBufferExecuteInvalidations(change->data.inval.ninvalidations,
+ change->data.inval.invalidations);
+ break;
+
+ /* Skip other changes because the transaction was aborted */
+ case REORDER_BUFFER_CHANGE_INSERT:
+ case REORDER_BUFFER_CHANGE_UPDATE:
+ case REORDER_BUFFER_CHANGE_DELETE:
+ case REORDER_BUFFER_CHANGE_MESSAGE:
+ case REORDER_BUFFER_CHANGE_INTERNAL_SNAPSHOT:
+ case REORDER_BUFFER_CHANGE_INTERNAL_COMMAND_ID:
+ case REORDER_BUFFER_CHANGE_INTERNAL_TUPLECID:
+ case REORDER_BUFFER_CHANGE_INTERNAL_SPEC_INSERT:
+ case REORDER_BUFFER_CHANGE_INTERNAL_SPEC_CONFIRM:
+ case REORDER_BUFFER_CHANGE_INTERNAL_SPEC_ABORT:
+ case REORDER_BUFFER_CHANGE_TRUNCATE:
+ break;
+ }
+ }
- /* check the memory limits and evict something if needed */
- ReorderBufferCheckMemoryLimit(rb);
+ /* clean up the iterator */
+ ReorderBufferIterTXNFinish(rb, iterstate);
}
/*
@@ -1261,36 +1361,61 @@ ReorderBufferIterCompare(Datum a, Datum b, void *arg)
static void
ReorderBufferIterTXNInit(ReorderBuffer *rb, ReorderBufferTXN *txn,
ReorderBufferIterTXNState *volatile *iter_state)
+{
+ ReorderBufferIterTXNInitInternal(rb, txn, iter_state, false);
+}
+
+static void
+ReorderBufferIterTXNInitForInval(ReorderBuffer *rb, ReorderBufferTXN *txn,
+ ReorderBufferIterTXNState *volatile *iter_state)
+{
+ ReorderBufferIterTXNInitInternal(rb, txn, iter_state, true);
+}
+
+/*
+ * Actual workhorse for ReorderBufferIterTXNInit and ReorderBufferIterTXNInitForInval
+ */
+static void
+ReorderBufferIterTXNInitInternal(ReorderBuffer *rb, ReorderBufferTXN *txn,
+ ReorderBufferIterTXNState *volatile *iter_state,
+ bool iterate_inval)
{
Size nr_txns = 0;
ReorderBufferIterTXNState *state;
dlist_iter cur_txn_i;
int32 off;
+ uint64 nentries;
*iter_state = NULL;
/* Check ordering of changes in the toplevel transaction. */
- AssertChangeLsnOrder(txn);
+ if (!iterate_inval)
+ AssertChangeLsnOrder(txn);
+
+ nentries = iterate_inval ? txn->nentries_distr : txn->nentries;
/*
* Calculate the size of our heap: one element for every transaction that
* contains changes. (Besides the transactions already in the reorder
* buffer, we count the one we were directly passed.)
*/
- if (txn->nentries > 0)
+ if (nentries > 0)
nr_txns++;
- dlist_foreach(cur_txn_i, &txn->subtxns)
+ if (!iterate_inval)
{
- ReorderBufferTXN *cur_txn;
+ dlist_foreach(cur_txn_i, &txn->subtxns)
+ {
+ ReorderBufferTXN *cur_txn;
- cur_txn = dlist_container(ReorderBufferTXN, node, cur_txn_i.cur);
+ cur_txn = dlist_container(ReorderBufferTXN, node, cur_txn_i.cur);
- /* Check ordering of changes in this subtransaction. */
- AssertChangeLsnOrder(cur_txn);
+ /* Check ordering of changes in this subtransaction. */
+ AssertChangeLsnOrder(cur_txn);
- if (cur_txn->nentries > 0)
- nr_txns++;
+ if (cur_txn->nentries > 0)
+ nr_txns++;
+ }
}
/* allocate iteration state */
@@ -1324,11 +1449,12 @@ ReorderBufferIterTXNInit(ReorderBuffer *rb, ReorderBufferTXN *txn,
off = 0;
/* add toplevel transaction if it contains changes */
- if (txn->nentries > 0)
+ if (nentries > 0)
{
ReorderBufferChange *cur_change;
+ dlist_head *head;
- if (rbtxn_is_serialized(txn))
+ if (!iterate_inval && rbtxn_is_serialized(txn))
{
/* serialize remaining changes */
ReorderBufferSerializeTXN(rb, txn);
@@ -1336,8 +1462,10 @@ ReorderBufferIterTXNInit(ReorderBuffer *rb, ReorderBufferTXN *txn,
&state->entries[off].segno);
}
+ head = iterate_inval ? &txn->distributed_inval : &txn->changes;
+
cur_change = dlist_head_element(ReorderBufferChange, node,
- &txn->changes);
+ head);
state->entries[off].lsn = cur_change->lsn;
state->entries[off].change = cur_change;
@@ -1346,34 +1474,42 @@ ReorderBufferIterTXNInit(ReorderBuffer *rb, ReorderBufferTXN *txn,
binaryheap_add_unordered(state->heap, Int32GetDatum(off++));
}
- /* add subtransactions if they contain changes */
- dlist_foreach(cur_txn_i, &txn->subtxns)
+ if (!iterate_inval)
{
- ReorderBufferTXN *cur_txn;
-
- cur_txn = dlist_container(ReorderBufferTXN, node, cur_txn_i.cur);
-
- if (cur_txn->nentries > 0)
+ /* add subtransactions if they contain changes */
+ dlist_foreach(cur_txn_i, &txn->subtxns)
{
- ReorderBufferChange *cur_change;
+ ReorderBufferTXN *cur_txn;
+
+ cur_txn = dlist_container(ReorderBufferTXN, node, cur_txn_i.cur);
- if (rbtxn_is_serialized(cur_txn))
+ if (cur_txn->nentries > 0)
{
- /* serialize remaining changes */
- ReorderBufferSerializeTXN(rb, cur_txn);
- ReorderBufferRestoreChanges(rb, cur_txn,
- &state->entries[off].file,
- &state->entries[off].segno);
- }
- cur_change = dlist_head_element(ReorderBufferChange, node,
- &cur_txn->changes);
+ ReorderBufferChange *cur_change;
+ dlist_head *head;
+
+ if (rbtxn_is_serialized(cur_txn))
+ {
+ /* serialize remaining changes */
+ ReorderBufferSerializeTXN(rb, cur_txn);
+ ReorderBufferRestoreChanges(rb, cur_txn,
+ &state->entries[off].file,
+ &state->entries[off].segno);
+ }
+
+ head = iterate_inval ? &cur_txn->distributed_inval : &cur_txn->changes;
+
+ cur_change = dlist_head_element(ReorderBufferChange, node,
+ head);
- state->entries[off].lsn = cur_change->lsn;
- state->entries[off].change = cur_change;
- state->entries[off].txn = cur_txn;
+ state->entries[off].lsn = cur_change->lsn;
+ state->entries[off].change = cur_change;
+ state->entries[off].txn = cur_txn;
- binaryheap_add_unordered(state->heap, Int32GetDatum(off++));
+ binaryheap_add_unordered(state->heap, Int32GetDatum(off++));
+ }
}
+
}
/* assemble a valid binary heap */
@@ -1388,10 +1524,29 @@ ReorderBufferIterTXNInit(ReorderBuffer *rb, ReorderBufferTXN *txn,
*/
static ReorderBufferChange *
ReorderBufferIterTXNNext(ReorderBuffer *rb, ReorderBufferIterTXNState *state)
+{
+ return ReorderBufferIterTXNNextInternal(rb, state, false);
+}
+
+static ReorderBufferChange *
+ReorderBufferIterTXNNextForInval(ReorderBuffer *rb, ReorderBufferIterTXNState *state)
+{
+ return ReorderBufferIterTXNNextInternal(rb, state, true);
+}
+
+/*
+ * Actual workhorse for ReorderBufferIterTXNNext and ReorderBufferIterTXNNextForInval
+ */
+static ReorderBufferChange *
+ReorderBufferIterTXNNextInternal(ReorderBuffer *rb, ReorderBufferIterTXNState *state,
+ bool iterate_inval)
{
ReorderBufferChange *change;
ReorderBufferIterTXNEntry *entry;
int32 off;
+ dlist_head *head;
+ uint64 nentries;
+ uint64 nentries_mem;
/* nothing there anymore */
if (state->heap->bh_size == 0)
@@ -1411,15 +1566,33 @@ ReorderBufferIterTXNNext(ReorderBuffer *rb, ReorderBufferIterTXNState *state)
change = entry->change;
+ if (iterate_inval)
+ {
+ head = &entry->txn->distributed_inval;
+ nentries = entry->txn->nentries_distr;
+
+ /*
+ * The inval queue won't be serialized, so the number of entries on the
+ * memory is same as whole entries.
+ */
+ nentries_mem = entry->txn->nentries_distr;
+ }
+ else
+ {
+ head = &entry->txn->changes;
+ nentries = entry->txn->nentries;
+ nentries_mem = entry->txn->nentries_mem;
+ }
+
/*
* update heap with information about which transaction has the next
* relevant change in LSN order
*/
/* there are in-memory changes */
- if (dlist_has_next(&entry->txn->changes, &entry->change->node))
+ if (dlist_has_next(head, &entry->change->node))
{
- dlist_node *next = dlist_next_node(&entry->txn->changes, &change->node);
+ dlist_node *next = dlist_next_node(head, &change->node);
ReorderBufferChange *next_change =
dlist_container(ReorderBufferChange, node, next);
@@ -1432,7 +1605,7 @@ ReorderBufferIterTXNNext(ReorderBuffer *rb, ReorderBufferIterTXNState *state)
}
/* try to load changes from disk */
- if (entry->txn->nentries != entry->txn->nentries_mem)
+ if (nentries != nentries_mem)
{
/*
* Ugly: restoring changes will reuse *Change records, thus delete the
@@ -1453,11 +1626,11 @@ ReorderBufferIterTXNNext(ReorderBuffer *rb, ReorderBufferIterTXNState *state)
/* successfully restored changes from disk */
ReorderBufferChange *next_change =
dlist_head_element(ReorderBufferChange, node,
- &entry->txn->changes);
+ head);
elog(DEBUG2, "restored %u/%u changes from disk",
- (uint32) entry->txn->nentries_mem,
- (uint32) entry->txn->nentries);
+ (uint32) nentries_mem,
+ (uint32) nentries);
Assert(entry->txn->nentries_mem);
/* txn stays the same */
@@ -1477,6 +1650,8 @@ ReorderBufferIterTXNNext(ReorderBuffer *rb, ReorderBufferIterTXNState *state)
/*
* Deallocate the iterator
+ *
+ * XXX No need to prepare internal function because it does not refer txn.
*/
static void
ReorderBufferIterTXNFinish(ReorderBuffer *rb,
@@ -1558,6 +1733,16 @@ ReorderBufferCleanupTXN(ReorderBuffer *rb, ReorderBufferTXN *txn)
/* Update the memory counter */
ReorderBufferChangeMemoryUpdate(rb, NULL, txn, false, mem_freed);
+ /* cleanup inval queue */
+ dlist_foreach_modify(iter, &txn->distributed_inval)
+ {
+ ReorderBufferChange *change;
+
+ change = dlist_container(ReorderBufferChange, node, iter.cur);
+
+ ReorderBufferReturnChange(rb, change, false);
+ }
+
/*
* Cleanup the tuplecids we stored for decoding catalog snapshot access.
* They are always stored in the toplevel transaction.
@@ -2575,6 +2760,7 @@ ReorderBufferProcessTXN(ReorderBuffer *rb, ReorderBufferTXN *txn,
/* make sure there's no cache pollution */
ReorderBufferExecuteInvalidations(txn->ninvalidations, txn->invalidations);
+ ReorderBufferExecuteInvalidationsInQueue(rb, txn);
if (using_subtxn)
RollbackAndReleaseCurrentSubTransaction();
@@ -2623,6 +2809,9 @@ ReorderBufferProcessTXN(ReorderBuffer *rb, ReorderBufferTXN *txn,
ReorderBufferExecuteInvalidations(txn->ninvalidations,
txn->invalidations);
+ /* XXX Should this put in the concurrent_abort case? */
+ ReorderBufferExecuteInvalidationsInQueue(rb, txn);
+
if (using_subtxn)
RollbackAndReleaseCurrentSubTransaction();
@@ -3329,6 +3518,21 @@ void
ReorderBufferAddInvalidations(ReorderBuffer *rb, TransactionId xid,
XLogRecPtr lsn, Size nmsgs,
SharedInvalidationMessage *msgs)
+{
+ ReorderBufferAddInvalidationsExtended(rb, xid, lsn, nmsgs, msgs, true);
+}
+
+/*
+ * An extended workhorse version of ReorderBufferAddInvalidations(). This can
+ * control whether invalidation messages can be accumulated in reorderbuffer
+ * transactions, which would be distributed to all concurrent transactions at
+ * commit.
+ */
+void
+ReorderBufferAddInvalidationsExtended(ReorderBuffer *rb, TransactionId xid,
+ XLogRecPtr lsn, Size nmsgs,
+ SharedInvalidationMessage *msgs,
+ bool needs_distribute)
{
ReorderBufferTXN *txn;
MemoryContext oldcontext;
@@ -3340,31 +3544,34 @@ ReorderBufferAddInvalidations(ReorderBuffer *rb, TransactionId xid,
/*
* Collect all the invalidations under the top transaction, if available,
- * so that we can execute them all together. See comments atop this
- * function.
+ * so that we can execute them all together. See comments atop
+ * ReorderBufferAddInvalidations.
*/
txn = rbtxn_get_toptxn(txn);
Assert(nmsgs > 0);
- /* Accumulate invalidations. */
- if (txn->ninvalidations == 0)
+ if (needs_distribute)
{
- txn->ninvalidations = nmsgs;
- txn->invalidations = (SharedInvalidationMessage *)
- palloc(sizeof(SharedInvalidationMessage) * nmsgs);
- memcpy(txn->invalidations, msgs,
- sizeof(SharedInvalidationMessage) * nmsgs);
- }
- else
- {
- txn->invalidations = (SharedInvalidationMessage *)
- repalloc(txn->invalidations, sizeof(SharedInvalidationMessage) *
- (txn->ninvalidations + nmsgs));
+ /* Accumulate invalidations. */
+ if (txn->ninvalidations == 0)
+ {
+ txn->ninvalidations = nmsgs;
+ txn->invalidations = (SharedInvalidationMessage *)
+ palloc(sizeof(SharedInvalidationMessage) * nmsgs);
+ memcpy(txn->invalidations, msgs,
+ sizeof(SharedInvalidationMessage) * nmsgs);
+ }
+ else
+ {
+ txn->invalidations = (SharedInvalidationMessage *)
+ repalloc(txn->invalidations,
+ sizeof(SharedInvalidationMessage) * (txn->ninvalidations + nmsgs));
- memcpy(txn->invalidations + txn->ninvalidations, msgs,
- nmsgs * sizeof(SharedInvalidationMessage));
- txn->ninvalidations += nmsgs;
+ memcpy(txn->invalidations + txn->ninvalidations, msgs,
+ nmsgs * sizeof(SharedInvalidationMessage));
+ txn->ninvalidations += nmsgs;
+ }
}
change = ReorderBufferGetChange(rb);
@@ -3377,6 +3584,28 @@ ReorderBufferAddInvalidations(ReorderBuffer *rb, TransactionId xid,
ReorderBufferQueueChange(rb, xid, lsn, change, false);
+ /*
+ * If the message is distributed from other transactions, also add to
+ * another queue.
+ *
+ * XXX: IIUC This must be done only to the toptxns, but is it right?
+ */
+ if (!needs_distribute && !TransactionIdIsValid(txn->toplevel_xid))
+ {
+ ReorderBufferChange *inval_change;
+
+ /* Duplicate the inval change to queue it */
+ inval_change = ReorderBufferGetChange(rb);
+ inval_change->action = REORDER_BUFFER_CHANGE_INVALIDATION;
+ inval_change->data.inval.ninvalidations = nmsgs;
+ inval_change->data.inval.invalidations = (SharedInvalidationMessage *)
+ palloc(sizeof(SharedInvalidationMessage) * nmsgs);
+ memcpy(inval_change->data.inval.invalidations, msgs,
+ sizeof(SharedInvalidationMessage) * nmsgs);
+
+ ReorderBufferQueueInval(rb, xid, lsn, inval_change);
+ }
+
MemoryContextSwitchTo(oldcontext);
}
diff --git a/src/backend/replication/logical/snapbuild.c b/src/backend/replication/logical/snapbuild.c
index 110e0b0a044..a8225c1d905 100644
--- a/src/backend/replication/logical/snapbuild.c
+++ b/src/backend/replication/logical/snapbuild.c
@@ -945,8 +945,9 @@ SnapBuildDistributeSnapshotAndInval(SnapBuild *builder, XLogRecPtr lsn, Transact
{
Assert(msgs != NULL);
- ReorderBufferAddInvalidations(builder->reorder, txn->xid, lsn,
- ninvalidations, msgs);
+ ReorderBufferAddInvalidationsExtended(builder->reorder, txn->xid, lsn,
+ ninvalidations, msgs,
+ false);
}
}
}
diff --git a/src/include/replication/reorderbuffer.h b/src/include/replication/reorderbuffer.h
index 4c56f219fd8..7bfd199c4b7 100644
--- a/src/include/replication/reorderbuffer.h
+++ b/src/include/replication/reorderbuffer.h
@@ -422,6 +422,9 @@ typedef struct ReorderBufferTXN
* Private data pointer of the output plugin.
*/
void *output_plugin_private;
+
+ dlist_head distributed_inval;
+ uint64 nentries_distr;
} ReorderBufferTXN;
/* so we can define the callbacks used inside struct ReorderBuffer itself */
@@ -709,6 +712,9 @@ extern void ReorderBufferAddNewTupleCids(ReorderBuffer *rb, TransactionId xid,
CommandId cmin, CommandId cmax, CommandId combocid);
extern void ReorderBufferAddInvalidations(ReorderBuffer *rb, TransactionId xid, XLogRecPtr lsn,
Size nmsgs, SharedInvalidationMessage *msgs);
+extern void ReorderBufferAddInvalidationsExtended(ReorderBuffer *rb, TransactionId xid, XLogRecPtr lsn,
+ Size nmsgs, SharedInvalidationMessage *msgs,
+ bool needs_distribute);
extern void ReorderBufferImmediateInvalidation(ReorderBuffer *rb, uint32 ninvalidations,
SharedInvalidationMessage *invalidations);
extern void ReorderBufferProcessXid(ReorderBuffer *rb, TransactionId xid, XLogRecPtr lsn);
--
2.47.1
[application/octet-stream] v4-master-0001-Avoid-distributing-invalidation-messages-s.patch (21.2K, ../../OSCPR01MB149667B316377CE0615E15138F567A@OSCPR01MB14966.jpnprd01.prod.outlook.com/3-v4-master-0001-Avoid-distributing-invalidation-messages-s.patch)
download | inline diff:
From f0eec7e9efcc995a95052af9fd230f7e23706efb Mon Sep 17 00:00:00 2001
From: Hayato Kuroda <kuroda.hayato@fujitsu.com>
Date: Wed, 21 May 2025 18:02:36 +0900
Subject: [PATCH v4-master] Avoid distributing invalidation messages several
times
---
.../replication/logical/reorderbuffer.c | 367 ++++++++++++++----
src/backend/replication/logical/snapbuild.c | 5 +-
src/include/replication/reorderbuffer.h | 6 +
3 files changed, 307 insertions(+), 71 deletions(-)
diff --git a/src/backend/replication/logical/reorderbuffer.c b/src/backend/replication/logical/reorderbuffer.c
index 67655111875..212ff4e0466 100644
--- a/src/backend/replication/logical/reorderbuffer.c
+++ b/src/backend/replication/logical/reorderbuffer.c
@@ -240,11 +240,30 @@ static void AssertTXNLsnOrder(ReorderBuffer *rb);
*/
static void ReorderBufferIterTXNInit(ReorderBuffer *rb, ReorderBufferTXN *txn,
ReorderBufferIterTXNState *volatile *iter_state);
+static void ReorderBufferIterTXNInitForInval(ReorderBuffer *rb,
+ ReorderBufferTXN *txn,
+ ReorderBufferIterTXNState *volatile *iter_state);
+static void ReorderBufferIterTXNInitInternal(ReorderBuffer *rb,
+ ReorderBufferTXN *txn,
+ ReorderBufferIterTXNState *volatile *iter_state,
+ bool iterate_inval);
+
static ReorderBufferChange *ReorderBufferIterTXNNext(ReorderBuffer *rb, ReorderBufferIterTXNState *state);
+static ReorderBufferChange *ReorderBufferIterTXNNextForInval(ReorderBuffer *rb, ReorderBufferIterTXNState *state);
+static ReorderBufferChange *ReorderBufferIterTXNNextInternal(ReorderBuffer *rb, ReorderBufferIterTXNState *state,
+ bool iterate_inval);
static void ReorderBufferIterTXNFinish(ReorderBuffer *rb,
ReorderBufferIterTXNState *state);
static void ReorderBufferExecuteInvalidations(uint32 nmsgs, SharedInvalidationMessage *msgs);
+static void ReorderBufferQueueChangeInternal(ReorderBuffer *rb, TransactionId xid, XLogRecPtr lsn,
+ ReorderBufferChange *change, bool toast_insert,
+ bool for_inval);
+static void ReorderBufferExecuteInvalidationsInQueue(ReorderBuffer *rb, ReorderBufferTXN *txn);
+
+static void ReorderBufferQueueInval(ReorderBuffer *rb, TransactionId xid,
+ XLogRecPtr lsn, ReorderBufferChange *change);
+
/*
* ---------------------------------------
* Disk serialization support functions
@@ -431,6 +450,7 @@ ReorderBufferAllocTXN(ReorderBuffer *rb)
dlist_init(&txn->changes);
dlist_init(&txn->tuplecids);
dlist_init(&txn->subtxns);
+ dlist_init(&txn->distributed_inval);
/* InvalidCommandId is not zero, so set it explicitly */
txn->command_id = InvalidCommandId;
@@ -790,6 +810,24 @@ ReorderBufferProcessPartialChange(ReorderBuffer *rb, ReorderBufferTXN *txn,
void
ReorderBufferQueueChange(ReorderBuffer *rb, TransactionId xid, XLogRecPtr lsn,
ReorderBufferChange *change, bool toast_insert)
+{
+ ReorderBufferQueueChangeInternal(rb, xid, lsn, change, toast_insert, false);
+}
+
+static void
+ReorderBufferQueueInval(ReorderBuffer *rb, TransactionId xid,
+ XLogRecPtr lsn, ReorderBufferChange *change)
+{
+ ReorderBufferQueueChangeInternal(rb, xid, lsn, change, false, true);
+}
+
+/*
+ * Actual workhorse for ReorderBufferQueueChange and ReorderBufferQueueInval
+ */
+static void
+ReorderBufferQueueChangeInternal(ReorderBuffer *rb, TransactionId xid,
+ XLogRecPtr lsn, ReorderBufferChange *change,
+ bool toast_insert, bool for_inval)
{
ReorderBufferTXN *txn;
@@ -800,7 +838,7 @@ ReorderBufferQueueChange(ReorderBuffer *rb, TransactionId xid, XLogRecPtr lsn,
* previous changes or by checking its CLOG, there is no point in
* collecting further changes for it.
*/
- if (rbtxn_is_aborted(txn))
+ if (!for_inval && rbtxn_is_aborted(txn))
{
/*
* We don't need to update memory accounting for this change as we
@@ -828,22 +866,84 @@ ReorderBufferQueueChange(ReorderBuffer *rb, TransactionId xid, XLogRecPtr lsn,
}
change->lsn = lsn;
- change->txn = txn;
+
+ /*
+ * Associate the change with current transaction only if we are not
+ * distributing invalidation messages. Associated one has shorter
+ * lifetime than the target transaction, which may cause dangling
+ * link issues.
+ */
+ change->txn = for_inval ? NULL : txn;
Assert(InvalidXLogRecPtr != lsn);
- dlist_push_tail(&txn->changes, &change->node);
- txn->nentries++;
- txn->nentries_mem++;
- /* update memory accounting information */
- ReorderBufferChangeMemoryUpdate(rb, change, NULL, true,
- ReorderBufferChangeSize(change));
+ if (for_inval)
+ {
+ dlist_push_tail(&txn->distributed_inval, &change->node);
+ txn->nentries_distr++;
+ }
+ else
+ {
+ dlist_push_tail(&txn->changes, &change->node);
+ txn->nentries++;
+ txn->nentries_mem++;
+
+ /*
+ * For normal changes, we must update memory usage and process pertial
+ * changes. This is not needed for distributed ones, because they
+ * won't be added to the total amount of memory.
+ */
+
+ /* update memory accounting information */
+ ReorderBufferChangeMemoryUpdate(rb, change, NULL, true,
+ ReorderBufferChangeSize(change));
- /* process partial change */
- ReorderBufferProcessPartialChange(rb, txn, change, toast_insert);
+ /* process partial change */
+ ReorderBufferProcessPartialChange(rb, txn, change, toast_insert);
- /* check the memory limits and evict something if needed */
- ReorderBufferCheckMemoryLimit(rb);
+ /* check the memory limits and evict something if needed */
+ ReorderBufferCheckMemoryLimit(rb);
+ }
+}
+
+static void
+ReorderBufferExecuteInvalidationsInQueue(ReorderBuffer *rb, ReorderBufferTXN *txn)
+{
+ ReorderBufferChange *change;
+ ReorderBufferIterTXNState *volatile iterstate = NULL;
+
+ ReorderBufferIterTXNInitForInval(rb, txn, &iterstate);
+ while ((change = ReorderBufferIterTXNNextForInval(rb, iterstate)) != NULL)
+ {
+ /* XXX do we have to do something here? */
+
+ switch (change->action)
+ {
+ /* Only interested in invalidation */
+ case REORDER_BUFFER_CHANGE_INVALIDATION:
+ /* Execute the invalidation messages locally */
+ ReorderBufferExecuteInvalidations(change->data.inval.ninvalidations,
+ change->data.inval.invalidations);
+ break;
+
+ /* Skip other changes because the transaction was aborted */
+ case REORDER_BUFFER_CHANGE_INSERT:
+ case REORDER_BUFFER_CHANGE_UPDATE:
+ case REORDER_BUFFER_CHANGE_DELETE:
+ case REORDER_BUFFER_CHANGE_MESSAGE:
+ case REORDER_BUFFER_CHANGE_INTERNAL_SNAPSHOT:
+ case REORDER_BUFFER_CHANGE_INTERNAL_COMMAND_ID:
+ case REORDER_BUFFER_CHANGE_INTERNAL_TUPLECID:
+ case REORDER_BUFFER_CHANGE_INTERNAL_SPEC_INSERT:
+ case REORDER_BUFFER_CHANGE_INTERNAL_SPEC_CONFIRM:
+ case REORDER_BUFFER_CHANGE_INTERNAL_SPEC_ABORT:
+ case REORDER_BUFFER_CHANGE_TRUNCATE:
+ break;
+ }
+ }
+
+ /* clean up the iterator */
+ ReorderBufferIterTXNFinish(rb, iterstate);
}
/*
@@ -1264,36 +1364,61 @@ ReorderBufferIterCompare(Datum a, Datum b, void *arg)
static void
ReorderBufferIterTXNInit(ReorderBuffer *rb, ReorderBufferTXN *txn,
ReorderBufferIterTXNState *volatile *iter_state)
+{
+ ReorderBufferIterTXNInitInternal(rb, txn, iter_state, false);
+}
+
+static void
+ReorderBufferIterTXNInitForInval(ReorderBuffer *rb, ReorderBufferTXN *txn,
+ ReorderBufferIterTXNState *volatile *iter_state)
+{
+ ReorderBufferIterTXNInitInternal(rb, txn, iter_state, true);
+}
+
+/*
+ * Actual workhorse for ReorderBufferIterTXNInit and ReorderBufferIterTXNInitForInval
+ */
+static void
+ReorderBufferIterTXNInitInternal(ReorderBuffer *rb, ReorderBufferTXN *txn,
+ ReorderBufferIterTXNState *volatile *iter_state,
+ bool iterate_inval)
{
Size nr_txns = 0;
ReorderBufferIterTXNState *state;
dlist_iter cur_txn_i;
int32 off;
+ uint64 nentries;
*iter_state = NULL;
/* Check ordering of changes in the toplevel transaction. */
- AssertChangeLsnOrder(txn);
+ if (!iterate_inval)
+ AssertChangeLsnOrder(txn);
+
+ nentries = iterate_inval ? txn->nentries_distr : txn->nentries;
/*
* Calculate the size of our heap: one element for every transaction that
* contains changes. (Besides the transactions already in the reorder
* buffer, we count the one we were directly passed.)
*/
- if (txn->nentries > 0)
+ if (nentries > 0)
nr_txns++;
- dlist_foreach(cur_txn_i, &txn->subtxns)
+ if (!iterate_inval)
{
- ReorderBufferTXN *cur_txn;
+ dlist_foreach(cur_txn_i, &txn->subtxns)
+ {
+ ReorderBufferTXN *cur_txn;
- cur_txn = dlist_container(ReorderBufferTXN, node, cur_txn_i.cur);
+ cur_txn = dlist_container(ReorderBufferTXN, node, cur_txn_i.cur);
- /* Check ordering of changes in this subtransaction. */
- AssertChangeLsnOrder(cur_txn);
+ /* Check ordering of changes in this subtransaction. */
+ AssertChangeLsnOrder(cur_txn);
- if (cur_txn->nentries > 0)
- nr_txns++;
+ if (cur_txn->nentries > 0)
+ nr_txns++;
+ }
}
/* allocate iteration state */
@@ -1327,11 +1452,12 @@ ReorderBufferIterTXNInit(ReorderBuffer *rb, ReorderBufferTXN *txn,
off = 0;
/* add toplevel transaction if it contains changes */
- if (txn->nentries > 0)
+ if (nentries > 0)
{
ReorderBufferChange *cur_change;
+ dlist_head *head;
- if (rbtxn_is_serialized(txn))
+ if (!iterate_inval && rbtxn_is_serialized(txn))
{
/* serialize remaining changes */
ReorderBufferSerializeTXN(rb, txn);
@@ -1339,8 +1465,10 @@ ReorderBufferIterTXNInit(ReorderBuffer *rb, ReorderBufferTXN *txn,
&state->entries[off].segno);
}
+ head = iterate_inval ? &txn->distributed_inval : &txn->changes;
+
cur_change = dlist_head_element(ReorderBufferChange, node,
- &txn->changes);
+ head);
state->entries[off].lsn = cur_change->lsn;
state->entries[off].change = cur_change;
@@ -1349,34 +1477,42 @@ ReorderBufferIterTXNInit(ReorderBuffer *rb, ReorderBufferTXN *txn,
binaryheap_add_unordered(state->heap, Int32GetDatum(off++));
}
- /* add subtransactions if they contain changes */
- dlist_foreach(cur_txn_i, &txn->subtxns)
+ if (!iterate_inval)
{
- ReorderBufferTXN *cur_txn;
-
- cur_txn = dlist_container(ReorderBufferTXN, node, cur_txn_i.cur);
-
- if (cur_txn->nentries > 0)
+ /* add subtransactions if they contain changes */
+ dlist_foreach(cur_txn_i, &txn->subtxns)
{
- ReorderBufferChange *cur_change;
+ ReorderBufferTXN *cur_txn;
+
+ cur_txn = dlist_container(ReorderBufferTXN, node, cur_txn_i.cur);
- if (rbtxn_is_serialized(cur_txn))
+ if (cur_txn->nentries > 0)
{
- /* serialize remaining changes */
- ReorderBufferSerializeTXN(rb, cur_txn);
- ReorderBufferRestoreChanges(rb, cur_txn,
- &state->entries[off].file,
- &state->entries[off].segno);
- }
- cur_change = dlist_head_element(ReorderBufferChange, node,
- &cur_txn->changes);
+ ReorderBufferChange *cur_change;
+ dlist_head *head;
+
+ if (rbtxn_is_serialized(cur_txn))
+ {
+ /* serialize remaining changes */
+ ReorderBufferSerializeTXN(rb, cur_txn);
+ ReorderBufferRestoreChanges(rb, cur_txn,
+ &state->entries[off].file,
+ &state->entries[off].segno);
+ }
+
+ head = iterate_inval ? &cur_txn->distributed_inval : &cur_txn->changes;
+
+ cur_change = dlist_head_element(ReorderBufferChange, node,
+ head);
- state->entries[off].lsn = cur_change->lsn;
- state->entries[off].change = cur_change;
- state->entries[off].txn = cur_txn;
+ state->entries[off].lsn = cur_change->lsn;
+ state->entries[off].change = cur_change;
+ state->entries[off].txn = cur_txn;
- binaryheap_add_unordered(state->heap, Int32GetDatum(off++));
+ binaryheap_add_unordered(state->heap, Int32GetDatum(off++));
+ }
}
+
}
/* assemble a valid binary heap */
@@ -1391,10 +1527,29 @@ ReorderBufferIterTXNInit(ReorderBuffer *rb, ReorderBufferTXN *txn,
*/
static ReorderBufferChange *
ReorderBufferIterTXNNext(ReorderBuffer *rb, ReorderBufferIterTXNState *state)
+{
+ return ReorderBufferIterTXNNextInternal(rb, state, false);
+}
+
+static ReorderBufferChange *
+ReorderBufferIterTXNNextForInval(ReorderBuffer *rb, ReorderBufferIterTXNState *state)
+{
+ return ReorderBufferIterTXNNextInternal(rb, state, true);
+}
+
+/*
+ * Actual workhorse for ReorderBufferIterTXNNext and ReorderBufferIterTXNNextForInval
+ */
+static ReorderBufferChange *
+ReorderBufferIterTXNNextInternal(ReorderBuffer *rb, ReorderBufferIterTXNState *state,
+ bool iterate_inval)
{
ReorderBufferChange *change;
ReorderBufferIterTXNEntry *entry;
int32 off;
+ dlist_head *head;
+ uint64 nentries;
+ uint64 nentries_mem;
/* nothing there anymore */
if (state->heap->bh_size == 0)
@@ -1414,15 +1569,33 @@ ReorderBufferIterTXNNext(ReorderBuffer *rb, ReorderBufferIterTXNState *state)
change = entry->change;
+ if (iterate_inval)
+ {
+ head = &entry->txn->distributed_inval;
+ nentries = entry->txn->nentries_distr;
+
+ /*
+ * The inval queue won't be serialized, so the number of entries on the
+ * memory is same as whole entries.
+ */
+ nentries_mem = entry->txn->nentries_distr;
+ }
+ else
+ {
+ head = &entry->txn->changes;
+ nentries = entry->txn->nentries;
+ nentries_mem = entry->txn->nentries_mem;
+ }
+
/*
* update heap with information about which transaction has the next
* relevant change in LSN order
*/
/* there are in-memory changes */
- if (dlist_has_next(&entry->txn->changes, &entry->change->node))
+ if (dlist_has_next(head, &entry->change->node))
{
- dlist_node *next = dlist_next_node(&entry->txn->changes, &change->node);
+ dlist_node *next = dlist_next_node(head, &change->node);
ReorderBufferChange *next_change =
dlist_container(ReorderBufferChange, node, next);
@@ -1435,7 +1608,7 @@ ReorderBufferIterTXNNext(ReorderBuffer *rb, ReorderBufferIterTXNState *state)
}
/* try to load changes from disk */
- if (entry->txn->nentries != entry->txn->nentries_mem)
+ if (nentries != nentries_mem)
{
/*
* Ugly: restoring changes will reuse *Change records, thus delete the
@@ -1456,11 +1629,11 @@ ReorderBufferIterTXNNext(ReorderBuffer *rb, ReorderBufferIterTXNState *state)
/* successfully restored changes from disk */
ReorderBufferChange *next_change =
dlist_head_element(ReorderBufferChange, node,
- &entry->txn->changes);
+ head);
elog(DEBUG2, "restored %u/%u changes from disk",
- (uint32) entry->txn->nentries_mem,
- (uint32) entry->txn->nentries);
+ (uint32) nentries_mem,
+ (uint32) nentries);
Assert(entry->txn->nentries_mem);
/* txn stays the same */
@@ -1480,6 +1653,8 @@ ReorderBufferIterTXNNext(ReorderBuffer *rb, ReorderBufferIterTXNState *state)
/*
* Deallocate the iterator
+ *
+ * XXX No need to prepare internal function because it does not refer txn.
*/
static void
ReorderBufferIterTXNFinish(ReorderBuffer *rb,
@@ -1561,6 +1736,16 @@ ReorderBufferCleanupTXN(ReorderBuffer *rb, ReorderBufferTXN *txn)
/* Update the memory counter */
ReorderBufferChangeMemoryUpdate(rb, NULL, txn, false, mem_freed);
+ /* cleanup inval queue */
+ dlist_foreach_modify(iter, &txn->distributed_inval)
+ {
+ ReorderBufferChange *change;
+
+ change = dlist_container(ReorderBufferChange, node, iter.cur);
+
+ ReorderBufferFreeChange(rb, change, false);
+ }
+
/*
* Cleanup the tuplecids we stored for decoding catalog snapshot access.
* They are always stored in the toplevel transaction.
@@ -2662,6 +2847,7 @@ ReorderBufferProcessTXN(ReorderBuffer *rb, ReorderBufferTXN *txn,
/* make sure there's no cache pollution */
ReorderBufferExecuteInvalidations(txn->ninvalidations, txn->invalidations);
+ ReorderBufferExecuteInvalidationsInQueue(rb, txn);
if (using_subtxn)
RollbackAndReleaseCurrentSubTransaction();
@@ -2713,6 +2899,9 @@ ReorderBufferProcessTXN(ReorderBuffer *rb, ReorderBufferTXN *txn,
ReorderBufferExecuteInvalidations(txn->ninvalidations,
txn->invalidations);
+ /* XXX Should this put in the concurrent_abort case? */
+ ReorderBufferExecuteInvalidationsInQueue(rb, txn);
+
if (using_subtxn)
RollbackAndReleaseCurrentSubTransaction();
@@ -3438,6 +3627,21 @@ void
ReorderBufferAddInvalidations(ReorderBuffer *rb, TransactionId xid,
XLogRecPtr lsn, Size nmsgs,
SharedInvalidationMessage *msgs)
+{
+ ReorderBufferAddInvalidationsExtended(rb, xid, lsn, nmsgs, msgs, true);
+}
+
+/*
+ * An extended workhorse version of ReorderBufferAddInvalidations(). This can
+ * control whether invalidation messages can be accumulated in reorderbuffer
+ * transactions, which would be distributed to all concurrent transactions at
+ * commit.
+ */
+void
+ReorderBufferAddInvalidationsExtended(ReorderBuffer *rb, TransactionId xid,
+ XLogRecPtr lsn, Size nmsgs,
+ SharedInvalidationMessage *msgs,
+ bool needs_distribute)
{
ReorderBufferTXN *txn;
MemoryContext oldcontext;
@@ -3449,31 +3653,34 @@ ReorderBufferAddInvalidations(ReorderBuffer *rb, TransactionId xid,
/*
* Collect all the invalidations under the top transaction, if available,
- * so that we can execute them all together. See comments atop this
- * function.
+ * so that we can execute them all together. See comments atop
+ * ReorderBufferAddInvalidations.
*/
txn = rbtxn_get_toptxn(txn);
Assert(nmsgs > 0);
- /* Accumulate invalidations. */
- if (txn->ninvalidations == 0)
+ if (needs_distribute)
{
- txn->ninvalidations = nmsgs;
- txn->invalidations = (SharedInvalidationMessage *)
- palloc(sizeof(SharedInvalidationMessage) * nmsgs);
- memcpy(txn->invalidations, msgs,
- sizeof(SharedInvalidationMessage) * nmsgs);
- }
- else
- {
- txn->invalidations = (SharedInvalidationMessage *)
- repalloc(txn->invalidations, sizeof(SharedInvalidationMessage) *
- (txn->ninvalidations + nmsgs));
+ /* Accumulate invalidations. */
+ if (txn->ninvalidations == 0)
+ {
+ txn->ninvalidations = nmsgs;
+ txn->invalidations = (SharedInvalidationMessage *)
+ palloc(sizeof(SharedInvalidationMessage) * nmsgs);
+ memcpy(txn->invalidations, msgs,
+ sizeof(SharedInvalidationMessage) * nmsgs);
+ }
+ else
+ {
+ txn->invalidations = (SharedInvalidationMessage *)
+ repalloc(txn->invalidations,
+ sizeof(SharedInvalidationMessage) * (txn->ninvalidations + nmsgs));
- memcpy(txn->invalidations + txn->ninvalidations, msgs,
- nmsgs * sizeof(SharedInvalidationMessage));
- txn->ninvalidations += nmsgs;
+ memcpy(txn->invalidations + txn->ninvalidations, msgs,
+ nmsgs * sizeof(SharedInvalidationMessage));
+ txn->ninvalidations += nmsgs;
+ }
}
change = ReorderBufferAllocChange(rb);
@@ -3486,6 +3693,28 @@ ReorderBufferAddInvalidations(ReorderBuffer *rb, TransactionId xid,
ReorderBufferQueueChange(rb, xid, lsn, change, false);
+ /*
+ * If the message is distributed from other transactions, also add to
+ * another queue.
+ *
+ * XXX: IIUC This must be done only to the toptxns, but is it right?
+ */
+ if (!needs_distribute && !TransactionIdIsValid(txn->toplevel_xid))
+ {
+ ReorderBufferChange *inval_change;
+
+ /* Duplicate the inval change to queue it */
+ inval_change = ReorderBufferAllocChange(rb);
+ inval_change->action = REORDER_BUFFER_CHANGE_INVALIDATION;
+ inval_change->data.inval.ninvalidations = nmsgs;
+ inval_change->data.inval.invalidations = (SharedInvalidationMessage *)
+ palloc(sizeof(SharedInvalidationMessage) * nmsgs);
+ memcpy(inval_change->data.inval.invalidations, msgs,
+ sizeof(SharedInvalidationMessage) * nmsgs);
+
+ ReorderBufferQueueInval(rb, xid, lsn, inval_change);
+ }
+
MemoryContextSwitchTo(oldcontext);
}
diff --git a/src/backend/replication/logical/snapbuild.c b/src/backend/replication/logical/snapbuild.c
index 0d7bddbe4ed..ceeabc6f420 100644
--- a/src/backend/replication/logical/snapbuild.c
+++ b/src/backend/replication/logical/snapbuild.c
@@ -807,8 +807,9 @@ SnapBuildDistributeSnapshotAndInval(SnapBuild *builder, XLogRecPtr lsn, Transact
{
Assert(msgs != NULL);
- ReorderBufferAddInvalidations(builder->reorder, txn->xid, lsn,
- ninvalidations, msgs);
+ ReorderBufferAddInvalidationsExtended(builder->reorder, txn->xid, lsn,
+ ninvalidations, msgs,
+ false);
}
}
}
diff --git a/src/include/replication/reorderbuffer.h b/src/include/replication/reorderbuffer.h
index 24e88c409ba..51969e41db2 100644
--- a/src/include/replication/reorderbuffer.h
+++ b/src/include/replication/reorderbuffer.h
@@ -452,6 +452,9 @@ typedef struct ReorderBufferTXN
* Private data pointer of the output plugin.
*/
void *output_plugin_private;
+
+ dlist_head distributed_inval;
+ uint64 nentries_distr;
} ReorderBufferTXN;
/* so we can define the callbacks used inside struct ReorderBuffer itself */
@@ -738,6 +741,9 @@ extern void ReorderBufferAddNewTupleCids(ReorderBuffer *rb, TransactionId xid,
CommandId cmin, CommandId cmax, CommandId combocid);
extern void ReorderBufferAddInvalidations(ReorderBuffer *rb, TransactionId xid, XLogRecPtr lsn,
Size nmsgs, SharedInvalidationMessage *msgs);
+extern void ReorderBufferAddInvalidationsExtended(ReorderBuffer *rb, TransactionId xid, XLogRecPtr lsn,
+ Size nmsgs, SharedInvalidationMessage *msgs,
+ bool needs_distribute);
extern void ReorderBufferImmediateInvalidation(ReorderBuffer *rb, uint32 ninvalidations,
SharedInvalidationMessage *invalidations);
extern void ReorderBufferProcessXid(ReorderBuffer *rb, TransactionId xid, XLogRecPtr lsn);
--
2.47.1
^ permalink raw reply [nested|flat] 94+ messages in thread
* Re: Logical replication 'invalid memory alloc request size 1585837200' after upgrading to 17.5
@ 2025-05-28 18:51 Masahiko Sawada <sawada.mshk@gmail.com>
parent: Amit Kapila <amit.kapila16@gmail.com>
1 sibling, 2 replies; 94+ messages in thread
From: Masahiko Sawada @ 2025-05-28 18:51 UTC (permalink / raw)
To: Amit Kapila <amit.kapila16@gmail.com>; +Cc: Hayato Kuroda (Fujitsu) <kuroda.hayato@fujitsu.com>; Duncan Sands <duncan.sands@deepbluecap.com>; pgsql-bugs@lists.postgresql.org <pgsql-bugs@lists.postgresql.org>
On Mon, May 26, 2025 at 4:19 AM Amit Kapila <amit.kapila16@gmail.com> wrote:
>
> On Mon, May 26, 2025 at 2:52 PM Hayato Kuroda (Fujitsu)
> <kuroda.hayato@fujitsu.com> wrote:
> >
> > > If the above hypothesis is true, we need to consider another idea so
> > > that we can execute invalidation messages in both cases.
> >
> > The straightforward fix is to check the change queue as well when the transaction
> > has invalidation messages. 0003 implemented that. One downside is that traversing
> > changes can affect performance. Currently we iterates all of changes even a
> > single REORDER_BUFFER_CHANGE_INVALIDATION. I cannot find better solutions for now.
> >
>
> It can impact the performance for large transactions with fewer
> invalidations, especially the ones which has spilled changes because
> it needs to traverse the entire list of changes again at the end.
What if we remember all executed REORDER_BUFFER_CHANGE_INVALIDATION in
a queue while replaying the transaction so that we can execute them at
the end in a non-error path, instead of re-traversing the entire list
of changes to execute the inval messages? As for concurrent abort
paths, probably we can consider re-traversing the entire list,
unconditionally invalidating all caches (using
InvalidateSystemCaches()), or somehow traversing the list of changes
only when there might be any REORDER_BUFFER_CHANGE_INVALIDATION in the
rest of changes?
Regards,
--
Masahiko Sawada
Amazon Web Services: https://aws.amazon.com
^ permalink raw reply [nested|flat] 94+ messages in thread
* Re: Logical replication 'invalid memory alloc request size 1585837200' after upgrading to 17.5
@ 2025-05-29 05:20 Amit Kapila <amit.kapila16@gmail.com>
parent: Masahiko Sawada <sawada.mshk@gmail.com>
1 sibling, 2 replies; 94+ messages in thread
From: Amit Kapila @ 2025-05-29 05:20 UTC (permalink / raw)
To: Masahiko Sawada <sawada.mshk@gmail.com>; +Cc: Hayato Kuroda (Fujitsu) <kuroda.hayato@fujitsu.com>; Duncan Sands <duncan.sands@deepbluecap.com>; pgsql-bugs@lists.postgresql.org <pgsql-bugs@lists.postgresql.org>
On Thu, May 29, 2025 at 12:22 AM Masahiko Sawada <sawada.mshk@gmail.com> wrote:
>
> On Mon, May 26, 2025 at 4:19 AM Amit Kapila <amit.kapila16@gmail.com> wrote:
> >
> > On Mon, May 26, 2025 at 2:52 PM Hayato Kuroda (Fujitsu)
> > <kuroda.hayato@fujitsu.com> wrote:
> > >
> > > > If the above hypothesis is true, we need to consider another idea so
> > > > that we can execute invalidation messages in both cases.
> > >
> > > The straightforward fix is to check the change queue as well when the transaction
> > > has invalidation messages. 0003 implemented that. One downside is that traversing
> > > changes can affect performance. Currently we iterates all of changes even a
> > > single REORDER_BUFFER_CHANGE_INVALIDATION. I cannot find better solutions for now.
> > >
> >
> > It can impact the performance for large transactions with fewer
> > invalidations, especially the ones which has spilled changes because
> > it needs to traverse the entire list of changes again at the end.
>
> What if we remember all executed REORDER_BUFFER_CHANGE_INVALIDATION in
> a queue while replaying the transaction so that we can execute them at
> the end in a non-error path, instead of re-traversing the entire list
> of changes to execute the inval messages?
>
The current proposed patch (v4) is also traversing only the required
inval messages, as it has maintained a separate queue for that. So,
what will be the advantage of forming such a queue during the
processing of changes? Are you imagining a local instead of a queue at
ReorderBufferTXN level? I feel we still need at ReorderBufferTXN level
to ensure that we can execute those changes across streaming blocks,
otherwise, the cleanup of such a local queue would be tricky and add
to maintenance effort.
One disadvantage of the approach you suggest is that the changes in
the new queue won't be accounted for in logical_decoding_work_mem
computation, which can be done in the proposed approach, although the
patch hasn't implemented it as of now.
A few comments on v4:
===================
1.
+static void
+ReorderBufferExecuteInvalidationsInQueue(ReorderBuffer *rb,
ReorderBufferTXN *txn)
+{
...
...
+ /* Skip other changes because the transaction was aborted */
+ case REORDER_BUFFER_CHANGE_INSERT:
+ case REORDER_BUFFER_CHANGE_UPDATE:
+ case REORDER_BUFFER_CHANGE_DELETE:
+ case REORDER_BUFFER_CHANGE_MESSAGE:
+ case REORDER_BUFFER_CHANGE_INTERNAL_SNAPSHOT:
+ case REORDER_BUFFER_CHANGE_INTERNAL_COMMAND_ID:
+ case REORDER_BUFFER_CHANGE_INTERNAL_TUPLECID:
+ case REORDER_BUFFER_CHANGE_INTERNAL_SPEC_INSERT:
+ case REORDER_BUFFER_CHANGE_INTERNAL_SPEC_CONFIRM:
+ case REORDER_BUFFER_CHANGE_INTERNAL_SPEC_ABORT:
+ case REORDER_BUFFER_CHANGE_TRUNCATE:
+ break;
This cases should never happen, so we should have either an Assert or
an elog here.
2.
+void
+ReorderBufferAddInvalidationsExtended(ReorderBuffer *rb, TransactionId xid,
+ XLogRecPtr lsn, Size nmsgs,
+ SharedInvalidationMessage *msgs,
+ bool needs_distribute)
{
...
+ * XXX: IIUC This must be done only to the toptxns, but is it right?
+ */
+ if (!needs_distribute && !TransactionIdIsValid(txn->toplevel_xid))
+ {
+ ReorderBufferChange *inval_change;
+
+ /* Duplicate the inval change to queue it */
+ inval_change = ReorderBufferAllocChange(rb);
+ inval_change->action = REORDER_BUFFER_CHANGE_INVALIDATION;
The name needs_distribute seems confusing because when this function
is invoked from SnapBuildDistributeSnapshotAndInval(), the parameter
is passed as false; it should be true in that case, and the meaning of
this parameter in the function should be reversed.
--
With Regards,
Amit Kapila.
^ permalink raw reply [nested|flat] 94+ messages in thread
* Re: Logical replication 'invalid memory alloc request size 1585837200' after upgrading to 17.5
@ 2025-05-29 06:07 Masahiko Sawada <sawada.mshk@gmail.com>
parent: Amit Kapila <amit.kapila16@gmail.com>
1 sibling, 1 reply; 94+ messages in thread
From: Masahiko Sawada @ 2025-05-29 06:07 UTC (permalink / raw)
To: Amit Kapila <amit.kapila16@gmail.com>; +Cc: Hayato Kuroda (Fujitsu) <kuroda.hayato@fujitsu.com>; Duncan Sands <duncan.sands@deepbluecap.com>; pgsql-bugs@lists.postgresql.org <pgsql-bugs@lists.postgresql.org>
On Wed, May 28, 2025 at 10:20 PM Amit Kapila <amit.kapila16@gmail.com> wrote:
>
> On Thu, May 29, 2025 at 12:22 AM Masahiko Sawada <sawada.mshk@gmail.com> wrote:
> >
> > On Mon, May 26, 2025 at 4:19 AM Amit Kapila <amit.kapila16@gmail.com> wrote:
> > >
> > > On Mon, May 26, 2025 at 2:52 PM Hayato Kuroda (Fujitsu)
> > > <kuroda.hayato@fujitsu.com> wrote:
> > > >
> > > > > If the above hypothesis is true, we need to consider another idea so
> > > > > that we can execute invalidation messages in both cases.
> > > >
> > > > The straightforward fix is to check the change queue as well when the transaction
> > > > has invalidation messages. 0003 implemented that. One downside is that traversing
> > > > changes can affect performance. Currently we iterates all of changes even a
> > > > single REORDER_BUFFER_CHANGE_INVALIDATION. I cannot find better solutions for now.
> > > >
> > >
> > > It can impact the performance for large transactions with fewer
> > > invalidations, especially the ones which has spilled changes because
> > > it needs to traverse the entire list of changes again at the end.
> >
> > What if we remember all executed REORDER_BUFFER_CHANGE_INVALIDATION in
> > a queue while replaying the transaction so that we can execute them at
> > the end in a non-error path, instead of re-traversing the entire list
> > of changes to execute the inval messages?
> >
>
> The current proposed patch (v4) is also traversing only the required
> inval messages, as it has maintained a separate queue for that. So,
> what will be the advantage of forming such a queue during the
> processing of changes? Are you imagining a local instead of a queue at
> ReorderBufferTXN level? I feel we still need at ReorderBufferTXN level
> to ensure that we can execute those changes across streaming blocks,
> otherwise, the cleanup of such a local queue would be tricky and add
> to maintenance effort.
Hmm, right. It seems that we keep accumulating inval messages across
streaming blocks.
> One disadvantage of the approach you suggest is that the changes in
> the new queue won't be accounted for in logical_decoding_work_mem
> computation, which can be done in the proposed approach, although the
> patch hasn't implemented it as of now.
If we serialize the new queue to the disk, we would need to restore
them in PG_CATCH() block in order to execute all inval messages, which
is something that I'd like to avoid as it would involve many
operations that could end up in an error.
If each ReorderBufferTXN has only non-distributed inval messages in
txn->invalidation and distribute only txn->invalidations to other
transactions, the scope of influence of a single Inval Message is
limited to transactions that are being decoded at the same time. How
much is there chance the size of txn->invalidations reach 1GB? Given
the size of SharedInvalidationMessage is 16 bytes, we need about 67k
inval messages generated across concurrent transactions.
Regards,
--
Masahiko Sawada
Amazon Web Services: https://aws.amazon.com
^ permalink raw reply [nested|flat] 94+ messages in thread
* RE: Logical replication 'invalid memory alloc request size 1585837200' after upgrading to 17.5
@ 2025-05-29 06:10 Hayato Kuroda (Fujitsu) <kuroda.hayato@fujitsu.com>
parent: Masahiko Sawada <sawada.mshk@gmail.com>
1 sibling, 0 replies; 94+ messages in thread
From: Hayato Kuroda (Fujitsu) @ 2025-05-29 06:10 UTC (permalink / raw)
To: Masahiko Sawada <sawada.mshk@gmail.com>; Amit Kapila <amit.kapila16@gmail.com>; +Cc: Duncan Sands <duncan.sands@deepbluecap.com>; pgsql-bugs@lists.postgresql.org <pgsql-bugs@lists.postgresql.org>
Dear Sawada-san,
> What if we remember all executed REORDER_BUFFER_CHANGE_INVALIDATION
> in
> a queue while replaying the transaction so that we can execute them at
> the end in a non-error path, instead of re-traversing the entire list
> of changes to execute the inval messages?
I think this idea is similar with v4 patch [1]. It adds another queue which stores the inval
messages when they are being distributed, and the inval queue is consumed while committing.
> As for concurrent abort
> paths, probably we can consider re-traversing the entire list,
> unconditionally invalidating all caches (using
> InvalidateSystemCaches()), or somehow traversing the list of changes
> only when there might be any REORDER_BUFFER_CHANGE_INVALIDATION in
> the
> rest of changes?
In my v4 patch, it is just OK to consume all inval messages in another queue,
because all needed messages are stored before we try to process txn.
Based on that, I feel v4 seems bit simpler approach.
[1]: https://www.postgresql.org/message-id/OSCPR01MB149667B316377CE0615E15138F567A%40OSCPR01MB14966.jpnpr...
Best regards,
Hayato Kuroda
FUJITSU LIMITED
^ permalink raw reply [nested|flat] 94+ messages in thread
* Re: Logical replication 'invalid memory alloc request size 1585837200' after upgrading to 17.5
@ 2025-05-29 08:22 Amit Kapila <amit.kapila16@gmail.com>
parent: Masahiko Sawada <sawada.mshk@gmail.com>
0 siblings, 1 reply; 94+ messages in thread
From: Amit Kapila @ 2025-05-29 08:22 UTC (permalink / raw)
To: Masahiko Sawada <sawada.mshk@gmail.com>; +Cc: Hayato Kuroda (Fujitsu) <kuroda.hayato@fujitsu.com>; Duncan Sands <duncan.sands@deepbluecap.com>; pgsql-bugs@lists.postgresql.org <pgsql-bugs@lists.postgresql.org>
On Thu, May 29, 2025 at 11:37 AM Masahiko Sawada <sawada.mshk@gmail.com> wrote:
>
> On Wed, May 28, 2025 at 10:20 PM Amit Kapila <amit.kapila16@gmail.com> wrote:
> >
> > On Thu, May 29, 2025 at 12:22 AM Masahiko Sawada <sawada.mshk@gmail.com> wrote:
> > >
> > > On Mon, May 26, 2025 at 4:19 AM Amit Kapila <amit.kapila16@gmail.com> wrote:
> > > >
> > > > On Mon, May 26, 2025 at 2:52 PM Hayato Kuroda (Fujitsu)
> > > > <kuroda.hayato@fujitsu.com> wrote:
> > > > >
> > > > > > If the above hypothesis is true, we need to consider another idea so
> > > > > > that we can execute invalidation messages in both cases.
> > > > >
> > > > > The straightforward fix is to check the change queue as well when the transaction
> > > > > has invalidation messages. 0003 implemented that. One downside is that traversing
> > > > > changes can affect performance. Currently we iterates all of changes even a
> > > > > single REORDER_BUFFER_CHANGE_INVALIDATION. I cannot find better solutions for now.
> > > > >
> > > >
> > > > It can impact the performance for large transactions with fewer
> > > > invalidations, especially the ones which has spilled changes because
> > > > it needs to traverse the entire list of changes again at the end.
> > >
> > > What if we remember all executed REORDER_BUFFER_CHANGE_INVALIDATION in
> > > a queue while replaying the transaction so that we can execute them at
> > > the end in a non-error path, instead of re-traversing the entire list
> > > of changes to execute the inval messages?
> > >
> >
> > The current proposed patch (v4) is also traversing only the required
> > inval messages, as it has maintained a separate queue for that. So,
> > what will be the advantage of forming such a queue during the
> > processing of changes? Are you imagining a local instead of a queue at
> > ReorderBufferTXN level? I feel we still need at ReorderBufferTXN level
> > to ensure that we can execute those changes across streaming blocks,
> > otherwise, the cleanup of such a local queue would be tricky and add
> > to maintenance effort.
>
> Hmm, right. It seems that we keep accumulating inval messages across
> streaming blocks.
>
> > One disadvantage of the approach you suggest is that the changes in
> > the new queue won't be accounted for in logical_decoding_work_mem
> > computation, which can be done in the proposed approach, although the
> > patch hasn't implemented it as of now.
>
> If we serialize the new queue to the disk, we would need to restore
> them in PG_CATCH() block in order to execute all inval messages, which
> is something that I'd like to avoid as it would involve many
> operations that could end up in an error.
>
> If each ReorderBufferTXN has only non-distributed inval messages in
> txn->invalidation and distribute only txn->invalidations to other
> transactions, the scope of influence of a single Inval Message is
> limited to transactions that are being decoded at the same time. How
> much is there chance the size of txn->invalidations reach 1GB? Given
> the size of SharedInvalidationMessage is 16 bytes, we need about 67k
> inval messages generated across concurrent transactions.
>
I agree that chances are much lower than current if txn->invalidations
doesn't contain invalidations from other transactions, but it is not
clear what exactly you are trying to advocate by it. Are you trying to
advocate that we should maintain a member similar to txn->invalidation
(say txn->distributed_invals) instead of a queue?
--
With Regards,
Amit Kapila.
^ permalink raw reply [nested|flat] 94+ messages in thread
* RE: Logical replication 'invalid memory alloc request size 1585837200' after upgrading to 17.5
@ 2025-05-29 12:09 Hayato Kuroda (Fujitsu) <kuroda.hayato@fujitsu.com>
parent: Amit Kapila <amit.kapila16@gmail.com>
1 sibling, 0 replies; 94+ messages in thread
From: Hayato Kuroda (Fujitsu) @ 2025-05-29 12:09 UTC (permalink / raw)
To: Amit Kapila <amit.kapila16@gmail.com>; Masahiko Sawada <sawada.mshk@gmail.com>; +Cc: Duncan Sands <duncan.sands@deepbluecap.com>; pgsql-bugs@lists.postgresql.org <pgsql-bugs@lists.postgresql.org>
Dear Amit,
I created updated patch. I did some self-reviewing and updated.
> One disadvantage of the approach you suggest is that the changes in
> the new queue won't be accounted for in logical_decoding_work_mem
> computation, which can be done in the proposed approach, although the
> patch hasn't implemented it as of now.
This part is not implemented yet, because it is under discussion. Roughly considered,
we must consider several points - it might be complex:
1. What is the ordering of serialization? Should we prioritize spilling
invalidation messages, or normal changes?
2. What is the filename? Can we use same one as changes?
3. IIUC we shouldn't stream inval messages even when stream=on/parallel case.
Isn't it hacky?
> A few comments on v4:
> ===================
> 1.
> +static void
> +ReorderBufferExecuteInvalidationsInQueue(ReorderBuffer *rb,
> ReorderBufferTXN *txn)
> +{
> ...
> ...
> + /* Skip other changes because the transaction was aborted */
> + case REORDER_BUFFER_CHANGE_INSERT:
> + case REORDER_BUFFER_CHANGE_UPDATE:
> + case REORDER_BUFFER_CHANGE_DELETE:
> + case REORDER_BUFFER_CHANGE_MESSAGE:
> + case REORDER_BUFFER_CHANGE_INTERNAL_SNAPSHOT:
> + case REORDER_BUFFER_CHANGE_INTERNAL_COMMAND_ID:
> + case REORDER_BUFFER_CHANGE_INTERNAL_TUPLECID:
> + case REORDER_BUFFER_CHANGE_INTERNAL_SPEC_INSERT:
> + case REORDER_BUFFER_CHANGE_INTERNAL_SPEC_CONFIRM:
> + case REORDER_BUFFER_CHANGE_INTERNAL_SPEC_ABORT:
> + case REORDER_BUFFER_CHANGE_TRUNCATE:
> + break;
>
> This cases should never happen, so we should have either an Assert or
> an elog here.
Fixed. Switch was removed and Assert() was added.
> 2.
> +void
> +ReorderBufferAddInvalidationsExtended(ReorderBuffer *rb, TransactionId xid,
> + XLogRecPtr lsn, Size nmsgs,
> + SharedInvalidationMessage *msgs,
> + bool needs_distribute)
> {
> ...
> + * XXX: IIUC This must be done only to the toptxns, but is it right?
> + */
> + if (!needs_distribute && !TransactionIdIsValid(txn->toplevel_xid))
> + {
> + ReorderBufferChange *inval_change;
> +
> + /* Duplicate the inval change to queue it */
> + inval_change = ReorderBufferAllocChange(rb);
> + inval_change->action = REORDER_BUFFER_CHANGE_INVALIDATION;
>
> The name needs_distribute seems confusing because when this function
> is invoked from SnapBuildDistributeSnapshotAndInval(), the parameter
> is passed as false; it should be true in that case, and the meaning of
> this parameter in the function should be reversed.
Fixed.
Best regards,
Hayato Kuroda
FUJITSU LIMITED
Attachments:
[application/octet-stream] v5-master-0001-Fix-invalid-memory-alloc-request-size-issu.patch (22.2K, ../../OSCPR01MB149668136257B13A675CDD1BBF566A@OSCPR01MB14966.jpnprd01.prod.outlook.com/2-v5-master-0001-Fix-invalid-memory-alloc-request-size-issu.patch)
download | inline diff:
From 8683d09e93830050f3dc70cbf32ef1b574139cde Mon Sep 17 00:00:00 2001
From: Hayato Kuroda <kuroda.hayato@fujitsu.com>
Date: Wed, 21 May 2025 18:02:36 +0900
Subject: [PATCH v5-master] Fix 'invalid memory alloc request size' issue while
doing logical decoding
4909b38af0 distributes invalidation messages for transactions to all concurrent
in-progress transactions. Since the source of invalidation messages was not
identified, the number of total messages could be increased exponentially, which
causes memory allocation failure.
To address the issue, we introduce a new queue containing distributed
invalidation messages. At the end of the transaction, only transactions in the
pre-existing list can be distributed. Messages in the queue are consumed at the
end of processing changes in ReorderBufferProcessTXN().
Currently, the new queue is not included in the total size of transactions.
BUG #18938
---
.../replication/logical/reorderbuffer.c | 367 ++++++++++++++----
src/backend/replication/logical/snapbuild.c | 5 +-
src/include/replication/reorderbuffer.h | 16 +
3 files changed, 317 insertions(+), 71 deletions(-)
diff --git a/src/backend/replication/logical/reorderbuffer.c b/src/backend/replication/logical/reorderbuffer.c
index 67655111875..7d7699993f9 100644
--- a/src/backend/replication/logical/reorderbuffer.c
+++ b/src/backend/replication/logical/reorderbuffer.c
@@ -240,11 +240,30 @@ static void AssertTXNLsnOrder(ReorderBuffer *rb);
*/
static void ReorderBufferIterTXNInit(ReorderBuffer *rb, ReorderBufferTXN *txn,
ReorderBufferIterTXNState *volatile *iter_state);
+static void ReorderBufferIterTXNInitForInval(ReorderBuffer *rb,
+ ReorderBufferTXN *txn,
+ ReorderBufferIterTXNState *volatile *iter_state);
+static void ReorderBufferIterTXNInitInternal(ReorderBuffer *rb,
+ ReorderBufferTXN *txn,
+ ReorderBufferIterTXNState *volatile *iter_state,
+ bool iterate_inval);
+
static ReorderBufferChange *ReorderBufferIterTXNNext(ReorderBuffer *rb, ReorderBufferIterTXNState *state);
+static ReorderBufferChange *ReorderBufferIterTXNNextForInval(ReorderBuffer *rb, ReorderBufferIterTXNState *state);
+static ReorderBufferChange *ReorderBufferIterTXNNextInternal(ReorderBuffer *rb, ReorderBufferIterTXNState *state,
+ bool iterate_inval);
static void ReorderBufferIterTXNFinish(ReorderBuffer *rb,
ReorderBufferIterTXNState *state);
static void ReorderBufferExecuteInvalidations(uint32 nmsgs, SharedInvalidationMessage *msgs);
+static void ReorderBufferQueueChangeInternal(ReorderBuffer *rb, TransactionId xid, XLogRecPtr lsn,
+ ReorderBufferChange *change, bool toast_insert,
+ bool for_inval);
+static void ReorderBufferExecuteInvalidationsInQueue(ReorderBuffer *rb, ReorderBufferTXN *txn);
+
+static void ReorderBufferQueueInval(ReorderBuffer *rb, TransactionId xid,
+ XLogRecPtr lsn, ReorderBufferChange *change);
+
/*
* ---------------------------------------
* Disk serialization support functions
@@ -431,6 +450,7 @@ ReorderBufferAllocTXN(ReorderBuffer *rb)
dlist_init(&txn->changes);
dlist_init(&txn->tuplecids);
dlist_init(&txn->subtxns);
+ dlist_init(&txn->distributed_inval);
/* InvalidCommandId is not zero, so set it explicitly */
txn->command_id = InvalidCommandId;
@@ -790,6 +810,24 @@ ReorderBufferProcessPartialChange(ReorderBuffer *rb, ReorderBufferTXN *txn,
void
ReorderBufferQueueChange(ReorderBuffer *rb, TransactionId xid, XLogRecPtr lsn,
ReorderBufferChange *change, bool toast_insert)
+{
+ ReorderBufferQueueChangeInternal(rb, xid, lsn, change, toast_insert, false);
+}
+
+static void
+ReorderBufferQueueInval(ReorderBuffer *rb, TransactionId xid,
+ XLogRecPtr lsn, ReorderBufferChange *change)
+{
+ ReorderBufferQueueChangeInternal(rb, xid, lsn, change, false, true);
+}
+
+/*
+ * Actual workhorse for ReorderBufferQueueChange and ReorderBufferQueueInval
+ */
+static void
+ReorderBufferQueueChangeInternal(ReorderBuffer *rb, TransactionId xid,
+ XLogRecPtr lsn, ReorderBufferChange *change,
+ bool toast_insert, bool for_inval)
{
ReorderBufferTXN *txn;
@@ -799,8 +837,12 @@ ReorderBufferQueueChange(ReorderBuffer *rb, TransactionId xid, XLogRecPtr lsn,
* If we have detected that the transaction is aborted while streaming the
* previous changes or by checking its CLOG, there is no point in
* collecting further changes for it.
+ *
+ * If the target queue is the invalidation queue, however, all invalidation
+ * messages must be executed at the end of processing the transaction. This
+ * means all changes must be queued.
*/
- if (rbtxn_is_aborted(txn))
+ if (!for_inval && rbtxn_is_aborted(txn))
{
/*
* We don't need to update memory accounting for this change as we
@@ -828,22 +870,71 @@ ReorderBufferQueueChange(ReorderBuffer *rb, TransactionId xid, XLogRecPtr lsn,
}
change->lsn = lsn;
- change->txn = txn;
+
+ /*
+ * Associate the change with current transaction only if we are not
+ * distributing invalidation messages. Associated one has shorter lifetime
+ * than the target transaction, which may cause dangling link issues.
+ */
+ change->txn = for_inval ? NULL : txn;
Assert(InvalidXLogRecPtr != lsn);
- dlist_push_tail(&txn->changes, &change->node);
- txn->nentries++;
- txn->nentries_mem++;
- /* update memory accounting information */
- ReorderBufferChangeMemoryUpdate(rb, change, NULL, true,
- ReorderBufferChangeSize(change));
+ if (for_inval)
+ {
+ Assert(change->action == REORDER_BUFFER_CHANGE_INVALIDATION);
+
+ dlist_push_tail(&txn->distributed_inval, &change->node);
+ txn->nentries_distr++;
+ }
+ else
+ {
+ dlist_push_tail(&txn->changes, &change->node);
+ txn->nentries++;
+ txn->nentries_mem++;
+
+ /*
+ * For normal changes, we must update memory usage and process pertial
+ * changes. This is not needed for distributed ones, because they
+ * won't be added to the total amount of memory.
+ */
+
+ /* update memory accounting information */
+ ReorderBufferChangeMemoryUpdate(rb, change, NULL, true,
+ ReorderBufferChangeSize(change));
- /* process partial change */
- ReorderBufferProcessPartialChange(rb, txn, change, toast_insert);
+ /* process partial change */
+ ReorderBufferProcessPartialChange(rb, txn, change, toast_insert);
- /* check the memory limits and evict something if needed */
- ReorderBufferCheckMemoryLimit(rb);
+ /* check the memory limits and evict something if needed */
+ ReorderBufferCheckMemoryLimit(rb);
+ }
+}
+
+/*
+ * Traverse the invalication message queue and execute them
+ */
+static void
+ReorderBufferExecuteInvalidationsInQueue(ReorderBuffer *rb, ReorderBufferTXN *txn)
+{
+ ReorderBufferChange *change;
+ ReorderBufferIterTXNState *volatile iterstate = NULL;
+
+ ReorderBufferIterTXNInitForInval(rb, txn, &iterstate);
+ while ((change = ReorderBufferIterTXNNextForInval(rb, iterstate)) != NULL)
+ {
+ /* XXX do we have to do something here? */
+
+ /* We expect only invalidation messages here */
+ Assert(change->action == REORDER_BUFFER_CHANGE_INVALIDATION);
+
+ /* Execute the invalidation messages locally */
+ ReorderBufferExecuteInvalidations(change->data.inval.ninvalidations,
+ change->data.inval.invalidations);
+ }
+
+ /* clean up the iterator */
+ ReorderBufferIterTXNFinish(rb, iterstate);
}
/*
@@ -1264,36 +1355,66 @@ ReorderBufferIterCompare(Datum a, Datum b, void *arg)
static void
ReorderBufferIterTXNInit(ReorderBuffer *rb, ReorderBufferTXN *txn,
ReorderBufferIterTXNState *volatile *iter_state)
+{
+ ReorderBufferIterTXNInitInternal(rb, txn, iter_state, false);
+}
+
+static void
+ReorderBufferIterTXNInitForInval(ReorderBuffer *rb, ReorderBufferTXN *txn,
+ ReorderBufferIterTXNState *volatile *iter_state)
+{
+ ReorderBufferIterTXNInitInternal(rb, txn, iter_state, true);
+}
+
+/*
+ * Actual workhorse for ReorderBufferIterTXNInit and ReorderBufferIterTXNInitForInval
+ */
+static void
+ReorderBufferIterTXNInitInternal(ReorderBuffer *rb, ReorderBufferTXN *txn,
+ ReorderBufferIterTXNState *volatile *iter_state,
+ bool iterate_inval)
{
Size nr_txns = 0;
ReorderBufferIterTXNState *state;
dlist_iter cur_txn_i;
int32 off;
+ uint64 nentries;
*iter_state = NULL;
/* Check ordering of changes in the toplevel transaction. */
- AssertChangeLsnOrder(txn);
+ if (!iterate_inval)
+ AssertChangeLsnOrder(txn);
+
+ nentries = iterate_inval ? txn->nentries_distr : txn->nentries;
/*
* Calculate the size of our heap: one element for every transaction that
* contains changes. (Besides the transactions already in the reorder
* buffer, we count the one we were directly passed.)
*/
- if (txn->nentries > 0)
+ if (nentries > 0)
nr_txns++;
- dlist_foreach(cur_txn_i, &txn->subtxns)
+ /*
+ * Check all subtransactions whether they contains changes. Only
+ * toptranasctions can be queued to the invalidation queue, so no need to
+ * check for iterate_inval case.
+ */
+ if (!iterate_inval)
{
- ReorderBufferTXN *cur_txn;
+ dlist_foreach(cur_txn_i, &txn->subtxns)
+ {
+ ReorderBufferTXN *cur_txn;
- cur_txn = dlist_container(ReorderBufferTXN, node, cur_txn_i.cur);
+ cur_txn = dlist_container(ReorderBufferTXN, node, cur_txn_i.cur);
- /* Check ordering of changes in this subtransaction. */
- AssertChangeLsnOrder(cur_txn);
+ /* Check ordering of changes in this subtransaction. */
+ AssertChangeLsnOrder(cur_txn);
- if (cur_txn->nentries > 0)
- nr_txns++;
+ if (cur_txn->nentries > 0)
+ nr_txns++;
+ }
}
/* allocate iteration state */
@@ -1327,11 +1448,12 @@ ReorderBufferIterTXNInit(ReorderBuffer *rb, ReorderBufferTXN *txn,
off = 0;
/* add toplevel transaction if it contains changes */
- if (txn->nentries > 0)
+ if (nentries > 0)
{
ReorderBufferChange *cur_change;
+ dlist_head *head;
- if (rbtxn_is_serialized(txn))
+ if (!iterate_inval && rbtxn_is_serialized(txn))
{
/* serialize remaining changes */
ReorderBufferSerializeTXN(rb, txn);
@@ -1339,8 +1461,10 @@ ReorderBufferIterTXNInit(ReorderBuffer *rb, ReorderBufferTXN *txn,
&state->entries[off].segno);
}
+ head = iterate_inval ? &txn->distributed_inval : &txn->changes;
+
cur_change = dlist_head_element(ReorderBufferChange, node,
- &txn->changes);
+ head);
state->entries[off].lsn = cur_change->lsn;
state->entries[off].change = cur_change;
@@ -1349,34 +1473,46 @@ ReorderBufferIterTXNInit(ReorderBuffer *rb, ReorderBufferTXN *txn,
binaryheap_add_unordered(state->heap, Int32GetDatum(off++));
}
- /* add subtransactions if they contain changes */
- dlist_foreach(cur_txn_i, &txn->subtxns)
+ /*
+ * Add subtransactions if they contain changes. No need to do in
+ * iterate_inval case because inval queues it is only used for the
+ * toptransactions.
+ */
+ if (!iterate_inval)
{
- ReorderBufferTXN *cur_txn;
-
- cur_txn = dlist_container(ReorderBufferTXN, node, cur_txn_i.cur);
-
- if (cur_txn->nentries > 0)
+ dlist_foreach(cur_txn_i, &txn->subtxns)
{
- ReorderBufferChange *cur_change;
+ ReorderBufferTXN *cur_txn;
+
+ cur_txn = dlist_container(ReorderBufferTXN, node, cur_txn_i.cur);
- if (rbtxn_is_serialized(cur_txn))
+ if (cur_txn->nentries > 0)
{
- /* serialize remaining changes */
- ReorderBufferSerializeTXN(rb, cur_txn);
- ReorderBufferRestoreChanges(rb, cur_txn,
- &state->entries[off].file,
- &state->entries[off].segno);
- }
- cur_change = dlist_head_element(ReorderBufferChange, node,
- &cur_txn->changes);
+ ReorderBufferChange *cur_change;
+ dlist_head *head;
+
+ if (rbtxn_is_serialized(cur_txn))
+ {
+ /* serialize remaining changes */
+ ReorderBufferSerializeTXN(rb, cur_txn);
+ ReorderBufferRestoreChanges(rb, cur_txn,
+ &state->entries[off].file,
+ &state->entries[off].segno);
+ }
+
+ head = iterate_inval ? &cur_txn->distributed_inval : &cur_txn->changes;
+
+ cur_change = dlist_head_element(ReorderBufferChange, node,
+ head);
- state->entries[off].lsn = cur_change->lsn;
- state->entries[off].change = cur_change;
- state->entries[off].txn = cur_txn;
+ state->entries[off].lsn = cur_change->lsn;
+ state->entries[off].change = cur_change;
+ state->entries[off].txn = cur_txn;
- binaryheap_add_unordered(state->heap, Int32GetDatum(off++));
+ binaryheap_add_unordered(state->heap, Int32GetDatum(off++));
+ }
}
+
}
/* assemble a valid binary heap */
@@ -1391,10 +1527,29 @@ ReorderBufferIterTXNInit(ReorderBuffer *rb, ReorderBufferTXN *txn,
*/
static ReorderBufferChange *
ReorderBufferIterTXNNext(ReorderBuffer *rb, ReorderBufferIterTXNState *state)
+{
+ return ReorderBufferIterTXNNextInternal(rb, state, false);
+}
+
+static ReorderBufferChange *
+ReorderBufferIterTXNNextForInval(ReorderBuffer *rb, ReorderBufferIterTXNState *state)
+{
+ return ReorderBufferIterTXNNextInternal(rb, state, true);
+}
+
+/*
+ * Actual workhorse for ReorderBufferIterTXNNext and ReorderBufferIterTXNNextForInval
+ */
+static ReorderBufferChange *
+ReorderBufferIterTXNNextInternal(ReorderBuffer *rb, ReorderBufferIterTXNState *state,
+ bool iterate_inval)
{
ReorderBufferChange *change;
ReorderBufferIterTXNEntry *entry;
int32 off;
+ dlist_head *head;
+ uint64 nentries;
+ uint64 nentries_mem;
/* nothing there anymore */
if (state->heap->bh_size == 0)
@@ -1414,15 +1569,33 @@ ReorderBufferIterTXNNext(ReorderBuffer *rb, ReorderBufferIterTXNState *state)
change = entry->change;
+ if (iterate_inval)
+ {
+ head = &entry->txn->distributed_inval;
+ nentries = entry->txn->nentries_distr;
+
+ /*
+ * The inval queue won't be serialized, so the number of entries on
+ * the memory is same as whole entries.
+ */
+ nentries_mem = entry->txn->nentries_distr;
+ }
+ else
+ {
+ head = &entry->txn->changes;
+ nentries = entry->txn->nentries;
+ nentries_mem = entry->txn->nentries_mem;
+ }
+
/*
* update heap with information about which transaction has the next
* relevant change in LSN order
*/
/* there are in-memory changes */
- if (dlist_has_next(&entry->txn->changes, &entry->change->node))
+ if (dlist_has_next(head, &entry->change->node))
{
- dlist_node *next = dlist_next_node(&entry->txn->changes, &change->node);
+ dlist_node *next = dlist_next_node(head, &change->node);
ReorderBufferChange *next_change =
dlist_container(ReorderBufferChange, node, next);
@@ -1435,7 +1608,7 @@ ReorderBufferIterTXNNext(ReorderBuffer *rb, ReorderBufferIterTXNState *state)
}
/* try to load changes from disk */
- if (entry->txn->nentries != entry->txn->nentries_mem)
+ if (nentries != nentries_mem)
{
/*
* Ugly: restoring changes will reuse *Change records, thus delete the
@@ -1456,11 +1629,11 @@ ReorderBufferIterTXNNext(ReorderBuffer *rb, ReorderBufferIterTXNState *state)
/* successfully restored changes from disk */
ReorderBufferChange *next_change =
dlist_head_element(ReorderBufferChange, node,
- &entry->txn->changes);
+ head);
elog(DEBUG2, "restored %u/%u changes from disk",
- (uint32) entry->txn->nentries_mem,
- (uint32) entry->txn->nentries);
+ (uint32) nentries_mem,
+ (uint32) nentries);
Assert(entry->txn->nentries_mem);
/* txn stays the same */
@@ -1480,6 +1653,8 @@ ReorderBufferIterTXNNext(ReorderBuffer *rb, ReorderBufferIterTXNState *state)
/*
* Deallocate the iterator
+ *
+ * XXX No need to prepare internal function because it does not refer txn.
*/
static void
ReorderBufferIterTXNFinish(ReorderBuffer *rb,
@@ -1561,6 +1736,16 @@ ReorderBufferCleanupTXN(ReorderBuffer *rb, ReorderBufferTXN *txn)
/* Update the memory counter */
ReorderBufferChangeMemoryUpdate(rb, NULL, txn, false, mem_freed);
+ /* cleanup inval queue */
+ dlist_foreach_modify(iter, &txn->distributed_inval)
+ {
+ ReorderBufferChange *change;
+
+ change = dlist_container(ReorderBufferChange, node, iter.cur);
+
+ ReorderBufferFreeChange(rb, change, false);
+ }
+
/*
* Cleanup the tuplecids we stored for decoding catalog snapshot access.
* They are always stored in the toplevel transaction.
@@ -2662,6 +2847,7 @@ ReorderBufferProcessTXN(ReorderBuffer *rb, ReorderBufferTXN *txn,
/* make sure there's no cache pollution */
ReorderBufferExecuteInvalidations(txn->ninvalidations, txn->invalidations);
+ ReorderBufferExecuteInvalidationsInQueue(rb, txn);
if (using_subtxn)
RollbackAndReleaseCurrentSubTransaction();
@@ -2713,6 +2899,9 @@ ReorderBufferProcessTXN(ReorderBuffer *rb, ReorderBufferTXN *txn,
ReorderBufferExecuteInvalidations(txn->ninvalidations,
txn->invalidations);
+ /* XXX Should this put in the concurrent_abort case? */
+ ReorderBufferExecuteInvalidationsInQueue(rb, txn);
+
if (using_subtxn)
RollbackAndReleaseCurrentSubTransaction();
@@ -3438,6 +3627,21 @@ void
ReorderBufferAddInvalidations(ReorderBuffer *rb, TransactionId xid,
XLogRecPtr lsn, Size nmsgs,
SharedInvalidationMessage *msgs)
+{
+ ReorderBufferAddInvalidationsExtended(rb, xid, lsn, nmsgs, msgs, false);
+}
+
+/*
+ * An extended workhorse version of ReorderBufferAddInvalidations(). This can
+ * control whether invalidation messages can be accumulated in reorderbuffer
+ * transactions, which would be distributed to all concurrent transactions at
+ * commit.
+ */
+void
+ReorderBufferAddInvalidationsExtended(ReorderBuffer *rb, TransactionId xid,
+ XLogRecPtr lsn, Size nmsgs,
+ SharedInvalidationMessage *msgs,
+ bool for_inval)
{
ReorderBufferTXN *txn;
MemoryContext oldcontext;
@@ -3449,31 +3653,34 @@ ReorderBufferAddInvalidations(ReorderBuffer *rb, TransactionId xid,
/*
* Collect all the invalidations under the top transaction, if available,
- * so that we can execute them all together. See comments atop this
- * function.
+ * so that we can execute them all together. See comments atop
+ * ReorderBufferAddInvalidations.
*/
txn = rbtxn_get_toptxn(txn);
Assert(nmsgs > 0);
- /* Accumulate invalidations. */
- if (txn->ninvalidations == 0)
+ if (!for_inval)
{
- txn->ninvalidations = nmsgs;
- txn->invalidations = (SharedInvalidationMessage *)
- palloc(sizeof(SharedInvalidationMessage) * nmsgs);
- memcpy(txn->invalidations, msgs,
- sizeof(SharedInvalidationMessage) * nmsgs);
- }
- else
- {
- txn->invalidations = (SharedInvalidationMessage *)
- repalloc(txn->invalidations, sizeof(SharedInvalidationMessage) *
- (txn->ninvalidations + nmsgs));
+ /* Accumulate invalidations. */
+ if (txn->ninvalidations == 0)
+ {
+ txn->ninvalidations = nmsgs;
+ txn->invalidations = (SharedInvalidationMessage *)
+ palloc(sizeof(SharedInvalidationMessage) * nmsgs);
+ memcpy(txn->invalidations, msgs,
+ sizeof(SharedInvalidationMessage) * nmsgs);
+ }
+ else
+ {
+ txn->invalidations = (SharedInvalidationMessage *)
+ repalloc(txn->invalidations,
+ sizeof(SharedInvalidationMessage) * (txn->ninvalidations + nmsgs));
- memcpy(txn->invalidations + txn->ninvalidations, msgs,
- nmsgs * sizeof(SharedInvalidationMessage));
- txn->ninvalidations += nmsgs;
+ memcpy(txn->invalidations + txn->ninvalidations, msgs,
+ nmsgs * sizeof(SharedInvalidationMessage));
+ txn->ninvalidations += nmsgs;
+ }
}
change = ReorderBufferAllocChange(rb);
@@ -3486,6 +3693,28 @@ ReorderBufferAddInvalidations(ReorderBuffer *rb, TransactionId xid,
ReorderBufferQueueChange(rb, xid, lsn, change, false);
+ /*
+ * If the message is distributed from other transactions, also add to
+ * another queue.
+ *
+ * XXX: IIUC This must be done only to the toptxns, but is it right?
+ */
+ if (for_inval && !TransactionIdIsValid(txn->toplevel_xid))
+ {
+ ReorderBufferChange *inval_change;
+
+ /* Duplicate the inval change to queue it */
+ inval_change = ReorderBufferAllocChange(rb);
+ inval_change->action = REORDER_BUFFER_CHANGE_INVALIDATION;
+ inval_change->data.inval.ninvalidations = nmsgs;
+ inval_change->data.inval.invalidations = (SharedInvalidationMessage *)
+ palloc(sizeof(SharedInvalidationMessage) * nmsgs);
+ memcpy(inval_change->data.inval.invalidations, msgs,
+ sizeof(SharedInvalidationMessage) * nmsgs);
+
+ ReorderBufferQueueInval(rb, xid, lsn, inval_change);
+ }
+
MemoryContextSwitchTo(oldcontext);
}
diff --git a/src/backend/replication/logical/snapbuild.c b/src/backend/replication/logical/snapbuild.c
index 0d7bddbe4ed..13d124afb8e 100644
--- a/src/backend/replication/logical/snapbuild.c
+++ b/src/backend/replication/logical/snapbuild.c
@@ -807,8 +807,9 @@ SnapBuildDistributeSnapshotAndInval(SnapBuild *builder, XLogRecPtr lsn, Transact
{
Assert(msgs != NULL);
- ReorderBufferAddInvalidations(builder->reorder, txn->xid, lsn,
- ninvalidations, msgs);
+ ReorderBufferAddInvalidationsExtended(builder->reorder, txn->xid, lsn,
+ ninvalidations, msgs,
+ true);
}
}
}
diff --git a/src/include/replication/reorderbuffer.h b/src/include/replication/reorderbuffer.h
index 24e88c409ba..9d9c8518532 100644
--- a/src/include/replication/reorderbuffer.h
+++ b/src/include/replication/reorderbuffer.h
@@ -422,6 +422,19 @@ typedef struct ReorderBufferTXN
uint32 ninvalidations;
SharedInvalidationMessage *invalidations;
+
+ /*
+ * How many ReorderBufferChange's are we distributed from other
+ * transactions.
+ */
+ uint64 nentries_distr;
+
+ /*
+ * List of ReorderBufferChange structs, but only contains distributed
+ * invalidation messages from concurrent transactions.
+ */
+ dlist_head distributed_inval;
+
/* ---
* Position in one of two lists:
* * list of subtransactions if we are *known* to be subxact
@@ -738,6 +751,9 @@ extern void ReorderBufferAddNewTupleCids(ReorderBuffer *rb, TransactionId xid,
CommandId cmin, CommandId cmax, CommandId combocid);
extern void ReorderBufferAddInvalidations(ReorderBuffer *rb, TransactionId xid, XLogRecPtr lsn,
Size nmsgs, SharedInvalidationMessage *msgs);
+extern void ReorderBufferAddInvalidationsExtended(ReorderBuffer *rb, TransactionId xid, XLogRecPtr lsn,
+ Size nmsgs, SharedInvalidationMessage *msgs,
+ bool needs_distribute);
extern void ReorderBufferImmediateInvalidation(ReorderBuffer *rb, uint32 ninvalidations,
SharedInvalidationMessage *invalidations);
extern void ReorderBufferProcessXid(ReorderBuffer *rb, TransactionId xid, XLogRecPtr lsn);
--
2.47.1
^ permalink raw reply [nested|flat] 94+ messages in thread
* Re: Logical replication 'invalid memory alloc request size 1585837200' after upgrading to 17.5
@ 2025-05-29 17:26 Masahiko Sawada <sawada.mshk@gmail.com>
parent: Amit Kapila <amit.kapila16@gmail.com>
0 siblings, 1 reply; 94+ messages in thread
From: Masahiko Sawada @ 2025-05-29 17:26 UTC (permalink / raw)
To: Amit Kapila <amit.kapila16@gmail.com>; +Cc: Hayato Kuroda (Fujitsu) <kuroda.hayato@fujitsu.com>; Duncan Sands <duncan.sands@deepbluecap.com>; pgsql-bugs@lists.postgresql.org <pgsql-bugs@lists.postgresql.org>
On Thu, May 29, 2025 at 1:22 AM Amit Kapila <amit.kapila16@gmail.com> wrote:
>
> On Thu, May 29, 2025 at 11:37 AM Masahiko Sawada <sawada.mshk@gmail.com> wrote:
> >
> > On Wed, May 28, 2025 at 10:20 PM Amit Kapila <amit.kapila16@gmail.com> wrote:
> > >
> > > On Thu, May 29, 2025 at 12:22 AM Masahiko Sawada <sawada.mshk@gmail.com> wrote:
> > > >
> > > > On Mon, May 26, 2025 at 4:19 AM Amit Kapila <amit.kapila16@gmail.com> wrote:
> > > > >
> > > > > On Mon, May 26, 2025 at 2:52 PM Hayato Kuroda (Fujitsu)
> > > > > <kuroda.hayato@fujitsu.com> wrote:
> > > > > >
> > > > > > > If the above hypothesis is true, we need to consider another idea so
> > > > > > > that we can execute invalidation messages in both cases.
> > > > > >
> > > > > > The straightforward fix is to check the change queue as well when the transaction
> > > > > > has invalidation messages. 0003 implemented that. One downside is that traversing
> > > > > > changes can affect performance. Currently we iterates all of changes even a
> > > > > > single REORDER_BUFFER_CHANGE_INVALIDATION. I cannot find better solutions for now.
> > > > > >
> > > > >
> > > > > It can impact the performance for large transactions with fewer
> > > > > invalidations, especially the ones which has spilled changes because
> > > > > it needs to traverse the entire list of changes again at the end.
> > > >
> > > > What if we remember all executed REORDER_BUFFER_CHANGE_INVALIDATION in
> > > > a queue while replaying the transaction so that we can execute them at
> > > > the end in a non-error path, instead of re-traversing the entire list
> > > > of changes to execute the inval messages?
> > > >
> > >
> > > The current proposed patch (v4) is also traversing only the required
> > > inval messages, as it has maintained a separate queue for that. So,
> > > what will be the advantage of forming such a queue during the
> > > processing of changes? Are you imagining a local instead of a queue at
> > > ReorderBufferTXN level? I feel we still need at ReorderBufferTXN level
> > > to ensure that we can execute those changes across streaming blocks,
> > > otherwise, the cleanup of such a local queue would be tricky and add
> > > to maintenance effort.
> >
> > Hmm, right. It seems that we keep accumulating inval messages across
> > streaming blocks.
> >
> > > One disadvantage of the approach you suggest is that the changes in
> > > the new queue won't be accounted for in logical_decoding_work_mem
> > > computation, which can be done in the proposed approach, although the
> > > patch hasn't implemented it as of now.
> >
> > If we serialize the new queue to the disk, we would need to restore
> > them in PG_CATCH() block in order to execute all inval messages, which
> > is something that I'd like to avoid as it would involve many
> > operations that could end up in an error.
> >
> > If each ReorderBufferTXN has only non-distributed inval messages in
> > txn->invalidation and distribute only txn->invalidations to other
> > transactions, the scope of influence of a single Inval Message is
> > limited to transactions that are being decoded at the same time. How
> > much is there chance the size of txn->invalidations reach 1GB? Given
> > the size of SharedInvalidationMessage is 16 bytes, we need about 67k
> > inval messages generated across concurrent transactions.
> >
>
> I agree that chances are much lower than current if txn->invalidations
> doesn't contain invalidations from other transactions, but it is not
> clear what exactly you are trying to advocate by it. Are you trying to
> advocate that we should maintain a member similar to txn->invalidation
> (say txn->distributed_invals) instead of a queue?
Yes, because I guess it's much simpler. I think it would not be a good
idea to introduce a new concept of accounting the memory usage of the
distributed inval messages too and serializing them, at least on back
branches. I think that In case where the txn->distriubted_inval is
about to overflow (not has to be 1GB) we can invalidate all caches
instread.
Regards,
--
Masahiko Sawada
Amazon Web Services: https://aws.amazon.com
^ permalink raw reply [nested|flat] 94+ messages in thread
* Re: Logical replication 'invalid memory alloc request size 1585837200' after upgrading to 17.5
@ 2025-05-30 04:01 vignesh C <vignesh21@gmail.com>
parent: Masahiko Sawada <sawada.mshk@gmail.com>
0 siblings, 1 reply; 94+ messages in thread
From: vignesh C @ 2025-05-30 04:01 UTC (permalink / raw)
To: Masahiko Sawada <sawada.mshk@gmail.com>; +Cc: Amit Kapila <amit.kapila16@gmail.com>; Hayato Kuroda (Fujitsu) <kuroda.hayato@fujitsu.com>; Duncan Sands <duncan.sands@deepbluecap.com>; pgsql-bugs@lists.postgresql.org <pgsql-bugs@lists.postgresql.org>
On Thu, 29 May 2025 at 22:57, Masahiko Sawada <sawada.mshk@gmail.com> wrote:
>
> > I agree that chances are much lower than current if txn->invalidations
> > doesn't contain invalidations from other transactions, but it is not
> > clear what exactly you are trying to advocate by it. Are you trying to
> > advocate that we should maintain a member similar to txn->invalidation
> > (say txn->distributed_invals) instead of a queue?
>
> Yes, because I guess it's much simpler. I think it would not be a good
> idea to introduce a new concept of accounting the memory usage of the
> distributed inval messages too and serializing them, at least on back
> branches. I think that In case where the txn->distriubted_inval is
> about to overflow (not has to be 1GB) we can invalidate all caches
> instread.
To identify overflow scenarios, I’m considering the following options:
a) Introduce a new txn_flags value, such as RBTXN_INVAL_ALL_CACHE, to
explicitly mark transactions that require full cache invalidation.
b) Add a dedicated parameter to indicate an overflow scenario.
c) setting the newly added nentries_distr to -1, to indicate an
overflow scenario.
Do you have any preference or thoughts on which of these approaches
would be cleaner?
Regards,
Vignesh
^ permalink raw reply [nested|flat] 94+ messages in thread
* Re: Logical replication 'invalid memory alloc request size 1585837200' after upgrading to 17.5
@ 2025-05-30 04:42 Amit Kapila <amit.kapila16@gmail.com>
parent: vignesh C <vignesh21@gmail.com>
0 siblings, 2 replies; 94+ messages in thread
From: Amit Kapila @ 2025-05-30 04:42 UTC (permalink / raw)
To: vignesh C <vignesh21@gmail.com>; +Cc: Masahiko Sawada <sawada.mshk@gmail.com>; Hayato Kuroda (Fujitsu) <kuroda.hayato@fujitsu.com>; Duncan Sands <duncan.sands@deepbluecap.com>; pgsql-bugs@lists.postgresql.org <pgsql-bugs@lists.postgresql.org>
On Fri, May 30, 2025 at 9:31 AM vignesh C <vignesh21@gmail.com> wrote:
>
> On Thu, 29 May 2025 at 22:57, Masahiko Sawada <sawada.mshk@gmail.com> wrote:
> >
> > > I agree that chances are much lower than current if txn->invalidations
> > > doesn't contain invalidations from other transactions, but it is not
> > > clear what exactly you are trying to advocate by it. Are you trying to
> > > advocate that we should maintain a member similar to txn->invalidation
> > > (say txn->distributed_invals) instead of a queue?
> >
> > Yes, because I guess it's much simpler. I think it would not be a good
> > idea to introduce a new concept of accounting the memory usage of the
> > distributed inval messages too and serializing them, at least on back
> > branches. I think that In case where the txn->distriubted_inval is
> > about to overflow (not has to be 1GB) we can invalidate all caches
> > instread.
>
I agree that it would be simpler, and to avoid invalid memory
allocation requests even for rare cases, we can have the backup logic
to invalidate all caches.
> To identify overflow scenarios, I’m considering the following options:
> a) Introduce a new txn_flags value, such as RBTXN_INVAL_ALL_CACHE, to
> explicitly mark transactions that require full cache invalidation.
> b) Add a dedicated parameter to indicate an overflow scenario.
> c) setting the newly added nentries_distr to -1, to indicate an
> overflow scenario.
>
> Do you have any preference or thoughts on which of these approaches
> would be cleaner?
>
I would prefer (a) as that is an explicit way to indicate that we need
to invalidate all caches. But let us see if Sawada-san has something
else in mind.
--
With Regards,
Amit Kapila.
^ permalink raw reply [nested|flat] 94+ messages in thread
* Re: Logical replication 'invalid memory alloc request size 1585837200' after upgrading to 17.5
@ 2025-05-30 12:15 vignesh C <vignesh21@gmail.com>
parent: Amit Kapila <amit.kapila16@gmail.com>
1 sibling, 1 reply; 94+ messages in thread
From: vignesh C @ 2025-05-30 12:15 UTC (permalink / raw)
To: Amit Kapila <amit.kapila16@gmail.com>; +Cc: Masahiko Sawada <sawada.mshk@gmail.com>; Hayato Kuroda (Fujitsu) <kuroda.hayato@fujitsu.com>; Duncan Sands <duncan.sands@deepbluecap.com>; pgsql-bugs@lists.postgresql.org <pgsql-bugs@lists.postgresql.org>
On Fri, 30 May 2025 at 10:12, Amit Kapila <amit.kapila16@gmail.com> wrote:
>
> On Fri, May 30, 2025 at 9:31 AM vignesh C <vignesh21@gmail.com> wrote:
> >
> > On Thu, 29 May 2025 at 22:57, Masahiko Sawada <sawada.mshk@gmail.com> wrote:
> > >
> > > > I agree that chances are much lower than current if txn->invalidations
> > > > doesn't contain invalidations from other transactions, but it is not
> > > > clear what exactly you are trying to advocate by it. Are you trying to
> > > > advocate that we should maintain a member similar to txn->invalidation
> > > > (say txn->distributed_invals) instead of a queue?
> > >
> > > Yes, because I guess it's much simpler. I think it would not be a good
> > > idea to introduce a new concept of accounting the memory usage of the
> > > distributed inval messages too and serializing them, at least on back
> > > branches. I think that In case where the txn->distriubted_inval is
> > > about to overflow (not has to be 1GB) we can invalidate all caches
> > > instread.
> >
>
> I agree that it would be simpler, and to avoid invalid memory
> allocation requests even for rare cases, we can have the backup logic
> to invalidate all caches.
>
> > To identify overflow scenarios, I’m considering the following options:
> > a) Introduce a new txn_flags value, such as RBTXN_INVAL_ALL_CACHE, to
> > explicitly mark transactions that require full cache invalidation.
> > b) Add a dedicated parameter to indicate an overflow scenario.
> > c) setting the newly added nentries_distr to -1, to indicate an
> > overflow scenario.
> >
> > Do you have any preference or thoughts on which of these approaches
> > would be cleaner?
> >
>
> I would prefer (a) as that is an explicit way to indicate that we need
> to invalidate all caches. But let us see if Sawada-san has something
> else in mind.
The attached v6 version patch has the changes for the same.
Thoughts?
Regards,
Vignesh
Attachments:
[text/x-patch] v6-master-0001-Fix-exponential-memory-allocation-issue-in-logica.patch (9.0K, ../../CALDaNm1-RXLA_6y_o1=j5Borz9kGapyimN=6jDbx1nDCmQzjGQ@mail.gmail.com/2-v6-master-0001-Fix-exponential-memory-allocation-issue-in-logica.patch)
download | inline diff:
From 7856d6039e3dc43e89c8a8b375e33985160874bd Mon Sep 17 00:00:00 2001
From: Vignesh <vignesh21@gmail.com>
Date: Fri, 30 May 2025 12:06:07 +0530
Subject: [PATCH v6] Fix exponential memory allocation issue in logical
decoding due to invalidation message distribution
Commit 4909b38af0 added logic to distribute invalidation messages from
one transaction to all concurrent in-progress transactions. However,
since the source of each invalidation message was not tracked, transactions
could redundantly re-distribute already propagated messages. This led to an
exponential increase in invalidation messages and, ultimately, memory
allocation failures.
To fix this, a new field is introduced to track distributed invalidation
messages separately. This ensures that only transactions in the pre-existing
list receive the invalidations, preventing redundant propagation.
The accumulated distributed invalidations are now consumed at the end of
transaction processing in `ReorderBufferProcessTXN()`.
---
.../replication/logical/reorderbuffer.c | 99 +++++++++++++++----
src/backend/replication/logical/snapbuild.c | 5 +-
src/include/replication/reorderbuffer.h | 18 ++++
3 files changed, 103 insertions(+), 19 deletions(-)
diff --git a/src/backend/replication/logical/reorderbuffer.c b/src/backend/replication/logical/reorderbuffer.c
index 67655111875..a6430052e68 100644
--- a/src/backend/replication/logical/reorderbuffer.c
+++ b/src/backend/replication/logical/reorderbuffer.c
@@ -109,6 +109,7 @@
#include "storage/procarray.h"
#include "storage/sinval.h"
#include "utils/builtins.h"
+#include "utils/inval.h"
#include "utils/memutils.h"
#include "utils/rel.h"
#include "utils/relfilenumbermap.h"
@@ -472,6 +473,12 @@ ReorderBufferFreeTXN(ReorderBuffer *rb, ReorderBufferTXN *txn)
txn->invalidations = NULL;
}
+ if (txn->distributed_invalidations)
+ {
+ pfree(txn->distributed_invalidations);
+ txn->distributed_invalidations = NULL;
+ }
+
/* Reset the toast hash */
ReorderBufferToastReset(rb, txn);
@@ -2661,7 +2668,14 @@ ReorderBufferProcessTXN(ReorderBuffer *rb, ReorderBufferTXN *txn,
AbortCurrentTransaction();
/* make sure there's no cache pollution */
- ReorderBufferExecuteInvalidations(txn->ninvalidations, txn->invalidations);
+ if (rbtxn_inval_all_cache(txn))
+ InvalidateSystemCaches();
+ else
+ {
+ ReorderBufferExecuteInvalidations(txn->ninvalidations, txn->invalidations);
+ ReorderBufferExecuteInvalidations(txn->ninvalidations_distr,
+ txn->distributed_invalidations);
+ }
if (using_subtxn)
RollbackAndReleaseCurrentSubTransaction();
@@ -2747,6 +2761,17 @@ ReorderBufferProcessTXN(ReorderBuffer *rb, ReorderBufferTXN *txn,
if (stream_started)
ReorderBufferMaybeMarkTXNStreamed(rb, txn);
+ /*
+ * Make sure there's no cache pollution. Unlike the PG_TRY part,
+ * this must be done unconditionally because the processing might
+ * fail before we reach invalidation messages.
+ */
+ if (rbtxn_inval_all_cache(txn))
+ InvalidateSystemCaches();
+ else
+ ReorderBufferExecuteInvalidations(txn->ninvalidations_distr,
+ txn->distributed_invalidations);
+
/* Reset the TXN so that it is allowed to stream remaining data. */
ReorderBufferResetTXN(rb, txn, snapshot_now,
command_id, prev_lsn,
@@ -3439,9 +3464,27 @@ ReorderBufferAddInvalidations(ReorderBuffer *rb, TransactionId xid,
XLogRecPtr lsn, Size nmsgs,
SharedInvalidationMessage *msgs)
{
- ReorderBufferTXN *txn;
+ ReorderBufferAddInvalidationsExtended(rb, xid, lsn, nmsgs, msgs, false);
+}
+
+/*
+ * An extended workhorse version of ReorderBufferAddInvalidations(). This can
+ * control whether invalidation messages can be accumulated in reorderbuffer
+ * transactions, which would be distributed to all concurrent transactions at
+ * commit.
+ */
+void
+ReorderBufferAddInvalidationsExtended(ReorderBuffer *rb, TransactionId xid,
+ XLogRecPtr lsn, Size nmsgs,
+ SharedInvalidationMessage *msgs,
+ bool for_inval)
+{
MemoryContext oldcontext;
ReorderBufferChange *change;
+ ReorderBufferTXN *txn;
+ uint32 *ninvalidations;
+ SharedInvalidationMessage **invalidations;
+ Size req_mem_size;
txn = ReorderBufferTXNByXid(rb, xid, true, NULL, lsn, true);
@@ -3449,31 +3492,53 @@ ReorderBufferAddInvalidations(ReorderBuffer *rb, TransactionId xid,
/*
* Collect all the invalidations under the top transaction, if available,
- * so that we can execute them all together. See comments atop this
- * function.
+ * so that we can execute them all together. See comments atop
+ * ReorderBufferAddInvalidations.
*/
txn = rbtxn_get_toptxn(txn);
Assert(nmsgs > 0);
- /* Accumulate invalidations. */
- if (txn->ninvalidations == 0)
+ ninvalidations = for_inval ? &txn->ninvalidations_distr : &txn->ninvalidations;
+ invalidations = for_inval ? &txn->distributed_invalidations : &txn->invalidations;
+
+ req_mem_size = sizeof(SharedInvalidationMessage) * (*ninvalidations + nmsgs);
+
+ /*
+ * If the number of invalidation messages is large, it's more efficient to
+ * invalidate the entire cache rather than processing each message
+ * individually.
+ */
+ if ((for_inval && !AllocSizeIsValid(req_mem_size)) ||
+ rbtxn_inval_all_cache(txn))
{
- txn->ninvalidations = nmsgs;
- txn->invalidations = (SharedInvalidationMessage *)
- palloc(sizeof(SharedInvalidationMessage) * nmsgs);
- memcpy(txn->invalidations, msgs,
- sizeof(SharedInvalidationMessage) * nmsgs);
+ txn->txn_flags |= RBTXN_INVAL_ALL_CACHE;
+
+ if (*invalidations)
+ {
+ pfree(*invalidations);
+ *invalidations = NULL;
+ *ninvalidations = 0;
+ }
}
else
{
- txn->invalidations = (SharedInvalidationMessage *)
- repalloc(txn->invalidations, sizeof(SharedInvalidationMessage) *
- (txn->ninvalidations + nmsgs));
+ /* Accumulate invalidations. */
+ if (*ninvalidations == 0)
+ {
+ *ninvalidations = nmsgs;
+ *invalidations = (SharedInvalidationMessage *) palloc(req_mem_size);
+ memcpy(*invalidations, msgs, req_mem_size);
+ }
+ else
+ {
+ *invalidations = (SharedInvalidationMessage *)
+ repalloc(*invalidations, req_mem_size);
- memcpy(txn->invalidations + txn->ninvalidations, msgs,
- nmsgs * sizeof(SharedInvalidationMessage));
- txn->ninvalidations += nmsgs;
+ memcpy(*invalidations + *ninvalidations, msgs,
+ nmsgs * sizeof(SharedInvalidationMessage));
+ *ninvalidations += nmsgs;
+ }
}
change = ReorderBufferAllocChange(rb);
diff --git a/src/backend/replication/logical/snapbuild.c b/src/backend/replication/logical/snapbuild.c
index 0d7bddbe4ed..13d124afb8e 100644
--- a/src/backend/replication/logical/snapbuild.c
+++ b/src/backend/replication/logical/snapbuild.c
@@ -807,8 +807,9 @@ SnapBuildDistributeSnapshotAndInval(SnapBuild *builder, XLogRecPtr lsn, Transact
{
Assert(msgs != NULL);
- ReorderBufferAddInvalidations(builder->reorder, txn->xid, lsn,
- ninvalidations, msgs);
+ ReorderBufferAddInvalidationsExtended(builder->reorder, txn->xid, lsn,
+ ninvalidations, msgs,
+ true);
}
}
}
diff --git a/src/include/replication/reorderbuffer.h b/src/include/replication/reorderbuffer.h
index 24e88c409ba..71a8ad478ef 100644
--- a/src/include/replication/reorderbuffer.h
+++ b/src/include/replication/reorderbuffer.h
@@ -176,6 +176,7 @@ typedef struct ReorderBufferChange
#define RBTXN_SENT_PREPARE 0x0200
#define RBTXN_IS_COMMITTED 0x0400
#define RBTXN_IS_ABORTED 0x0800
+#define RBTXN_INVAL_ALL_CACHE 0x1000
#define RBTXN_PREPARE_STATUS_MASK (RBTXN_IS_PREPARED | RBTXN_SKIPPED_PREPARE | RBTXN_SENT_PREPARE)
@@ -215,6 +216,12 @@ typedef struct ReorderBufferChange
((txn)->txn_flags & RBTXN_HAS_STREAMABLE_CHANGE) != 0 \
)
+/* Should the complete cache be invalidate ? */
+#define rbtxn_inval_all_cache(txn) \
+( \
+ ((txn)->txn_flags & RBTXN_INVAL_ALL_CACHE) != 0 \
+)
+
/*
* Has this transaction been streamed to downstream?
*
@@ -422,6 +429,14 @@ typedef struct ReorderBufferTXN
uint32 ninvalidations;
SharedInvalidationMessage *invalidations;
+ /*
+ * Stores the cache invalidation messages distributed by the committing
+ * transaction.
+ */
+ uint32 ninvalidations_distr;
+ SharedInvalidationMessage *distributed_invalidations;
+
+
/* ---
* Position in one of two lists:
* * list of subtransactions if we are *known* to be subxact
@@ -738,6 +753,9 @@ extern void ReorderBufferAddNewTupleCids(ReorderBuffer *rb, TransactionId xid,
CommandId cmin, CommandId cmax, CommandId combocid);
extern void ReorderBufferAddInvalidations(ReorderBuffer *rb, TransactionId xid, XLogRecPtr lsn,
Size nmsgs, SharedInvalidationMessage *msgs);
+extern void ReorderBufferAddInvalidationsExtended(ReorderBuffer *rb, TransactionId xid, XLogRecPtr lsn,
+ Size nmsgs, SharedInvalidationMessage *msgs,
+ bool for_inval);
extern void ReorderBufferImmediateInvalidation(ReorderBuffer *rb, uint32 ninvalidations,
SharedInvalidationMessage *invalidations);
extern void ReorderBufferProcessXid(ReorderBuffer *rb, TransactionId xid, XLogRecPtr lsn);
--
2.43.0
^ permalink raw reply [nested|flat] 94+ messages in thread
* Re: Logical replication 'invalid memory alloc request size 1585837200' after upgrading to 17.5
@ 2025-05-30 16:45 Masahiko Sawada <sawada.mshk@gmail.com>
parent: Amit Kapila <amit.kapila16@gmail.com>
1 sibling, 0 replies; 94+ messages in thread
From: Masahiko Sawada @ 2025-05-30 16:45 UTC (permalink / raw)
To: Amit Kapila <amit.kapila16@gmail.com>; +Cc: vignesh C <vignesh21@gmail.com>; Hayato Kuroda (Fujitsu) <kuroda.hayato@fujitsu.com>; Duncan Sands <duncan.sands@deepbluecap.com>; pgsql-bugs@lists.postgresql.org <pgsql-bugs@lists.postgresql.org>
On Thu, May 29, 2025 at 9:42 PM Amit Kapila <amit.kapila16@gmail.com> wrote:
>
> On Fri, May 30, 2025 at 9:31 AM vignesh C <vignesh21@gmail.com> wrote:
> >
> > On Thu, 29 May 2025 at 22:57, Masahiko Sawada <sawada.mshk@gmail.com> wrote:
> > >
> > > > I agree that chances are much lower than current if txn->invalidations
> > > > doesn't contain invalidations from other transactions, but it is not
> > > > clear what exactly you are trying to advocate by it. Are you trying to
> > > > advocate that we should maintain a member similar to txn->invalidation
> > > > (say txn->distributed_invals) instead of a queue?
> > >
> > > Yes, because I guess it's much simpler. I think it would not be a good
> > > idea to introduce a new concept of accounting the memory usage of the
> > > distributed inval messages too and serializing them, at least on back
> > > branches. I think that In case where the txn->distriubted_inval is
> > > about to overflow (not has to be 1GB) we can invalidate all caches
> > > instread.
> >
>
> I agree that it would be simpler, and to avoid invalid memory
> allocation requests even for rare cases, we can have the backup logic
> to invalidate all caches.
>
> > To identify overflow scenarios, I’m considering the following options:
> > a) Introduce a new txn_flags value, such as RBTXN_INVAL_ALL_CACHE, to
> > explicitly mark transactions that require full cache invalidation.
> > b) Add a dedicated parameter to indicate an overflow scenario.
> > c) setting the newly added nentries_distr to -1, to indicate an
> > overflow scenario.
> >
> > Do you have any preference or thoughts on which of these approaches
> > would be cleaner?
> >
>
> I would prefer (a) as that is an explicit way to indicate that we need
> to invalidate all caches. But let us see if Sawada-san has something
> else in mind.
(a) makes sense to me too. One concern in back branches is that unused
bits in txn_flags might be used by extensions, but it's unlikely.
Regards,
--
Masahiko Sawada
Amazon Web Services: https://aws.amazon.com
^ permalink raw reply [nested|flat] 94+ messages in thread
* Re: Logical replication 'invalid memory alloc request size 1585837200' after upgrading to 17.5
@ 2025-05-30 17:29 Masahiko Sawada <sawada.mshk@gmail.com>
parent: vignesh C <vignesh21@gmail.com>
0 siblings, 2 replies; 94+ messages in thread
From: Masahiko Sawada @ 2025-05-30 17:29 UTC (permalink / raw)
To: vignesh C <vignesh21@gmail.com>; +Cc: Amit Kapila <amit.kapila16@gmail.com>; Hayato Kuroda (Fujitsu) <kuroda.hayato@fujitsu.com>; Duncan Sands <duncan.sands@deepbluecap.com>; pgsql-bugs@lists.postgresql.org <pgsql-bugs@lists.postgresql.org>
On Fri, May 30, 2025 at 5:15 AM vignesh C <vignesh21@gmail.com> wrote:
>
> On Fri, 30 May 2025 at 10:12, Amit Kapila <amit.kapila16@gmail.com> wrote:
> >
> > On Fri, May 30, 2025 at 9:31 AM vignesh C <vignesh21@gmail.com> wrote:
> > >
> > > On Thu, 29 May 2025 at 22:57, Masahiko Sawada <sawada.mshk@gmail.com> wrote:
> > > >
> > > > > I agree that chances are much lower than current if txn->invalidations
> > > > > doesn't contain invalidations from other transactions, but it is not
> > > > > clear what exactly you are trying to advocate by it. Are you trying to
> > > > > advocate that we should maintain a member similar to txn->invalidation
> > > > > (say txn->distributed_invals) instead of a queue?
> > > >
> > > > Yes, because I guess it's much simpler. I think it would not be a good
> > > > idea to introduce a new concept of accounting the memory usage of the
> > > > distributed inval messages too and serializing them, at least on back
> > > > branches. I think that In case where the txn->distriubted_inval is
> > > > about to overflow (not has to be 1GB) we can invalidate all caches
> > > > instread.
> > >
> >
> > I agree that it would be simpler, and to avoid invalid memory
> > allocation requests even for rare cases, we can have the backup logic
> > to invalidate all caches.
> >
> > > To identify overflow scenarios, I’m considering the following options:
> > > a) Introduce a new txn_flags value, such as RBTXN_INVAL_ALL_CACHE, to
> > > explicitly mark transactions that require full cache invalidation.
> > > b) Add a dedicated parameter to indicate an overflow scenario.
> > > c) setting the newly added nentries_distr to -1, to indicate an
> > > overflow scenario.
> > >
> > > Do you have any preference or thoughts on which of these approaches
> > > would be cleaner?
> > >
> >
> > I would prefer (a) as that is an explicit way to indicate that we need
> > to invalidate all caches. But let us see if Sawada-san has something
> > else in mind.
>
> The attached v6 version patch has the changes for the same.
> Thoughts?
Thank you for updating the patch. Here are some review comments:
@@ -3439,9 +3464,27 @@ ReorderBufferAddInvalidations(ReorderBuffer
*rb, TransactionId xid,
XLogRecPtr lsn, Size nmsgs,
SharedInvalidationMessage *msgs)
{
- ReorderBufferTXN *txn;
+ ReorderBufferAddInvalidationsExtended(rb, xid, lsn, nmsgs, msgs, false);
+}
If the patch is the changes for master do we need to have an extended
version of ReorderBufferAddInvalidation()?
---
+ /*
+ * Make sure there's no cache pollution. Unlike the PG_TRY part,
+ * this must be done unconditionally because the processing might
+ * fail before we reach invalidation messages.
+ */
+ if (rbtxn_inval_all_cache(txn))
+ InvalidateSystemCaches();
+ else
+ ReorderBufferExecuteInvalidations(txn->ninvalidations_distr,
+
txn->distributed_invalidations);
+
If we don't need to execute the distributed inval message in an error
path other than detecting concurrent abort, we should describe the
reason.
---
Given that we don't account the memory usage of both
txn->invalidations and txn->distributed_invalidations, probably we can
have a lower limit, say 8MB (or lower?), to avoid memory exhaustion.
---
+ if ((for_inval && !AllocSizeIsValid(req_mem_size)) ||
+ rbtxn_inval_all_cache(txn))
{
- txn->ninvalidations = nmsgs;
- txn->invalidations = (SharedInvalidationMessage *)
- palloc(sizeof(SharedInvalidationMessage) * nmsgs);
- memcpy(txn->invalidations, msgs,
- sizeof(SharedInvalidationMessage) * nmsgs);
+ txn->txn_flags |= RBTXN_INVAL_ALL_CACHE;
+
+ if (*invalidations)
+ {
+ pfree(*invalidations);
+ *invalidations = NULL;
+ *ninvalidations = 0;
+ }
RBTXN_INVAL_ALL_CACHE seems to have an effect only on the distributed
inval messages. One question is do we need to care about the overflow
of txn->invalidations as well? If no, does it make sense to have a
separate function like ReorderBufferAddDistributedInvalidtions()
instead of having an extended version of
ReorderBufferAddInvalidations()? Some common routines can also be
declared as a static function if needed.
Regards,
--
Masahiko Sawada
Amazon Web Services: https://aws.amazon.com
^ permalink raw reply [nested|flat] 94+ messages in thread
* Re: Logical replication 'invalid memory alloc request size 1585837200' after upgrading to 17.5
@ 2025-05-31 04:50 Amit Kapila <amit.kapila16@gmail.com>
parent: Masahiko Sawada <sawada.mshk@gmail.com>
1 sibling, 1 reply; 94+ messages in thread
From: Amit Kapila @ 2025-05-31 04:50 UTC (permalink / raw)
To: Masahiko Sawada <sawada.mshk@gmail.com>; +Cc: vignesh C <vignesh21@gmail.com>; Hayato Kuroda (Fujitsu) <kuroda.hayato@fujitsu.com>; Duncan Sands <duncan.sands@deepbluecap.com>; pgsql-bugs@lists.postgresql.org <pgsql-bugs@lists.postgresql.org>
On Fri, May 30, 2025 at 11:00 PM Masahiko Sawada <sawada.mshk@gmail.com> wrote:
>
> Thank you for updating the patch. Here are some review comments:
>
> ---
> + /*
> + * Make sure there's no cache pollution. Unlike the PG_TRY part,
> + * this must be done unconditionally because the processing might
> + * fail before we reach invalidation messages.
> + */
> + if (rbtxn_inval_all_cache(txn))
> + InvalidateSystemCaches();
> + else
> + ReorderBufferExecuteInvalidations(txn->ninvalidations_distr,
> +
> txn->distributed_invalidations);
> +
>
> If we don't need to execute the distributed inval message in an error
> path other than detecting concurrent abort, we should describe the
> reason.
>
The concurrent abort handling is done for streaming and prepared
transactions, where we send the transaction changes to the client
before we read COMMIT/COMMIT PREPARED/ROLLBACK/ROLLBACK PREPARED. Now,
among these COMMIT/ROLLBACK PREPARED cases are handled as part of a
prepared transaction case. For ROLLBACK, we will never perform any
changes from the current transaction, so we don't need distributed
invalidations to be executed. For COMMIT, if we encounter any errors
while processing changes (this is when we reach the ERROR path, which
is not a concurrent abort), then we will reprocess all changes and, at
the end, execute both the current transaction and distributed
invalidations. Now, one possibility is that if, after ERROR, the
caller does slot_advance to skip the ERROR, then we will probably miss
executing the distributed invalidations, leading to data loss
afterwards. If the above theory is correct, then it is better to
execute distributed invalidation even in non-concurrent-abort cases in
the ERROR path.
> ---
> Given that we don't account the memory usage of both
> txn->invalidations and txn->distributed_invalidations, probably we can
> have a lower limit, say 8MB (or lower?), to avoid memory exhaustion.
>
Are you thinking if there are many waslenders in the system, and we
kept the limit higher, say 256 MB, then all will start consuming that
much memory, leading to memory exhaustion? If so, then I agree with
your point to keep this limit as 8MB, we can probably explain the same
in comments as well.
> ---
> + if ((for_inval && !AllocSizeIsValid(req_mem_size)) ||
> + rbtxn_inval_all_cache(txn))
> {
> - txn->ninvalidations = nmsgs;
> - txn->invalidations = (SharedInvalidationMessage *)
> - palloc(sizeof(SharedInvalidationMessage) * nmsgs);
> - memcpy(txn->invalidations, msgs,
> - sizeof(SharedInvalidationMessage) * nmsgs);
> + txn->txn_flags |= RBTXN_INVAL_ALL_CACHE;
> +
> + if (*invalidations)
> + {
> + pfree(*invalidations);
> + *invalidations = NULL;
> + *ninvalidations = 0;
> + }
>
> RBTXN_INVAL_ALL_CACHE seems to have an effect only on the distributed
> inval messages. One question is do we need to care about the overflow
> of txn->invalidations as well?
>
I don't think so. This has been working for a long time, so unless we
see a case where the overflow can happen, it is better not to change
it.
>
If no, does it make sense to have a
> separate function like ReorderBufferAddDistributedInvalidtions()
> instead of having an extended version of
> ReorderBufferAddInvalidations()? Some common routines can also be
> declared as a static function if needed.
>
+1.
--
With Regards,
Amit Kapila.
^ permalink raw reply [nested|flat] 94+ messages in thread
* Re: Logical replication 'invalid memory alloc request size 1585837200' after upgrading to 17.5
@ 2025-05-31 07:57 vignesh C <vignesh21@gmail.com>
parent: Masahiko Sawada <sawada.mshk@gmail.com>
1 sibling, 1 reply; 94+ messages in thread
From: vignesh C @ 2025-05-31 07:57 UTC (permalink / raw)
To: Masahiko Sawada <sawada.mshk@gmail.com>; +Cc: Amit Kapila <amit.kapila16@gmail.com>; Hayato Kuroda (Fujitsu) <kuroda.hayato@fujitsu.com>; Duncan Sands <duncan.sands@deepbluecap.com>; pgsql-bugs@lists.postgresql.org <pgsql-bugs@lists.postgresql.org>
On Fri, 30 May 2025 at 23:00, Masahiko Sawada <sawada.mshk@gmail.com> wrote:
>
> Thank you for updating the patch. Here are some review comments:
>
> @@ -3439,9 +3464,27 @@ ReorderBufferAddInvalidations(ReorderBuffer
> *rb, TransactionId xid,
> XLogRecPtr lsn, Size nmsgs,
> SharedInvalidationMessage *msgs)
> {
> - ReorderBufferTXN *txn;
> + ReorderBufferAddInvalidationsExtended(rb, xid, lsn, nmsgs, msgs, false);
> +}
>
> If the patch is the changes for master do we need to have an extended
> version of ReorderBufferAddInvalidation()?
This has been removed now and ReorderBufferAddDistributedInvalidtions
has been added
> ---
> + /*
> + * Make sure there's no cache pollution. Unlike the PG_TRY part,
> + * this must be done unconditionally because the processing might
> + * fail before we reach invalidation messages.
> + */
> + if (rbtxn_inval_all_cache(txn))
> + InvalidateSystemCaches();
> + else
> + ReorderBufferExecuteInvalidations(txn->ninvalidations_distr,
> +
> txn->distributed_invalidations);
> +
>
> If we don't need to execute the distributed inval message in an error
> path other than detecting concurrent abort, we should describe the
> reason.
Removed it to keep it in the common error path
> ---
> Given that we don't account the memory usage of both
> txn->invalidations and txn->distributed_invalidations, probably we can
> have a lower limit, say 8MB (or lower?), to avoid memory exhaustion.
Modified
> ---
> + if ((for_inval && !AllocSizeIsValid(req_mem_size)) ||
> + rbtxn_inval_all_cache(txn))
> {
> - txn->ninvalidations = nmsgs;
> - txn->invalidations = (SharedInvalidationMessage *)
> - palloc(sizeof(SharedInvalidationMessage) * nmsgs);
> - memcpy(txn->invalidations, msgs,
> - sizeof(SharedInvalidationMessage) * nmsgs);
> + txn->txn_flags |= RBTXN_INVAL_ALL_CACHE;
> +
> + if (*invalidations)
> + {
> + pfree(*invalidations);
> + *invalidations = NULL;
> + *ninvalidations = 0;
> + }
>
> RBTXN_INVAL_ALL_CACHE seems to have an effect only on the distributed
> inval messages. One question is do we need to care about the overflow
> of txn->invalidations as well? If no, does it make sense to have a
> separate function like ReorderBufferAddDistributedInvalidtions()
> instead of having an extended version of
> ReorderBufferAddInvalidations()? Some common routines can also be
> declared as a static function if needed.
Modified
The attached v7 version patch has the changes for the same.
Regards,
Vignesh
Attachments:
[text/x-patch] v7-master-0001-Fix-exponential-memory-allocation-issue-in-logica.patch (11.7K, ../../CALDaNm0iqYWveBLwGSgyn5WZQ7F-hjPNtxLtctUi=Oa=-jTHDw@mail.gmail.com/2-v7-master-0001-Fix-exponential-memory-allocation-issue-in-logica.patch)
download | inline diff:
From eed986e44fae4d5b843589dbedd4733983ecac9a Mon Sep 17 00:00:00 2001
From: Vignesh <vignesh21@gmail.com>
Date: Fri, 30 May 2025 12:06:07 +0530
Subject: [PATCH v7] Fix exponential memory allocation issue in logical
decoding due to invalidation message distribution
Commit 4909b38af0 added logic to distribute invalidation messages from
one transaction to all concurrent in-progress transactions. However,
since the source of each invalidation message was not tracked, transactions
could redundantly re-distribute already propagated messages. This led to an
exponential increase in invalidation messages and, ultimately, memory
allocation failures.
To fix this, a new field is introduced to track distributed invalidation
messages separately. This ensures that only transactions in the pre-existing
list receive the invalidations, preventing redundant propagation.
The accumulated distributed invalidations are now consumed at the end of
transaction processing in `ReorderBufferProcessTXN()`.
---
.../replication/logical/reorderbuffer.c | 171 ++++++++++++++----
src/backend/replication/logical/snapbuild.c | 5 +-
src/include/replication/reorderbuffer.h | 18 ++
3 files changed, 153 insertions(+), 41 deletions(-)
diff --git a/src/backend/replication/logical/reorderbuffer.c b/src/backend/replication/logical/reorderbuffer.c
index 67655111875..509d32dc286 100644
--- a/src/backend/replication/logical/reorderbuffer.c
+++ b/src/backend/replication/logical/reorderbuffer.c
@@ -109,6 +109,7 @@
#include "storage/procarray.h"
#include "storage/sinval.h"
#include "utils/builtins.h"
+#include "utils/inval.h"
#include "utils/memutils.h"
#include "utils/rel.h"
#include "utils/relfilenumbermap.h"
@@ -472,6 +473,12 @@ ReorderBufferFreeTXN(ReorderBuffer *rb, ReorderBufferTXN *txn)
txn->invalidations = NULL;
}
+ if (txn->distributed_invalidations)
+ {
+ pfree(txn->distributed_invalidations);
+ txn->distributed_invalidations = NULL;
+ }
+
/* Reset the toast hash */
ReorderBufferToastReset(rb, txn);
@@ -2661,7 +2668,14 @@ ReorderBufferProcessTXN(ReorderBuffer *rb, ReorderBufferTXN *txn,
AbortCurrentTransaction();
/* make sure there's no cache pollution */
- ReorderBufferExecuteInvalidations(txn->ninvalidations, txn->invalidations);
+ if (rbtxn_inval_all_cache(txn))
+ InvalidateSystemCaches();
+ else
+ {
+ ReorderBufferExecuteInvalidations(txn->ninvalidations, txn->invalidations);
+ ReorderBufferExecuteInvalidations(txn->ninvalidations_distr,
+ txn->distributed_invalidations);
+ }
if (using_subtxn)
RollbackAndReleaseCurrentSubTransaction();
@@ -2710,8 +2724,14 @@ ReorderBufferProcessTXN(ReorderBuffer *rb, ReorderBufferTXN *txn,
AbortCurrentTransaction();
/* make sure there's no cache pollution */
- ReorderBufferExecuteInvalidations(txn->ninvalidations,
- txn->invalidations);
+ if (rbtxn_inval_all_cache(txn))
+ InvalidateSystemCaches();
+ else
+ {
+ ReorderBufferExecuteInvalidations(txn->ninvalidations, txn->invalidations);
+ ReorderBufferExecuteInvalidations(txn->ninvalidations_distr,
+ txn->distributed_invalidations);
+ }
if (using_subtxn)
RollbackAndReleaseCurrentSubTransaction();
@@ -3422,58 +3442,57 @@ ReorderBufferAddNewTupleCids(ReorderBuffer *rb, TransactionId xid,
}
/*
- * Accumulate the invalidations for executing them later.
+ * Helper function to store invalidation messages and queue them as a
+ * change in the reorder buffer. Handles both regular and distributed
+ * invalidations.
*
- * This needs to be called for each XLOG_XACT_INVALIDATIONS message and
- * accumulates all the invalidation messages in the toplevel transaction, if
- * available, otherwise in the current transaction, as well as in the form of
- * change in reorder buffer. We require to record it in form of the change
+ * This accumulates all the invalidation messages in the toplevel transaction,
+ * if available, otherwise in the current transaction, as well as in the form
+ * of change in reorder buffer. We require to record it in form of the change
* so that we can execute only the required invalidations instead of executing
* all the invalidations on each CommandId increment. We also need to
* accumulate these in the txn buffer because in some cases where we skip
* processing the transaction (see ReorderBufferForget), we need to execute
* all the invalidations together.
*/
-void
-ReorderBufferAddInvalidations(ReorderBuffer *rb, TransactionId xid,
- XLogRecPtr lsn, Size nmsgs,
- SharedInvalidationMessage *msgs)
+static void
+ReorderBufferAddInvalidationsCommon(ReorderBuffer *rb, TransactionId xid,
+ XLogRecPtr lsn, Size nmsgs,
+ SharedInvalidationMessage *msgs,
+ ReorderBufferTXN *txn,
+ bool for_inval)
{
- ReorderBufferTXN *txn;
MemoryContext oldcontext;
ReorderBufferChange *change;
- txn = ReorderBufferTXNByXid(rb, xid, true, NULL, lsn, true);
-
oldcontext = MemoryContextSwitchTo(rb->context);
- /*
- * Collect all the invalidations under the top transaction, if available,
- * so that we can execute them all together. See comments atop this
- * function.
- */
- txn = rbtxn_get_toptxn(txn);
+ if (!for_inval || (for_inval && !rbtxn_inval_all_cache(txn)))
+ {
+ uint32 *ninvalidations;
+ SharedInvalidationMessage **invalidations;
- Assert(nmsgs > 0);
+ ninvalidations = for_inval ? &txn->ninvalidations_distr : &txn->ninvalidations;
+ invalidations = for_inval ? &txn->distributed_invalidations : &txn->invalidations;
- /* Accumulate invalidations. */
- if (txn->ninvalidations == 0)
- {
- txn->ninvalidations = nmsgs;
- txn->invalidations = (SharedInvalidationMessage *)
- palloc(sizeof(SharedInvalidationMessage) * nmsgs);
- memcpy(txn->invalidations, msgs,
- sizeof(SharedInvalidationMessage) * nmsgs);
- }
- else
- {
- txn->invalidations = (SharedInvalidationMessage *)
- repalloc(txn->invalidations, sizeof(SharedInvalidationMessage) *
- (txn->ninvalidations + nmsgs));
+ /* Accumulate invalidations. */
+ if (*ninvalidations == 0)
+ {
+ *ninvalidations = nmsgs;
+ *invalidations = (SharedInvalidationMessage *)
+ palloc(sizeof(SharedInvalidationMessage) * nmsgs);
+ memcpy(*invalidations, msgs, sizeof(SharedInvalidationMessage) * nmsgs);
+ }
+ else
+ {
+ *invalidations = (SharedInvalidationMessage *)
+ repalloc(*invalidations, sizeof(SharedInvalidationMessage) *
+ (*ninvalidations + nmsgs));
- memcpy(txn->invalidations + txn->ninvalidations, msgs,
- nmsgs * sizeof(SharedInvalidationMessage));
- txn->ninvalidations += nmsgs;
+ memcpy(*invalidations + *ninvalidations, msgs,
+ nmsgs * sizeof(SharedInvalidationMessage));
+ *ninvalidations += nmsgs;
+ }
}
change = ReorderBufferAllocChange(rb);
@@ -3489,6 +3508,80 @@ ReorderBufferAddInvalidations(ReorderBuffer *rb, TransactionId xid,
MemoryContextSwitchTo(oldcontext);
}
+/*
+ * Accumulate the invalidations for executing them later.
+ *
+ * This needs to be called for each XLOG_XACT_INVALIDATIONS message.
+ */
+void
+ReorderBufferAddInvalidations(ReorderBuffer *rb, TransactionId xid,
+ XLogRecPtr lsn, Size nmsgs,
+ SharedInvalidationMessage *msgs)
+{
+ ReorderBufferTXN *txn;
+
+ txn = ReorderBufferTXNByXid(rb, xid, true, NULL, lsn, true);
+
+ /*
+ * Collect all the invalidations under the top transaction, if available,
+ * so that we can execute them all together. See comments atop
+ * ReorderBufferAddInvalidationsCommon function.
+ */
+ txn = rbtxn_get_toptxn(txn);
+
+ Assert(nmsgs > 0);
+
+ ReorderBufferAddInvalidationsCommon(rb, xid, lsn, nmsgs, msgs, txn, false);
+}
+
+/*
+ * Accumulate the invalidations sent by committed transactions for executing
+ * them later.
+ *
+ * This needs to be called by committed transactions to distribute the
+ * invalidations to the in-progress transactions.
+ */
+void
+ReorderBufferAddDistributedInvalidtions(ReorderBuffer *rb, TransactionId xid,
+ XLogRecPtr lsn, Size nmsgs,
+ SharedInvalidationMessage *msgs)
+{
+ ReorderBufferTXN *txn;
+ Size req_mem_size;
+
+ txn = ReorderBufferTXNByXid(rb, xid, true, NULL, lsn, true);
+
+ /*
+ * Collect all the invalidations under the top transaction, if available,
+ * so that we can execute them all together. See comments
+ * ReorderBufferAddInvalidationsCommon ReorderBufferAddInvalidations.
+ */
+ txn = rbtxn_get_toptxn(txn);
+
+ Assert(nmsgs > 0);
+
+ req_mem_size = sizeof(SharedInvalidationMessage) * (txn->ninvalidations_distr + nmsgs);
+
+ /*
+ * If the number of invalidation messages is larger than 8MB, it's more
+ * efficient to invalidate the entire cache rather than processing each
+ * message individually.
+ */
+ if (req_mem_size > (8 * 1024 * 1024) || rbtxn_inval_all_cache(txn))
+ {
+ txn->txn_flags |= RBTXN_INVAL_ALL_CACHE;
+
+ if (txn->distributed_invalidations)
+ {
+ pfree(txn->distributed_invalidations);
+ txn->distributed_invalidations = NULL;
+ txn->ninvalidations_distr = 0;
+ }
+ }
+
+ ReorderBufferAddInvalidationsCommon(rb, xid, lsn, nmsgs, msgs, txn, true);
+}
+
/*
* Apply all invalidations we know. Possibly we only need parts at this point
* in the changestream but we don't know which those are.
diff --git a/src/backend/replication/logical/snapbuild.c b/src/backend/replication/logical/snapbuild.c
index 0d7bddbe4ed..a2d93f35a2f 100644
--- a/src/backend/replication/logical/snapbuild.c
+++ b/src/backend/replication/logical/snapbuild.c
@@ -807,8 +807,9 @@ SnapBuildDistributeSnapshotAndInval(SnapBuild *builder, XLogRecPtr lsn, Transact
{
Assert(msgs != NULL);
- ReorderBufferAddInvalidations(builder->reorder, txn->xid, lsn,
- ninvalidations, msgs);
+ ReorderBufferAddDistributedInvalidtions(builder->reorder,
+ txn->xid, lsn,
+ ninvalidations, msgs);
}
}
}
diff --git a/src/include/replication/reorderbuffer.h b/src/include/replication/reorderbuffer.h
index 24e88c409ba..a56c3e25d5a 100644
--- a/src/include/replication/reorderbuffer.h
+++ b/src/include/replication/reorderbuffer.h
@@ -176,6 +176,7 @@ typedef struct ReorderBufferChange
#define RBTXN_SENT_PREPARE 0x0200
#define RBTXN_IS_COMMITTED 0x0400
#define RBTXN_IS_ABORTED 0x0800
+#define RBTXN_INVAL_ALL_CACHE 0x1000
#define RBTXN_PREPARE_STATUS_MASK (RBTXN_IS_PREPARED | RBTXN_SKIPPED_PREPARE | RBTXN_SENT_PREPARE)
@@ -215,6 +216,12 @@ typedef struct ReorderBufferChange
((txn)->txn_flags & RBTXN_HAS_STREAMABLE_CHANGE) != 0 \
)
+/* Should the complete cache be invalidated? */
+#define rbtxn_inval_all_cache(txn) \
+( \
+ ((txn)->txn_flags & RBTXN_INVAL_ALL_CACHE) != 0 \
+)
+
/*
* Has this transaction been streamed to downstream?
*
@@ -422,6 +429,14 @@ typedef struct ReorderBufferTXN
uint32 ninvalidations;
SharedInvalidationMessage *invalidations;
+ /*
+ * Stores the cache invalidation messages distributed by the committing
+ * transaction.
+ */
+ uint32 ninvalidations_distr;
+ SharedInvalidationMessage *distributed_invalidations;
+
+
/* ---
* Position in one of two lists:
* * list of subtransactions if we are *known* to be subxact
@@ -738,6 +753,9 @@ extern void ReorderBufferAddNewTupleCids(ReorderBuffer *rb, TransactionId xid,
CommandId cmin, CommandId cmax, CommandId combocid);
extern void ReorderBufferAddInvalidations(ReorderBuffer *rb, TransactionId xid, XLogRecPtr lsn,
Size nmsgs, SharedInvalidationMessage *msgs);
+extern void ReorderBufferAddDistributedInvalidtions(ReorderBuffer *rb, TransactionId xid,
+ XLogRecPtr lsn, Size nmsgs,
+ SharedInvalidationMessage *msgs);
extern void ReorderBufferImmediateInvalidation(ReorderBuffer *rb, uint32 ninvalidations,
SharedInvalidationMessage *invalidations);
extern void ReorderBufferProcessXid(ReorderBuffer *rb, TransactionId xid, XLogRecPtr lsn);
--
2.43.0
^ permalink raw reply [nested|flat] 94+ messages in thread
* Re: Logical replication 'invalid memory alloc request size 1585837200' after upgrading to 17.5
@ 2025-06-02 03:13 vignesh C <vignesh21@gmail.com>
parent: Amit Kapila <amit.kapila16@gmail.com>
0 siblings, 1 reply; 94+ messages in thread
From: vignesh C @ 2025-06-02 03:13 UTC (permalink / raw)
To: Amit Kapila <amit.kapila16@gmail.com>; +Cc: Masahiko Sawada <sawada.mshk@gmail.com>; Hayato Kuroda (Fujitsu) <kuroda.hayato@fujitsu.com>; Duncan Sands <duncan.sands@deepbluecap.com>; pgsql-bugs@lists.postgresql.org <pgsql-bugs@lists.postgresql.org>
On Sat, 31 May 2025 at 10:20, Amit Kapila <amit.kapila16@gmail.com> wrote:
>
> On Fri, May 30, 2025 at 11:00 PM Masahiko Sawada <sawada.mshk@gmail.com> wrote:
> >
> > Thank you for updating the patch. Here are some review comments:
> >
> > ---
> > + /*
> > + * Make sure there's no cache pollution. Unlike the PG_TRY part,
> > + * this must be done unconditionally because the processing might
> > + * fail before we reach invalidation messages.
> > + */
> > + if (rbtxn_inval_all_cache(txn))
> > + InvalidateSystemCaches();
> > + else
> > + ReorderBufferExecuteInvalidations(txn->ninvalidations_distr,
> > +
> > txn->distributed_invalidations);
> > +
> >
> > If we don't need to execute the distributed inval message in an error
> > path other than detecting concurrent abort, we should describe the
> > reason.
> >
>
> The concurrent abort handling is done for streaming and prepared
> transactions, where we send the transaction changes to the client
> before we read COMMIT/COMMIT PREPARED/ROLLBACK/ROLLBACK PREPARED. Now,
> among these COMMIT/ROLLBACK PREPARED cases are handled as part of a
> prepared transaction case. For ROLLBACK, we will never perform any
> changes from the current transaction, so we don't need distributed
> invalidations to be executed. For COMMIT, if we encounter any errors
> while processing changes (this is when we reach the ERROR path, which
> is not a concurrent abort), then we will reprocess all changes and, at
> the end, execute both the current transaction and distributed
> invalidations. Now, one possibility is that if, after ERROR, the
> caller does slot_advance to skip the ERROR, then we will probably miss
> executing the distributed invalidations, leading to data loss
> afterwards. If the above theory is correct, then it is better to
> execute distributed invalidation even in non-concurrent-abort cases in
> the ERROR path.
One possible reason this scenario may not occur is that
pg_logical_slot_get_changes_guts uses a PG_CATCH block to handle
exceptions, during which it calls InvalidateSystemCaches to clear the
system cache. Because of this, I believe the scenario might not
actually happen.
@Sawada-san / others — Are there any other cases where this could still occur?
Regards,
Vignesh
^ permalink raw reply [nested|flat] 94+ messages in thread
* Re: Logical replication 'invalid memory alloc request size 1585837200' after upgrading to 17.5
@ 2025-06-02 07:12 Duncan Sands <duncan.sands@deepbluecap.com>
parent: Hayato Kuroda (Fujitsu) <kuroda.hayato@fujitsu.com>
0 siblings, 0 replies; 94+ messages in thread
From: Duncan Sands @ 2025-06-02 07:12 UTC (permalink / raw)
To: Hayato Kuroda (Fujitsu) <kuroda.hayato@fujitsu.com>; Masahiko Sawada <sawada.mshk@gmail.com>; Amit Kapila <amit.kapila16@gmail.com>; +Cc: pgsql-bugs@lists.postgresql.org <pgsql-bugs@lists.postgresql.org>
Dear Hayato Kuroda, I tried
v4-PG17-0001-Avoid-distributing-invalidation-messages-sev.patch and I can
confirm that it also resolves my original issue.
Best wishes, Duncan.
On 28/05/2025 14:27, Hayato Kuroda (Fujitsu) wrote:
> Dear Sawada-san, Amit,
>
>>> It can impact the performance for large transactions with fewer
>>> invalidations, especially the ones which has spilled changes because
>>> it needs to traverse the entire list of changes again at the end.
>>
>> Agreed.
>>
>>> The
>>> other idea would be to add new member(s) in ReorderBufferTXN to
>>> receive distributed invalidations. For adding the new member in
>>> ReorderBufferTXN: (a) in HEAD, it should be okay, (b) for
>>> backbranches, we may be able to add at the end, but we should check if
>>> there are any extensions using sizeof(ReorderBufferTxn) and if they
>>> are using what we need to do.
>>
>> If we can make sure that that change won't break the existing
>> extensions, I think this would be the most reasonable solution.
>
> Based on the discussion, I created PoC for master/PG17. Please see attached.
> The basic idea is to introduce the new queue which only contains distributed inval
> messages. Contents are consumed at end of transactions. I feel some of codes can
> be re-used so that internal functions are introduced. At least, it could pass
> regression tests and workloads discussed here.
>
> Best regards,
> Hayato Kuroda
> FUJITSU LIMITED
>
^ permalink raw reply [nested|flat] 94+ messages in thread
* Re: Logical replication 'invalid memory alloc request size 1585837200' after upgrading to 17.5
@ 2025-06-02 10:22 vignesh C <vignesh21@gmail.com>
parent: vignesh C <vignesh21@gmail.com>
0 siblings, 2 replies; 94+ messages in thread
From: vignesh C @ 2025-06-02 10:22 UTC (permalink / raw)
To: Masahiko Sawada <sawada.mshk@gmail.com>; +Cc: Amit Kapila <amit.kapila16@gmail.com>; Hayato Kuroda (Fujitsu) <kuroda.hayato@fujitsu.com>; Duncan Sands <duncan.sands@deepbluecap.com>; pgsql-bugs@lists.postgresql.org <pgsql-bugs@lists.postgresql.org>
On Sat, 31 May 2025 at 13:27, vignesh C <vignesh21@gmail.com> wrote:
>
> On Fri, 30 May 2025 at 23:00, Masahiko Sawada <sawada.mshk@gmail.com> wrote:
> >
> > Thank you for updating the patch. Here are some review comments:
> >
> > @@ -3439,9 +3464,27 @@ ReorderBufferAddInvalidations(ReorderBuffer
> > *rb, TransactionId xid,
> > XLogRecPtr lsn, Size nmsgs,
> > SharedInvalidationMessage *msgs)
> > {
> > - ReorderBufferTXN *txn;
> > + ReorderBufferAddInvalidationsExtended(rb, xid, lsn, nmsgs, msgs, false);
> > +}
> >
> > If the patch is the changes for master do we need to have an extended
> > version of ReorderBufferAddInvalidation()?
>
> This has been removed now and ReorderBufferAddDistributedInvalidtions
> has been added
>
> > ---
> > + /*
> > + * Make sure there's no cache pollution. Unlike the PG_TRY part,
> > + * this must be done unconditionally because the processing might
> > + * fail before we reach invalidation messages.
> > + */
> > + if (rbtxn_inval_all_cache(txn))
> > + InvalidateSystemCaches();
> > + else
> > + ReorderBufferExecuteInvalidations(txn->ninvalidations_distr,
> > +
> > txn->distributed_invalidations);
> > +
> >
> > If we don't need to execute the distributed inval message in an error
> > path other than detecting concurrent abort, we should describe the
> > reason.
>
> Removed it to keep it in the common error path
>
> > ---
> > Given that we don't account the memory usage of both
> > txn->invalidations and txn->distributed_invalidations, probably we can
> > have a lower limit, say 8MB (or lower?), to avoid memory exhaustion.
>
> Modified
>
> > ---
> > + if ((for_inval && !AllocSizeIsValid(req_mem_size)) ||
> > + rbtxn_inval_all_cache(txn))
> > {
> > - txn->ninvalidations = nmsgs;
> > - txn->invalidations = (SharedInvalidationMessage *)
> > - palloc(sizeof(SharedInvalidationMessage) * nmsgs);
> > - memcpy(txn->invalidations, msgs,
> > - sizeof(SharedInvalidationMessage) * nmsgs);
> > + txn->txn_flags |= RBTXN_INVAL_ALL_CACHE;
> > +
> > + if (*invalidations)
> > + {
> > + pfree(*invalidations);
> > + *invalidations = NULL;
> > + *ninvalidations = 0;
> > + }
> >
> > RBTXN_INVAL_ALL_CACHE seems to have an effect only on the distributed
> > inval messages. One question is do we need to care about the overflow
> > of txn->invalidations as well? If no, does it make sense to have a
> > separate function like ReorderBufferAddDistributedInvalidtions()
> > instead of having an extended version of
> > ReorderBufferAddInvalidations()? Some common routines can also be
> > declared as a static function if needed.
>
> Modified
>
> The attached v7 version patch has the changes for the same.
Here is the patch, including updates for the back branches.
The main difference from master is that the newly added structure
members are appended at the end in the back branches to preserve
compatibility. The invalidation addition logic remains consistent with
master: a new function, ReorderBufferAddDistributedInvalidations, has
been introduced to handle distributed invalidations. Shared logic
between ReorderBufferAddInvalidations and
ReorderBufferAddDistributedInvalidations has been factored out into a
common helper, ReorderBufferAddInvalidationsCommon. This approach
simplifies future merges and, in my assessment, does not introduce any
backward compatibility issues.
Regards,
Vignesh
Attachments:
[application/octet-stream] v8-PG15-0001-Fix-exponential-memory-allocation-issue-in-l.patch (12.0K, ../../CALDaNm2sGfKsZwm8rG132N_i0AvyxkYu8TL46kNcx7KwAUABxA@mail.gmail.com/2-v8-PG15-0001-Fix-exponential-memory-allocation-issue-in-l.patch)
download | inline diff:
From bc5b34c4f603a75869eb40a1a2b8574a1f056ff0 Mon Sep 17 00:00:00 2001
From: Vignesh <vignesh21@gmail.com>
Date: Mon, 2 Jun 2025 11:47:40 +0530
Subject: [PATCH v8-PG15] Fix exponential memory allocation issue in logical
decoding due to invalidation message distribution
Commit 9f21be08e8 added logic to distribute invalidation messages from
one transaction to all concurrent in-progress transactions. However,
since the source of each invalidation message was not tracked, transactions
could redundantly re-distribute already propagated messages. This led to an
exponential increase in invalidation messages and, ultimately, memory
allocation failures.
To fix this, a new field is introduced to track distributed invalidation
messages separately. This ensures that only transactions in the pre-existing
list receive the invalidations, preventing redundant propagation.
The accumulated distributed invalidations are now consumed at the end of
transaction processing in `ReorderBufferProcessTXN()`.
---
.../replication/logical/reorderbuffer.c | 178 ++++++++++++++----
src/backend/replication/logical/snapbuild.c | 5 +-
src/include/replication/reorderbuffer.h | 18 ++
3 files changed, 161 insertions(+), 40 deletions(-)
diff --git a/src/backend/replication/logical/reorderbuffer.c b/src/backend/replication/logical/reorderbuffer.c
index 835180e12f6..a5cd4f06f42 100644
--- a/src/backend/replication/logical/reorderbuffer.c
+++ b/src/backend/replication/logical/reorderbuffer.c
@@ -103,6 +103,7 @@
#include "storage/sinval.h"
#include "utils/builtins.h"
#include "utils/combocid.h"
+#include "utils/inval.h"
#include "utils/memdebug.h"
#include "utils/memutils.h"
#include "utils/rel.h"
@@ -457,6 +458,12 @@ ReorderBufferReturnTXN(ReorderBuffer *rb, ReorderBufferTXN *txn)
txn->invalidations = NULL;
}
+ if (txn->distributed_invalidations)
+ {
+ pfree(txn->distributed_invalidations);
+ txn->distributed_invalidations = NULL;
+ }
+
/* Reset the toast hash */
ReorderBufferToastReset(rb, txn);
@@ -2503,7 +2510,14 @@ ReorderBufferProcessTXN(ReorderBuffer *rb, ReorderBufferTXN *txn,
AbortCurrentTransaction();
/* make sure there's no cache pollution */
- ReorderBufferExecuteInvalidations(txn->ninvalidations, txn->invalidations);
+ if (rbtxn_inval_all_cache(txn))
+ InvalidateSystemCaches();
+ else
+ {
+ ReorderBufferExecuteInvalidations(txn->ninvalidations, txn->invalidations);
+ ReorderBufferExecuteInvalidations(txn->ninvalidations_distr,
+ txn->distributed_invalidations);
+ }
if (using_subtxn)
RollbackAndReleaseCurrentSubTransaction();
@@ -2579,6 +2593,20 @@ ReorderBufferProcessTXN(ReorderBuffer *rb, ReorderBufferTXN *txn,
errdata = NULL;
curtxn->concurrent_abort = true;
+ /*
+ * Make sure there's no cache pollution. Unlike the PG_TRY section,
+ * this must be done unconditionally, since processing may fail
+ * before we reach the invalidation messages. This isn't needed
+ * in non-rollback cases, as the system cache is cleared either
+ * by the API's error handling or when the walsender exits.
+ */
+ if (rbtxn_inval_all_cache(txn))
+ InvalidateSystemCaches();
+ else
+ ReorderBufferExecuteInvalidations(txn->ninvalidations_distr,
+ txn->distributed_invalidations);
+
+
/* Reset the TXN so that it is allowed to stream remaining data. */
ReorderBufferResetTXN(rb, txn, snapshot_now,
command_id, prev_lsn,
@@ -3223,59 +3251,57 @@ ReorderBufferAddNewTupleCids(ReorderBuffer *rb, TransactionId xid,
}
/*
- * Accumulate the invalidations for executing them later.
+ * Helper function to store invalidation messages and queue them as a
+ * change in the reorder buffer. Handles both regular and distributed
+ * invalidations.
*
- * This needs to be called for each XLOG_XACT_INVALIDATIONS message and
- * accumulates all the invalidation messages in the toplevel transaction, if
- * available, otherwise in the current transaction, as well as in the form of
- * change in reorder buffer. We require to record it in form of the change
+ * This accumulates all the invalidation messages in the toplevel transaction,
+ * if available, otherwise in the current transaction, as well as in the form
+ * of change in reorder buffer. We require to record it in form of the change
* so that we can execute only the required invalidations instead of executing
* all the invalidations on each CommandId increment. We also need to
* accumulate these in the txn buffer because in some cases where we skip
* processing the transaction (see ReorderBufferForget), we need to execute
* all the invalidations together.
*/
-void
-ReorderBufferAddInvalidations(ReorderBuffer *rb, TransactionId xid,
- XLogRecPtr lsn, Size nmsgs,
- SharedInvalidationMessage *msgs)
+static void
+ReorderBufferAddInvalidationsCommon(ReorderBuffer *rb, TransactionId xid,
+ XLogRecPtr lsn, Size nmsgs,
+ SharedInvalidationMessage *msgs,
+ ReorderBufferTXN *txn,
+ bool for_inval)
{
- ReorderBufferTXN *txn;
MemoryContext oldcontext;
ReorderBufferChange *change;
- txn = ReorderBufferTXNByXid(rb, xid, true, NULL, lsn, true);
-
oldcontext = MemoryContextSwitchTo(rb->context);
- /*
- * Collect all the invalidations under the top transaction, if available,
- * so that we can execute them all together. See comments atop this
- * function.
- */
- if (txn->toptxn)
- txn = txn->toptxn;
+ if (!for_inval || (for_inval && !rbtxn_inval_all_cache(txn)))
+ {
+ uint32 *ninvalidations;
+ SharedInvalidationMessage **invalidations;
- Assert(nmsgs > 0);
+ ninvalidations = for_inval ? &txn->ninvalidations_distr : &txn->ninvalidations;
+ invalidations = for_inval ? &txn->distributed_invalidations : &txn->invalidations;
- /* Accumulate invalidations. */
- if (txn->ninvalidations == 0)
- {
- txn->ninvalidations = nmsgs;
- txn->invalidations = (SharedInvalidationMessage *)
- palloc(sizeof(SharedInvalidationMessage) * nmsgs);
- memcpy(txn->invalidations, msgs,
- sizeof(SharedInvalidationMessage) * nmsgs);
- }
- else
- {
- txn->invalidations = (SharedInvalidationMessage *)
- repalloc(txn->invalidations, sizeof(SharedInvalidationMessage) *
- (txn->ninvalidations + nmsgs));
+ /* Accumulate invalidations. */
+ if (*ninvalidations == 0)
+ {
+ *ninvalidations = nmsgs;
+ *invalidations = (SharedInvalidationMessage *)
+ palloc(sizeof(SharedInvalidationMessage) * nmsgs);
+ memcpy(*invalidations, msgs, sizeof(SharedInvalidationMessage) * nmsgs);
+ }
+ else
+ {
+ *invalidations = (SharedInvalidationMessage *)
+ repalloc(*invalidations, sizeof(SharedInvalidationMessage) *
+ (*ninvalidations + nmsgs));
- memcpy(txn->invalidations + txn->ninvalidations, msgs,
- nmsgs * sizeof(SharedInvalidationMessage));
- txn->ninvalidations += nmsgs;
+ memcpy(*invalidations + *ninvalidations, msgs,
+ nmsgs * sizeof(SharedInvalidationMessage));
+ *ninvalidations += nmsgs;
+ }
}
change = ReorderBufferGetChange(rb);
@@ -3291,6 +3317,82 @@ ReorderBufferAddInvalidations(ReorderBuffer *rb, TransactionId xid,
MemoryContextSwitchTo(oldcontext);
}
+/*
+ * Accumulate the invalidations for executing them later.
+ *
+ * This needs to be called for each XLOG_XACT_INVALIDATIONS message.
+ */
+void
+ReorderBufferAddInvalidations(ReorderBuffer *rb, TransactionId xid,
+ XLogRecPtr lsn, Size nmsgs,
+ SharedInvalidationMessage *msgs)
+{
+ ReorderBufferTXN *txn;
+
+ txn = ReorderBufferTXNByXid(rb, xid, true, NULL, lsn, true);
+
+ /*
+ * Collect all the invalidations under the top transaction, if available,
+ * so that we can execute them all together. See comments atop
+ * ReorderBufferAddInvalidationsCommon function.
+ */
+ if (txn->toptxn)
+ txn = txn->toptxn;
+
+ Assert(nmsgs > 0);
+
+ ReorderBufferAddInvalidationsCommon(rb, xid, lsn, nmsgs, msgs, txn, false);
+}
+
+/*
+ * Accumulate the invalidations sent by committed transactions for executing
+ * them later.
+ *
+ * This needs to be called by committed transactions to distribute the
+ * invalidations to the in-progress transactions.
+ */
+void
+ReorderBufferAddDistributedInvalidtions(ReorderBuffer *rb, TransactionId xid,
+ XLogRecPtr lsn, Size nmsgs,
+ SharedInvalidationMessage *msgs)
+{
+ ReorderBufferTXN *txn;
+ Size req_mem_size;
+
+ txn = ReorderBufferTXNByXid(rb, xid, true, NULL, lsn, true);
+
+ /*
+ * Collect all the invalidations under the top transaction, if available,
+ * so that we can execute them all together. See comments
+ * ReorderBufferAddInvalidationsCommon ReorderBufferAddInvalidations.
+ */
+ if (txn->toptxn)
+ txn = txn->toptxn;
+
+ Assert(nmsgs > 0);
+
+ req_mem_size = sizeof(SharedInvalidationMessage) * (txn->ninvalidations_distr + nmsgs);
+
+ /*
+ * If the number of invalidation messages is larger than 8MB, it's more
+ * efficient to invalidate the entire cache rather than processing each
+ * message individually.
+ */
+ if (req_mem_size > (8 * 1024 * 1024) || rbtxn_inval_all_cache(txn))
+ {
+ txn->txn_flags |= RBTXN_INVAL_ALL_CACHE;
+
+ if (txn->distributed_invalidations)
+ {
+ pfree(txn->distributed_invalidations);
+ txn->distributed_invalidations = NULL;
+ txn->ninvalidations_distr = 0;
+ }
+ }
+
+ ReorderBufferAddInvalidationsCommon(rb, xid, lsn, nmsgs, msgs, txn, true);
+}
+
/*
* Apply all invalidations we know. Possibly we only need parts at this point
* in the changestream but we don't know which those are.
diff --git a/src/backend/replication/logical/snapbuild.c b/src/backend/replication/logical/snapbuild.c
index 2ca8e6e9aff..e693db3b3b8 100644
--- a/src/backend/replication/logical/snapbuild.c
+++ b/src/backend/replication/logical/snapbuild.c
@@ -938,8 +938,9 @@ SnapBuildDistributeSnapshotAndInval(SnapBuild *builder, XLogRecPtr lsn, Transact
{
Assert(msgs != NULL);
- ReorderBufferAddInvalidations(builder->reorder, txn->xid, lsn,
- ninvalidations, msgs);
+ ReorderBufferAddDistributedInvalidtions(builder->reorder,
+ txn->xid, lsn,
+ ninvalidations, msgs);
}
}
}
diff --git a/src/include/replication/reorderbuffer.h b/src/include/replication/reorderbuffer.h
index 402bb7a2728..674e873ef8e 100644
--- a/src/include/replication/reorderbuffer.h
+++ b/src/include/replication/reorderbuffer.h
@@ -176,6 +176,7 @@ typedef struct ReorderBufferChange
#define RBTXN_HAS_PARTIAL_CHANGE 0x0020
#define RBTXN_PREPARE 0x0040
#define RBTXN_SKIPPED_PREPARE 0x0080
+#define RBTXN_INVAL_ALL_CACHE 0x0100
/* Does the transaction have catalog changes? */
#define rbtxn_has_catalog_changes(txn) \
@@ -207,6 +208,12 @@ typedef struct ReorderBufferChange
((txn)->txn_flags & RBTXN_HAS_PARTIAL_CHANGE) != 0 \
)
+/* Should the complete cache be invalidated? */
+#define rbtxn_inval_all_cache(txn) \
+( \
+ ((txn)->txn_flags & RBTXN_INVAL_ALL_CACHE) != 0 \
+)
+
/*
* Has this transaction been streamed to downstream?
*
@@ -395,6 +402,13 @@ typedef struct ReorderBufferTXN
* Private data pointer of the output plugin.
*/
void *output_plugin_private;
+
+ /*
+ * Stores the cache invalidation messages distributed by the committing
+ * transaction.
+ */
+ uint32 ninvalidations_distr;
+ SharedInvalidationMessage *distributed_invalidations;
} ReorderBufferTXN;
/* so we can define the callbacks used inside struct ReorderBuffer itself */
@@ -661,6 +675,10 @@ extern void ReorderBufferAddNewTupleCids(ReorderBuffer *, TransactionId, XLogRec
CommandId cmin, CommandId cmax, CommandId combocid);
extern void ReorderBufferAddInvalidations(ReorderBuffer *, TransactionId, XLogRecPtr lsn,
Size nmsgs, SharedInvalidationMessage *msgs);
+extern void ReorderBufferAddDistributedInvalidtions(ReorderBuffer *rb,
+ TransactionId xid,
+ XLogRecPtr lsn, Size nmsgs,
+ SharedInvalidationMessage *msgs);
extern void ReorderBufferImmediateInvalidation(ReorderBuffer *, uint32 ninvalidations,
SharedInvalidationMessage *invalidations);
extern void ReorderBufferProcessXid(ReorderBuffer *, TransactionId xid, XLogRecPtr lsn);
--
2.43.0
[application/octet-stream] v8-PG14-0001-Fix-exponential-memory-allocation-issue-in-l.patch (11.9K, ../../CALDaNm2sGfKsZwm8rG132N_i0AvyxkYu8TL46kNcx7KwAUABxA@mail.gmail.com/3-v8-PG14-0001-Fix-exponential-memory-allocation-issue-in-l.patch)
download | inline diff:
From fb71bf11bf7b3e823af96a39623a1d530e4e0252 Mon Sep 17 00:00:00 2001
From: Vignesh <vignesh21@gmail.com>
Date: Mon, 2 Jun 2025 12:13:05 +0530
Subject: [PATCH v8-PG14] Fix exponential memory allocation issue in logical
decoding due to invalidation message distribution
Commit 0434033e8b added logic to distribute invalidation messages from
one transaction to all concurrent in-progress transactions. However,
since the source of each invalidation message was not tracked, transactions
could redundantly re-distribute already propagated messages. This led to an
exponential increase in invalidation messages and, ultimately, memory
allocation failures.
To fix this, a new field is introduced to track distributed invalidation
messages separately. This ensures that only transactions in the pre-existing
list receive the invalidations, preventing redundant propagation.
The accumulated distributed invalidations are now consumed at the end of
transaction processing in `ReorderBufferProcessTXN()`.
---
.../replication/logical/reorderbuffer.c | 178 ++++++++++++++----
src/backend/replication/logical/snapbuild.c | 5 +-
src/include/replication/reorderbuffer.h | 18 ++
3 files changed, 161 insertions(+), 40 deletions(-)
diff --git a/src/backend/replication/logical/reorderbuffer.c b/src/backend/replication/logical/reorderbuffer.c
index a52b51a5e7d..ef869a3504a 100644
--- a/src/backend/replication/logical/reorderbuffer.c
+++ b/src/backend/replication/logical/reorderbuffer.c
@@ -103,6 +103,7 @@
#include "storage/sinval.h"
#include "utils/builtins.h"
#include "utils/combocid.h"
+#include "utils/inval.h"
#include "utils/memdebug.h"
#include "utils/memutils.h"
#include "utils/rel.h"
@@ -453,6 +454,12 @@ ReorderBufferReturnTXN(ReorderBuffer *rb, ReorderBufferTXN *txn)
txn->invalidations = NULL;
}
+ if (txn->distributed_invalidations)
+ {
+ pfree(txn->distributed_invalidations);
+ txn->distributed_invalidations = NULL;
+ }
+
/* Reset the toast hash */
ReorderBufferToastReset(rb, txn);
@@ -2500,7 +2507,14 @@ ReorderBufferProcessTXN(ReorderBuffer *rb, ReorderBufferTXN *txn,
AbortCurrentTransaction();
/* make sure there's no cache pollution */
- ReorderBufferExecuteInvalidations(txn->ninvalidations, txn->invalidations);
+ if (rbtxn_inval_all_cache(txn))
+ InvalidateSystemCaches();
+ else
+ {
+ ReorderBufferExecuteInvalidations(txn->ninvalidations, txn->invalidations);
+ ReorderBufferExecuteInvalidations(txn->ninvalidations_distr,
+ txn->distributed_invalidations);
+ }
if (using_subtxn)
RollbackAndReleaseCurrentSubTransaction();
@@ -2576,6 +2590,20 @@ ReorderBufferProcessTXN(ReorderBuffer *rb, ReorderBufferTXN *txn,
errdata = NULL;
curtxn->concurrent_abort = true;
+ /*
+ * Make sure there's no cache pollution. Unlike the PG_TRY section,
+ * this must be done unconditionally, since processing may fail
+ * before we reach the invalidation messages. This isn't needed
+ * in non-rollback cases, as the system cache is cleared either
+ * by the API's error handling or when the walsender exits.
+ */
+ if (rbtxn_inval_all_cache(txn))
+ InvalidateSystemCaches();
+ else
+ ReorderBufferExecuteInvalidations(txn->ninvalidations_distr,
+ txn->distributed_invalidations);
+
+
/* Reset the TXN so that it is allowed to stream remaining data. */
ReorderBufferResetTXN(rb, txn, snapshot_now,
command_id, prev_lsn,
@@ -3219,59 +3247,57 @@ ReorderBufferAddNewTupleCids(ReorderBuffer *rb, TransactionId xid,
}
/*
- * Accumulate the invalidations for executing them later.
+ * Helper function to store invalidation messages and queue them as a
+ * change in the reorder buffer. Handles both regular and distributed
+ * invalidations.
*
- * This needs to be called for each XLOG_XACT_INVALIDATIONS message and
- * accumulates all the invalidation messages in the toplevel transaction, if
- * available, otherwise in the current transaction, as well as in the form of
- * change in reorder buffer. We require to record it in form of the change
+ * This accumulates all the invalidation messages in the toplevel transaction,
+ * if available, otherwise in the current transaction, as well as in the form
+ * of change in reorder buffer. We require to record it in form of the change
* so that we can execute only the required invalidations instead of executing
* all the invalidations on each CommandId increment. We also need to
* accumulate these in the txn buffer because in some cases where we skip
* processing the transaction (see ReorderBufferForget), we need to execute
* all the invalidations together.
*/
-void
-ReorderBufferAddInvalidations(ReorderBuffer *rb, TransactionId xid,
- XLogRecPtr lsn, Size nmsgs,
- SharedInvalidationMessage *msgs)
+static void
+ReorderBufferAddInvalidationsCommon(ReorderBuffer *rb, TransactionId xid,
+ XLogRecPtr lsn, Size nmsgs,
+ SharedInvalidationMessage *msgs,
+ ReorderBufferTXN *txn,
+ bool for_inval)
{
- ReorderBufferTXN *txn;
MemoryContext oldcontext;
ReorderBufferChange *change;
- txn = ReorderBufferTXNByXid(rb, xid, true, NULL, lsn, true);
-
oldcontext = MemoryContextSwitchTo(rb->context);
- /*
- * Collect all the invalidations under the top transaction, if available,
- * so that we can execute them all together. See comments atop this
- * function.
- */
- if (txn->toptxn)
- txn = txn->toptxn;
+ if (!for_inval || (for_inval && !rbtxn_inval_all_cache(txn)))
+ {
+ uint32 *ninvalidations;
+ SharedInvalidationMessage **invalidations;
- Assert(nmsgs > 0);
+ ninvalidations = for_inval ? &txn->ninvalidations_distr : &txn->ninvalidations;
+ invalidations = for_inval ? &txn->distributed_invalidations : &txn->invalidations;
- /* Accumulate invalidations. */
- if (txn->ninvalidations == 0)
- {
- txn->ninvalidations = nmsgs;
- txn->invalidations = (SharedInvalidationMessage *)
- palloc(sizeof(SharedInvalidationMessage) * nmsgs);
- memcpy(txn->invalidations, msgs,
- sizeof(SharedInvalidationMessage) * nmsgs);
- }
- else
- {
- txn->invalidations = (SharedInvalidationMessage *)
- repalloc(txn->invalidations, sizeof(SharedInvalidationMessage) *
- (txn->ninvalidations + nmsgs));
+ /* Accumulate invalidations. */
+ if (*ninvalidations == 0)
+ {
+ *ninvalidations = nmsgs;
+ *invalidations = (SharedInvalidationMessage *)
+ palloc(sizeof(SharedInvalidationMessage) * nmsgs);
+ memcpy(*invalidations, msgs, sizeof(SharedInvalidationMessage) * nmsgs);
+ }
+ else
+ {
+ *invalidations = (SharedInvalidationMessage *)
+ repalloc(*invalidations, sizeof(SharedInvalidationMessage) *
+ (*ninvalidations + nmsgs));
- memcpy(txn->invalidations + txn->ninvalidations, msgs,
- nmsgs * sizeof(SharedInvalidationMessage));
- txn->ninvalidations += nmsgs;
+ memcpy(*invalidations + *ninvalidations, msgs,
+ nmsgs * sizeof(SharedInvalidationMessage));
+ *ninvalidations += nmsgs;
+ }
}
change = ReorderBufferGetChange(rb);
@@ -3287,6 +3313,82 @@ ReorderBufferAddInvalidations(ReorderBuffer *rb, TransactionId xid,
MemoryContextSwitchTo(oldcontext);
}
+/*
+ * Accumulate the invalidations for executing them later.
+ *
+ * This needs to be called for each XLOG_XACT_INVALIDATIONS message.
+ */
+void
+ReorderBufferAddInvalidations(ReorderBuffer *rb, TransactionId xid,
+ XLogRecPtr lsn, Size nmsgs,
+ SharedInvalidationMessage *msgs)
+{
+ ReorderBufferTXN *txn;
+
+ txn = ReorderBufferTXNByXid(rb, xid, true, NULL, lsn, true);
+
+ /*
+ * Collect all the invalidations under the top transaction, if available,
+ * so that we can execute them all together. See comments atop
+ * ReorderBufferAddInvalidationsCommon function.
+ */
+ if (txn->toptxn)
+ txn = txn->toptxn;
+
+ Assert(nmsgs > 0);
+
+ ReorderBufferAddInvalidationsCommon(rb, xid, lsn, nmsgs, msgs, txn, false);
+}
+
+/*
+ * Accumulate the invalidations sent by committed transactions for executing
+ * them later.
+ *
+ * This needs to be called by committed transactions to distribute the
+ * invalidations to the in-progress transactions.
+ */
+void
+ReorderBufferAddDistributedInvalidtions(ReorderBuffer *rb, TransactionId xid,
+ XLogRecPtr lsn, Size nmsgs,
+ SharedInvalidationMessage *msgs)
+{
+ ReorderBufferTXN *txn;
+ Size req_mem_size;
+
+ txn = ReorderBufferTXNByXid(rb, xid, true, NULL, lsn, true);
+
+ /*
+ * Collect all the invalidations under the top transaction, if available,
+ * so that we can execute them all together. See comments
+ * ReorderBufferAddInvalidationsCommon ReorderBufferAddInvalidations.
+ */
+ if (txn->toptxn)
+ txn = txn->toptxn;
+
+ Assert(nmsgs > 0);
+
+ req_mem_size = sizeof(SharedInvalidationMessage) * (txn->ninvalidations_distr + nmsgs);
+
+ /*
+ * If the number of invalidation messages is larger than 8MB, it's more
+ * efficient to invalidate the entire cache rather than processing each
+ * message individually.
+ */
+ if (req_mem_size > (8 * 1024 * 1024) || rbtxn_inval_all_cache(txn))
+ {
+ txn->txn_flags |= RBTXN_INVAL_ALL_CACHE;
+
+ if (txn->distributed_invalidations)
+ {
+ pfree(txn->distributed_invalidations);
+ txn->distributed_invalidations = NULL;
+ txn->ninvalidations_distr = 0;
+ }
+ }
+
+ ReorderBufferAddInvalidationsCommon(rb, xid, lsn, nmsgs, msgs, txn, true);
+}
+
/*
* Apply all invalidations we know. Possibly we only need parts at this point
* in the changestream but we don't know which those are.
diff --git a/src/backend/replication/logical/snapbuild.c b/src/backend/replication/logical/snapbuild.c
index 6025f2727ae..734d8f2b37e 100644
--- a/src/backend/replication/logical/snapbuild.c
+++ b/src/backend/replication/logical/snapbuild.c
@@ -929,8 +929,9 @@ SnapBuildDistributeSnapshotAndInval(SnapBuild *builder, XLogRecPtr lsn, Transact
{
Assert(msgs != NULL);
- ReorderBufferAddInvalidations(builder->reorder, txn->xid, lsn,
- ninvalidations, msgs);
+ ReorderBufferAddDistributedInvalidtions(builder->reorder,
+ txn->xid, lsn,
+ ninvalidations, msgs);
}
}
}
diff --git a/src/include/replication/reorderbuffer.h b/src/include/replication/reorderbuffer.h
index d399975e8a5..7cce50e91d7 100644
--- a/src/include/replication/reorderbuffer.h
+++ b/src/include/replication/reorderbuffer.h
@@ -176,6 +176,7 @@ typedef struct ReorderBufferChange
#define RBTXN_HAS_PARTIAL_CHANGE 0x0020
#define RBTXN_PREPARE 0x0040
#define RBTXN_SKIPPED_PREPARE 0x0080
+#define RBTXN_INVAL_ALL_CACHE 0x0100
/* Does the transaction have catalog changes? */
#define rbtxn_has_catalog_changes(txn) \
@@ -207,6 +208,12 @@ typedef struct ReorderBufferChange
((txn)->txn_flags & RBTXN_HAS_PARTIAL_CHANGE) != 0 \
)
+/* Should the complete cache be invalidated? */
+#define rbtxn_inval_all_cache(txn) \
+( \
+ ((txn)->txn_flags & RBTXN_INVAL_ALL_CACHE) != 0 \
+)
+
/*
* Has this transaction been streamed to downstream?
*
@@ -391,6 +398,13 @@ typedef struct ReorderBufferTXN
* Private data pointer of the output plugin.
*/
void *output_plugin_private;
+
+ /*
+ * Stores the cache invalidation messages distributed by the committing
+ * transaction.
+ */
+ uint32 ninvalidations_distr;
+ SharedInvalidationMessage *distributed_invalidations;
} ReorderBufferTXN;
/* so we can define the callbacks used inside struct ReorderBuffer itself */
@@ -657,6 +671,10 @@ void ReorderBufferAddNewTupleCids(ReorderBuffer *, TransactionId, XLogRecPtr ls
CommandId cmin, CommandId cmax, CommandId combocid);
void ReorderBufferAddInvalidations(ReorderBuffer *, TransactionId, XLogRecPtr lsn,
Size nmsgs, SharedInvalidationMessage *msgs);
+void ReorderBufferAddDistributedInvalidtions(ReorderBuffer *rb,
+ TransactionId xid,
+ XLogRecPtr lsn, Size nmsgs,
+ SharedInvalidationMessage *msgs);
void ReorderBufferImmediateInvalidation(ReorderBuffer *, uint32 ninvalidations,
SharedInvalidationMessage *invalidations);
void ReorderBufferProcessXid(ReorderBuffer *, TransactionId xid, XLogRecPtr lsn);
--
2.43.0
[application/octet-stream] v8-PG13-0001-Fix-exponential-memory-allocation-issue-in-l.patch (11.1K, ../../CALDaNm2sGfKsZwm8rG132N_i0AvyxkYu8TL46kNcx7KwAUABxA@mail.gmail.com/4-v8-PG13-0001-Fix-exponential-memory-allocation-issue-in-l.patch)
download | inline diff:
From 06f46c3c5db9133ee3723fd484c5759cc99d3179 Mon Sep 17 00:00:00 2001
From: Vignesh <vignesh21@gmail.com>
Date: Mon, 2 Jun 2025 13:53:46 +0530
Subject: [PATCH v8-PG13] Fix exponential memory allocation issue in logical
decoding due to invalidation message distribution
Commit 247ee94150 added logic to distribute invalidation messages from
one transaction to all concurrent in-progress transactions. However,
since the source of each invalidation message was not tracked, transactions
could redundantly re-distribute already propagated messages. This led to an
exponential increase in invalidation messages and, ultimately, memory
allocation failures.
To fix this, a new field is introduced to track distributed invalidation
messages separately. This ensures that only transactions in the pre-existing
list receive the invalidations, preventing redundant propagation.
The accumulated distributed invalidations are now consumed at the end of
transaction processing in `ReorderBufferProcessTXN()`.
---
.../replication/logical/reorderbuffer.c | 136 ++++++++++++++----
src/backend/replication/logical/snapbuild.c | 5 +-
src/include/replication/reorderbuffer.h | 18 +++
3 files changed, 131 insertions(+), 28 deletions(-)
diff --git a/src/backend/replication/logical/reorderbuffer.c b/src/backend/replication/logical/reorderbuffer.c
index fa9413fa2a0..447ab745630 100644
--- a/src/backend/replication/logical/reorderbuffer.c
+++ b/src/backend/replication/logical/reorderbuffer.c
@@ -103,6 +103,7 @@
#include "storage/sinval.h"
#include "utils/builtins.h"
#include "utils/combocid.h"
+#include "utils/inval.h"
#include "utils/memdebug.h"
#include "utils/memutils.h"
#include "utils/rel.h"
@@ -220,7 +221,8 @@ static void ReorderBufferIterTXNInit(ReorderBuffer *rb, ReorderBufferTXN *txn,
static ReorderBufferChange *ReorderBufferIterTXNNext(ReorderBuffer *rb, ReorderBufferIterTXNState *state);
static void ReorderBufferIterTXNFinish(ReorderBuffer *rb,
ReorderBufferIterTXNState *state);
-static void ReorderBufferExecuteInvalidations(ReorderBuffer *rb, ReorderBufferTXN *txn);
+static void ReorderBufferExecuteInvalidations(ReorderBuffer *rb, ReorderBufferTXN *txn,
+ bool is_distributed);
/*
* ---------------------------------------
@@ -406,6 +408,12 @@ ReorderBufferReturnTXN(ReorderBuffer *rb, ReorderBufferTXN *txn)
txn->invalidations = NULL;
}
+ if (txn->distributed_invalidations)
+ {
+ pfree(txn->distributed_invalidations);
+ txn->distributed_invalidations = NULL;
+ }
+
/* Reset the toast hash */
ReorderBufferToastReset(rb, txn);
@@ -1883,7 +1891,7 @@ ReorderBufferCommit(ReorderBuffer *rb, TransactionId xid,
* see new catalog contents, so execute all
* invalidations.
*/
- ReorderBufferExecuteInvalidations(rb, txn);
+ ReorderBufferExecuteInvalidations(rb, txn, false);
}
break;
@@ -1921,7 +1929,13 @@ ReorderBufferCommit(ReorderBuffer *rb, TransactionId xid,
AbortCurrentTransaction();
/* make sure there's no cache pollution */
- ReorderBufferExecuteInvalidations(rb, txn);
+ if (rbtxn_inval_all_cache(txn))
+ InvalidateSystemCaches();
+ else
+ {
+ ReorderBufferExecuteInvalidations(rb, txn, false);
+ ReorderBufferExecuteInvalidations(rb, txn, true);
+ }
if (using_subtxn)
RollbackAndReleaseCurrentSubTransaction();
@@ -1947,7 +1961,7 @@ ReorderBufferCommit(ReorderBuffer *rb, TransactionId xid,
AbortCurrentTransaction();
/* make sure there's no cache pollution */
- ReorderBufferExecuteInvalidations(rb, txn);
+ ReorderBufferExecuteInvalidations(rb, txn, false);
if (using_subtxn)
RollbackAndReleaseCurrentSubTransaction();
@@ -2253,6 +2267,42 @@ ReorderBufferAddNewTupleCids(ReorderBuffer *rb, TransactionId xid,
txn->ntuplecids++;
}
+/*
+ * Helper function to store invalidation messages, handles both regular and
+ * distributed invalidations.
+ */
+static void
+ReorderBufferAddInvalidationsCommon(ReorderBuffer *rb, TransactionId xid,
+ XLogRecPtr lsn, Size nmsgs,
+ SharedInvalidationMessage *msgs,
+ ReorderBufferTXN *txn, bool for_inval)
+{
+ MemoryContext oldcontext;
+ uint32 *ninvalidations;
+ SharedInvalidationMessage **invalidations;
+
+ oldcontext = MemoryContextSwitchTo(rb->context);
+
+ ninvalidations = for_inval ? &txn->ninvalidations_distr : &txn->ninvalidations;
+ invalidations = for_inval ? &txn->distributed_invalidations : &txn->invalidations;
+
+ /* Accumulate invalidations. */
+ if (*ninvalidations == 0)
+ {
+ *ninvalidations = nmsgs;
+ *invalidations = (SharedInvalidationMessage *)
+ palloc(sizeof(SharedInvalidationMessage) * nmsgs);
+ memcpy(*invalidations, msgs, sizeof(SharedInvalidationMessage) * nmsgs);
+ }
+ else
+ {
+ *invalidations = (SharedInvalidationMessage *)
+ repalloc(*invalidations, sizeof(SharedInvalidationMessage) *
+ (*ninvalidations + nmsgs));
+ }
+ MemoryContextSwitchTo(oldcontext);
+}
+
/*
* Setup the invalidation of the toplevel transaction.
*
@@ -2264,12 +2314,9 @@ ReorderBufferAddInvalidations(ReorderBuffer *rb, TransactionId xid,
SharedInvalidationMessage *msgs)
{
ReorderBufferTXN *txn;
- MemoryContext oldcontext;
txn = ReorderBufferTXNByXid(rb, xid, true, NULL, lsn, true);
- oldcontext = MemoryContextSwitchTo(rb->context);
-
/*
* Collect all the invalidations under the top transaction, if available,
* so that we can execute them all together.
@@ -2282,27 +2329,58 @@ ReorderBufferAddInvalidations(ReorderBuffer *rb, TransactionId xid,
Assert(nmsgs > 0);
- /* Accumulate invalidations. */
- if (txn->ninvalidations == 0)
+ ReorderBufferAddInvalidationsCommon(rb, xid, lsn, nmsgs, msgs, txn, false);
+}
+
+/*
+ * Accumulate the invalidations sent by committed transactions for executing
+ * them later.
+ *
+ * This needs to be called by committed transactions to distribute the
+ * invalidations to the in-progress transactions.
+ */
+void
+ReorderBufferAddDistributedInvalidtions(ReorderBuffer *rb, TransactionId xid,
+ XLogRecPtr lsn, Size nmsgs,
+ SharedInvalidationMessage *msgs)
+{
+ ReorderBufferTXN *txn;
+ Size req_mem_size;
+
+ txn = ReorderBufferTXNByXid(rb, xid, true, NULL, lsn, true);
+
+ /*
+ * Collect all the invalidations under the top transaction, if available,
+ * so that we can execute them all together.
+ */
+ if (txn->toplevel_xid)
{
- txn->ninvalidations = nmsgs;
- txn->invalidations = (SharedInvalidationMessage *)
- palloc(sizeof(SharedInvalidationMessage) * nmsgs);
- memcpy(txn->invalidations, msgs,
- sizeof(SharedInvalidationMessage) * nmsgs);
+ txn = ReorderBufferTXNByXid(rb, txn->toplevel_xid, true, NULL, lsn,
+ true);
}
- else
+
+ Assert(nmsgs > 0);
+
+ req_mem_size = sizeof(SharedInvalidationMessage) * (txn->ninvalidations_distr + nmsgs);
+
+ /*
+ * If the number of invalidation messages is larger than 8MB, it's more
+ * efficient to invalidate the entire cache rather than processing each
+ * message individually.
+ */
+ if (req_mem_size > (8 * 1024 * 1024) || rbtxn_inval_all_cache(txn))
{
- txn->invalidations = (SharedInvalidationMessage *)
- repalloc(txn->invalidations, sizeof(SharedInvalidationMessage) *
- (txn->ninvalidations + nmsgs));
+ txn->txn_flags |= RBTXN_INVAL_ALL_CACHE;
- memcpy(txn->invalidations + txn->ninvalidations, msgs,
- nmsgs * sizeof(SharedInvalidationMessage));
- txn->ninvalidations += nmsgs;
+ if (txn->distributed_invalidations)
+ {
+ pfree(txn->distributed_invalidations);
+ txn->distributed_invalidations = NULL;
+ txn->ninvalidations_distr = 0;
+ }
}
-
- MemoryContextSwitchTo(oldcontext);
+ else
+ ReorderBufferAddInvalidationsCommon(rb, xid, lsn, nmsgs, msgs, txn, true);
}
/*
@@ -2310,12 +2388,18 @@ ReorderBufferAddInvalidations(ReorderBuffer *rb, TransactionId xid,
* in the changestream but we don't know which those are.
*/
static void
-ReorderBufferExecuteInvalidations(ReorderBuffer *rb, ReorderBufferTXN *txn)
+ReorderBufferExecuteInvalidations(ReorderBuffer *rb, ReorderBufferTXN *txn,
+ bool is_distributed)
{
int i;
+ uint32 ninvalidations;
+ SharedInvalidationMessage *invalidations;
- for (i = 0; i < txn->ninvalidations; i++)
- LocalExecuteInvalidationMessage(&txn->invalidations[i]);
+ ninvalidations = is_distributed ? txn->ninvalidations_distr : txn->ninvalidations;
+ invalidations = is_distributed ? txn->distributed_invalidations : txn->invalidations;
+
+ for (i = 0; i < ninvalidations; i++)
+ LocalExecuteInvalidationMessage(&invalidations[i]);
}
/*
diff --git a/src/backend/replication/logical/snapbuild.c b/src/backend/replication/logical/snapbuild.c
index 3bda41c5251..86e508e14be 100644
--- a/src/backend/replication/logical/snapbuild.c
+++ b/src/backend/replication/logical/snapbuild.c
@@ -940,8 +940,9 @@ SnapBuildDistributeSnapshotAndInval(SnapBuild *builder, XLogRecPtr lsn, Transact
{
Assert(msgs != NULL);
- ReorderBufferAddInvalidations(builder->reorder, txn->xid, lsn,
- ninvalidations, msgs);
+ ReorderBufferAddDistributedInvalidtions(builder->reorder,
+ txn->xid, lsn,
+ ninvalidations, msgs);
}
}
}
diff --git a/src/include/replication/reorderbuffer.h b/src/include/replication/reorderbuffer.h
index 545cee891ed..6e097cac495 100644
--- a/src/include/replication/reorderbuffer.h
+++ b/src/include/replication/reorderbuffer.h
@@ -69,6 +69,12 @@ enum ReorderBufferChangeType
/* forward declaration */
struct ReorderBufferTXN;
+/* Should the complete cache be invalidated? */
+#define rbtxn_inval_all_cache(txn) \
+( \
+ ((txn)->txn_flags & RBTXN_INVAL_ALL_CACHE) != 0 \
+)
+
/*
* a single 'change', can be an insert (with one tuple), an update (old, new),
* or a delete (old).
@@ -163,6 +169,7 @@ typedef struct ReorderBufferChange
#define RBTXN_HAS_CATALOG_CHANGES 0x0001
#define RBTXN_IS_SUBXACT 0x0002
#define RBTXN_IS_SERIALIZED 0x0004
+#define RBTXN_INVAL_ALL_CACHE 0x0008
/* Does the transaction have catalog changes? */
#define rbtxn_has_catalog_changes(txn) \
@@ -311,6 +318,13 @@ typedef struct ReorderBufferTXN
* Size of this transaction (changes currently in memory, in bytes).
*/
Size size;
+
+ /*
+ * Stores the cache invalidation messages distributed by the committing
+ * transaction.
+ */
+ uint32 ninvalidations_distr;
+ SharedInvalidationMessage *distributed_invalidations;
} ReorderBufferTXN;
/* so we can define the callbacks used inside struct ReorderBuffer itself */
@@ -451,6 +465,10 @@ void ReorderBufferAddNewTupleCids(ReorderBuffer *, TransactionId, XLogRecPtr ls
CommandId cmin, CommandId cmax, CommandId combocid);
void ReorderBufferAddInvalidations(ReorderBuffer *, TransactionId, XLogRecPtr lsn,
Size nmsgs, SharedInvalidationMessage *msgs);
+void ReorderBufferAddDistributedInvalidtions(ReorderBuffer *rb,
+ TransactionId xid,
+ XLogRecPtr lsn, Size nmsgs,
+ SharedInvalidationMessage *msgs);
void ReorderBufferImmediateInvalidation(ReorderBuffer *, uint32 ninvalidations,
SharedInvalidationMessage *invalidations);
void ReorderBufferProcessXid(ReorderBuffer *, TransactionId xid, XLogRecPtr lsn);
--
2.43.0
[application/octet-stream] v8-PG17-0001-Fix-exponential-memory-allocation-issue-in-l.patch (11.9K, ../../CALDaNm2sGfKsZwm8rG132N_i0AvyxkYu8TL46kNcx7KwAUABxA@mail.gmail.com/5-v8-PG17-0001-Fix-exponential-memory-allocation-issue-in-l.patch)
download | inline diff:
From 3e8a81cd27d8da8f3e0d02587e30d2131028399d Mon Sep 17 00:00:00 2001
From: Vignesh <vignesh21@gmail.com>
Date: Mon, 2 Jun 2025 10:33:53 +0530
Subject: [PATCH v8-PG17] Fix exponential memory allocation issue in logical
decoding due to invalidation message distribution
Commit cadaf0ac46 added logic to distribute invalidation messages from
one transaction to all concurrent in-progress transactions. However,
since the source of each invalidation message was not tracked, transactions
could redundantly re-distribute already propagated messages. This led to an
exponential increase in invalidation messages and, ultimately, memory
allocation failures.
To fix this, a new field is introduced to track distributed invalidation
messages separately. This ensures that only transactions in the pre-existing
list receive the invalidations, preventing redundant propagation.
The accumulated distributed invalidations are now consumed at the end of
transaction processing in `ReorderBufferProcessTXN()`.
---
.../replication/logical/reorderbuffer.c | 175 ++++++++++++++----
src/backend/replication/logical/snapbuild.c | 5 +-
src/include/replication/reorderbuffer.h | 17 ++
3 files changed, 158 insertions(+), 39 deletions(-)
diff --git a/src/backend/replication/logical/reorderbuffer.c b/src/backend/replication/logical/reorderbuffer.c
index 03eb005c39d..01a2cb87cb1 100644
--- a/src/backend/replication/logical/reorderbuffer.c
+++ b/src/backend/replication/logical/reorderbuffer.c
@@ -108,6 +108,7 @@
#include "storage/fd.h"
#include "storage/sinval.h"
#include "utils/builtins.h"
+#include "utils/inval.h"
#include "utils/memutils.h"
#include "utils/rel.h"
#include "utils/relfilenumbermap.h"
@@ -469,6 +470,12 @@ ReorderBufferReturnTXN(ReorderBuffer *rb, ReorderBufferTXN *txn)
txn->invalidations = NULL;
}
+ if (txn->distributed_invalidations)
+ {
+ pfree(txn->distributed_invalidations);
+ txn->distributed_invalidations = NULL;
+ }
+
/* Reset the toast hash */
ReorderBufferToastReset(rb, txn);
@@ -2574,7 +2581,14 @@ ReorderBufferProcessTXN(ReorderBuffer *rb, ReorderBufferTXN *txn,
AbortCurrentTransaction();
/* make sure there's no cache pollution */
- ReorderBufferExecuteInvalidations(txn->ninvalidations, txn->invalidations);
+ if (rbtxn_inval_all_cache(txn))
+ InvalidateSystemCaches();
+ else
+ {
+ ReorderBufferExecuteInvalidations(txn->ninvalidations, txn->invalidations);
+ ReorderBufferExecuteInvalidations(txn->ninvalidations_distr,
+ txn->distributed_invalidations);
+ }
if (using_subtxn)
RollbackAndReleaseCurrentSubTransaction();
@@ -2650,6 +2664,20 @@ ReorderBufferProcessTXN(ReorderBuffer *rb, ReorderBufferTXN *txn,
errdata = NULL;
curtxn->concurrent_abort = true;
+ /*
+ * Make sure there's no cache pollution. Unlike the PG_TRY
+ * section, this must be done unconditionally, since processing
+ * may fail before we reach the invalidation messages. This isn't
+ * needed in non-rollback cases, as the system cache is cleared
+ * either by the API's error handling or when the walsender exits.
+ */
+ if (rbtxn_inval_all_cache(txn))
+ InvalidateSystemCaches();
+ else
+ ReorderBufferExecuteInvalidations(txn->ninvalidations_distr,
+ txn->distributed_invalidations);
+
+
/* Reset the TXN so that it is allowed to stream remaining data. */
ReorderBufferResetTXN(rb, txn, snapshot_now,
command_id, prev_lsn,
@@ -3313,58 +3341,57 @@ ReorderBufferAddNewTupleCids(ReorderBuffer *rb, TransactionId xid,
}
/*
- * Accumulate the invalidations for executing them later.
+ * Helper function to store invalidation messages and queue them as a
+ * change in the reorder buffer. Handles both regular and distributed
+ * invalidations.
*
- * This needs to be called for each XLOG_XACT_INVALIDATIONS message and
- * accumulates all the invalidation messages in the toplevel transaction, if
- * available, otherwise in the current transaction, as well as in the form of
- * change in reorder buffer. We require to record it in form of the change
+ * This accumulates all the invalidation messages in the toplevel transaction,
+ * if available, otherwise in the current transaction, as well as in the form
+ * of change in reorder buffer. We require to record it in form of the change
* so that we can execute only the required invalidations instead of executing
* all the invalidations on each CommandId increment. We also need to
* accumulate these in the txn buffer because in some cases where we skip
* processing the transaction (see ReorderBufferForget), we need to execute
* all the invalidations together.
*/
-void
-ReorderBufferAddInvalidations(ReorderBuffer *rb, TransactionId xid,
- XLogRecPtr lsn, Size nmsgs,
- SharedInvalidationMessage *msgs)
+static void
+ReorderBufferAddInvalidationsCommon(ReorderBuffer *rb, TransactionId xid,
+ XLogRecPtr lsn, Size nmsgs,
+ SharedInvalidationMessage *msgs,
+ ReorderBufferTXN *txn,
+ bool for_inval)
{
- ReorderBufferTXN *txn;
MemoryContext oldcontext;
ReorderBufferChange *change;
- txn = ReorderBufferTXNByXid(rb, xid, true, NULL, lsn, true);
-
oldcontext = MemoryContextSwitchTo(rb->context);
- /*
- * Collect all the invalidations under the top transaction, if available,
- * so that we can execute them all together. See comments atop this
- * function.
- */
- txn = rbtxn_get_toptxn(txn);
+ if (!for_inval || (for_inval && !rbtxn_inval_all_cache(txn)))
+ {
+ uint32 *ninvalidations;
+ SharedInvalidationMessage **invalidations;
- Assert(nmsgs > 0);
+ ninvalidations = for_inval ? &txn->ninvalidations_distr : &txn->ninvalidations;
+ invalidations = for_inval ? &txn->distributed_invalidations : &txn->invalidations;
- /* Accumulate invalidations. */
- if (txn->ninvalidations == 0)
- {
- txn->ninvalidations = nmsgs;
- txn->invalidations = (SharedInvalidationMessage *)
- palloc(sizeof(SharedInvalidationMessage) * nmsgs);
- memcpy(txn->invalidations, msgs,
- sizeof(SharedInvalidationMessage) * nmsgs);
- }
- else
- {
- txn->invalidations = (SharedInvalidationMessage *)
- repalloc(txn->invalidations, sizeof(SharedInvalidationMessage) *
- (txn->ninvalidations + nmsgs));
+ /* Accumulate invalidations. */
+ if (*ninvalidations == 0)
+ {
+ *ninvalidations = nmsgs;
+ *invalidations = (SharedInvalidationMessage *)
+ palloc(sizeof(SharedInvalidationMessage) * nmsgs);
+ memcpy(*invalidations, msgs, sizeof(SharedInvalidationMessage) * nmsgs);
+ }
+ else
+ {
+ *invalidations = (SharedInvalidationMessage *)
+ repalloc(*invalidations, sizeof(SharedInvalidationMessage) *
+ (*ninvalidations + nmsgs));
- memcpy(txn->invalidations + txn->ninvalidations, msgs,
- nmsgs * sizeof(SharedInvalidationMessage));
- txn->ninvalidations += nmsgs;
+ memcpy(*invalidations + *ninvalidations, msgs,
+ nmsgs * sizeof(SharedInvalidationMessage));
+ *ninvalidations += nmsgs;
+ }
}
change = ReorderBufferGetChange(rb);
@@ -3380,6 +3407,80 @@ ReorderBufferAddInvalidations(ReorderBuffer *rb, TransactionId xid,
MemoryContextSwitchTo(oldcontext);
}
+/*
+ * Accumulate the invalidations for executing them later.
+ *
+ * This needs to be called for each XLOG_XACT_INVALIDATIONS message.
+ */
+void
+ReorderBufferAddInvalidations(ReorderBuffer *rb, TransactionId xid,
+ XLogRecPtr lsn, Size nmsgs,
+ SharedInvalidationMessage *msgs)
+{
+ ReorderBufferTXN *txn;
+
+ txn = ReorderBufferTXNByXid(rb, xid, true, NULL, lsn, true);
+
+ /*
+ * Collect all the invalidations under the top transaction, if available,
+ * so that we can execute them all together. See comments atop
+ * ReorderBufferAddInvalidationsCommon function.
+ */
+ txn = rbtxn_get_toptxn(txn);
+
+ Assert(nmsgs > 0);
+
+ ReorderBufferAddInvalidationsCommon(rb, xid, lsn, nmsgs, msgs, txn, false);
+}
+
+/*
+ * Accumulate the invalidations sent by committed transactions for executing
+ * them later.
+ *
+ * This needs to be called by committed transactions to distribute the
+ * invalidations to the in-progress transactions.
+ */
+void
+ReorderBufferAddDistributedInvalidtions(ReorderBuffer *rb, TransactionId xid,
+ XLogRecPtr lsn, Size nmsgs,
+ SharedInvalidationMessage *msgs)
+{
+ ReorderBufferTXN *txn;
+ Size req_mem_size;
+
+ txn = ReorderBufferTXNByXid(rb, xid, true, NULL, lsn, true);
+
+ /*
+ * Collect all the invalidations under the top transaction, if available,
+ * so that we can execute them all together. See comments
+ * ReorderBufferAddInvalidationsCommon ReorderBufferAddInvalidations.
+ */
+ txn = rbtxn_get_toptxn(txn);
+
+ Assert(nmsgs > 0);
+
+ req_mem_size = sizeof(SharedInvalidationMessage) * (txn->ninvalidations_distr + nmsgs);
+
+ /*
+ * If the number of invalidation messages is larger than 8MB, it's more
+ * efficient to invalidate the entire cache rather than processing each
+ * message individually.
+ */
+ if (req_mem_size > (8 * 1024 * 1024) || rbtxn_inval_all_cache(txn))
+ {
+ txn->txn_flags |= RBTXN_INVAL_ALL_CACHE;
+
+ if (txn->distributed_invalidations)
+ {
+ pfree(txn->distributed_invalidations);
+ txn->distributed_invalidations = NULL;
+ txn->ninvalidations_distr = 0;
+ }
+ }
+
+ ReorderBufferAddInvalidationsCommon(rb, xid, lsn, nmsgs, msgs, txn, true);
+}
+
/*
* Apply all invalidations we know. Possibly we only need parts at this point
* in the changestream but we don't know which those are.
diff --git a/src/backend/replication/logical/snapbuild.c b/src/backend/replication/logical/snapbuild.c
index 110e0b0a044..7ca5ce71ebf 100644
--- a/src/backend/replication/logical/snapbuild.c
+++ b/src/backend/replication/logical/snapbuild.c
@@ -945,8 +945,9 @@ SnapBuildDistributeSnapshotAndInval(SnapBuild *builder, XLogRecPtr lsn, Transact
{
Assert(msgs != NULL);
- ReorderBufferAddInvalidations(builder->reorder, txn->xid, lsn,
- ninvalidations, msgs);
+ ReorderBufferAddDistributedInvalidtions(builder->reorder,
+ txn->xid, lsn,
+ ninvalidations, msgs);
}
}
}
diff --git a/src/include/replication/reorderbuffer.h b/src/include/replication/reorderbuffer.h
index 4c56f219fd8..ab4a0c63151 100644
--- a/src/include/replication/reorderbuffer.h
+++ b/src/include/replication/reorderbuffer.h
@@ -168,6 +168,7 @@ typedef struct ReorderBufferChange
#define RBTXN_PREPARE 0x0040
#define RBTXN_SKIPPED_PREPARE 0x0080
#define RBTXN_HAS_STREAMABLE_CHANGE 0x0100
+#define RBTXN_INVAL_ALL_CACHE 0x0200
/* Does the transaction have catalog changes? */
#define rbtxn_has_catalog_changes(txn) \
@@ -205,6 +206,12 @@ typedef struct ReorderBufferChange
((txn)->txn_flags & RBTXN_HAS_STREAMABLE_CHANGE) != 0 \
)
+/* Should the complete cache be invalidated? */
+#define rbtxn_inval_all_cache(txn) \
+( \
+ ((txn)->txn_flags & RBTXN_INVAL_ALL_CACHE) != 0 \
+)
+
/*
* Has this transaction been streamed to downstream?
*
@@ -422,6 +429,13 @@ typedef struct ReorderBufferTXN
* Private data pointer of the output plugin.
*/
void *output_plugin_private;
+
+ /*
+ * Stores the cache invalidation messages distributed by the committing
+ * transaction.
+ */
+ uint32 ninvalidations_distr;
+ SharedInvalidationMessage *distributed_invalidations;
} ReorderBufferTXN;
/* so we can define the callbacks used inside struct ReorderBuffer itself */
@@ -709,6 +723,9 @@ extern void ReorderBufferAddNewTupleCids(ReorderBuffer *rb, TransactionId xid,
CommandId cmin, CommandId cmax, CommandId combocid);
extern void ReorderBufferAddInvalidations(ReorderBuffer *rb, TransactionId xid, XLogRecPtr lsn,
Size nmsgs, SharedInvalidationMessage *msgs);
+extern void ReorderBufferAddDistributedInvalidtions(ReorderBuffer *rb, TransactionId xid,
+ XLogRecPtr lsn, Size nmsgs,
+ SharedInvalidationMessage *msgs);
extern void ReorderBufferImmediateInvalidation(ReorderBuffer *rb, uint32 ninvalidations,
SharedInvalidationMessage *invalidations);
extern void ReorderBufferProcessXid(ReorderBuffer *rb, TransactionId xid, XLogRecPtr lsn);
--
2.43.0
[application/octet-stream] v8-master-0001-Fix-exponential-memory-allocation-issue-in.patch (12.0K, ../../CALDaNm2sGfKsZwm8rG132N_i0AvyxkYu8TL46kNcx7KwAUABxA@mail.gmail.com/6-v8-master-0001-Fix-exponential-memory-allocation-issue-in.patch)
download | inline diff:
From 81db514f0703c8752b8ce3e7cbe95cc7be5530ce Mon Sep 17 00:00:00 2001
From: Vignesh <vignesh21@gmail.com>
Date: Fri, 30 May 2025 12:06:07 +0530
Subject: [PATCH v8-master] Fix exponential memory allocation issue in logical
decoding due to invalidation message distribution
Commit 4909b38af0 added logic to distribute invalidation messages from
one transaction to all concurrent in-progress transactions. However,
since the source of each invalidation message was not tracked, transactions
could redundantly re-distribute already propagated messages. This led to an
exponential increase in invalidation messages and, ultimately, memory
allocation failures.
To fix this, a new field is introduced to track distributed invalidation
messages separately. This ensures that only transactions in the pre-existing
list receive the invalidations, preventing redundant propagation.
The accumulated distributed invalidations are now consumed at the end of
transaction processing in `ReorderBufferProcessTXN()`.
---
.../replication/logical/reorderbuffer.c | 175 ++++++++++++++----
src/backend/replication/logical/snapbuild.c | 5 +-
src/include/replication/reorderbuffer.h | 18 ++
3 files changed, 159 insertions(+), 39 deletions(-)
diff --git a/src/backend/replication/logical/reorderbuffer.c b/src/backend/replication/logical/reorderbuffer.c
index 67655111875..e2e02315639 100644
--- a/src/backend/replication/logical/reorderbuffer.c
+++ b/src/backend/replication/logical/reorderbuffer.c
@@ -109,6 +109,7 @@
#include "storage/procarray.h"
#include "storage/sinval.h"
#include "utils/builtins.h"
+#include "utils/inval.h"
#include "utils/memutils.h"
#include "utils/rel.h"
#include "utils/relfilenumbermap.h"
@@ -472,6 +473,12 @@ ReorderBufferFreeTXN(ReorderBuffer *rb, ReorderBufferTXN *txn)
txn->invalidations = NULL;
}
+ if (txn->distributed_invalidations)
+ {
+ pfree(txn->distributed_invalidations);
+ txn->distributed_invalidations = NULL;
+ }
+
/* Reset the toast hash */
ReorderBufferToastReset(rb, txn);
@@ -2661,7 +2668,14 @@ ReorderBufferProcessTXN(ReorderBuffer *rb, ReorderBufferTXN *txn,
AbortCurrentTransaction();
/* make sure there's no cache pollution */
- ReorderBufferExecuteInvalidations(txn->ninvalidations, txn->invalidations);
+ if (rbtxn_inval_all_cache(txn))
+ InvalidateSystemCaches();
+ else
+ {
+ ReorderBufferExecuteInvalidations(txn->ninvalidations, txn->invalidations);
+ ReorderBufferExecuteInvalidations(txn->ninvalidations_distr,
+ txn->distributed_invalidations);
+ }
if (using_subtxn)
RollbackAndReleaseCurrentSubTransaction();
@@ -2747,6 +2761,20 @@ ReorderBufferProcessTXN(ReorderBuffer *rb, ReorderBufferTXN *txn,
if (stream_started)
ReorderBufferMaybeMarkTXNStreamed(rb, txn);
+ /*
+ * Make sure there's no cache pollution. Unlike the PG_TRY section,
+ * this must be done unconditionally, since processing may fail
+ * before we reach the invalidation messages. This isn't needed
+ * in non-rollback cases, as the system cache is cleared either
+ * by the API's error handling or when the walsender exits.
+ */
+ if (rbtxn_inval_all_cache(txn))
+ InvalidateSystemCaches();
+ else
+ ReorderBufferExecuteInvalidations(txn->ninvalidations_distr,
+ txn->distributed_invalidations);
+
+
/* Reset the TXN so that it is allowed to stream remaining data. */
ReorderBufferResetTXN(rb, txn, snapshot_now,
command_id, prev_lsn,
@@ -3422,58 +3450,57 @@ ReorderBufferAddNewTupleCids(ReorderBuffer *rb, TransactionId xid,
}
/*
- * Accumulate the invalidations for executing them later.
+ * Helper function to store invalidation messages and queue them as a
+ * change in the reorder buffer. Handles both regular and distributed
+ * invalidations.
*
- * This needs to be called for each XLOG_XACT_INVALIDATIONS message and
- * accumulates all the invalidation messages in the toplevel transaction, if
- * available, otherwise in the current transaction, as well as in the form of
- * change in reorder buffer. We require to record it in form of the change
+ * This accumulates all the invalidation messages in the toplevel transaction,
+ * if available, otherwise in the current transaction, as well as in the form
+ * of change in reorder buffer. We require to record it in form of the change
* so that we can execute only the required invalidations instead of executing
* all the invalidations on each CommandId increment. We also need to
* accumulate these in the txn buffer because in some cases where we skip
* processing the transaction (see ReorderBufferForget), we need to execute
* all the invalidations together.
*/
-void
-ReorderBufferAddInvalidations(ReorderBuffer *rb, TransactionId xid,
- XLogRecPtr lsn, Size nmsgs,
- SharedInvalidationMessage *msgs)
+static void
+ReorderBufferAddInvalidationsCommon(ReorderBuffer *rb, TransactionId xid,
+ XLogRecPtr lsn, Size nmsgs,
+ SharedInvalidationMessage *msgs,
+ ReorderBufferTXN *txn,
+ bool for_inval)
{
- ReorderBufferTXN *txn;
MemoryContext oldcontext;
ReorderBufferChange *change;
- txn = ReorderBufferTXNByXid(rb, xid, true, NULL, lsn, true);
-
oldcontext = MemoryContextSwitchTo(rb->context);
- /*
- * Collect all the invalidations under the top transaction, if available,
- * so that we can execute them all together. See comments atop this
- * function.
- */
- txn = rbtxn_get_toptxn(txn);
+ if (!for_inval || (for_inval && !rbtxn_inval_all_cache(txn)))
+ {
+ uint32 *ninvalidations;
+ SharedInvalidationMessage **invalidations;
- Assert(nmsgs > 0);
+ ninvalidations = for_inval ? &txn->ninvalidations_distr : &txn->ninvalidations;
+ invalidations = for_inval ? &txn->distributed_invalidations : &txn->invalidations;
- /* Accumulate invalidations. */
- if (txn->ninvalidations == 0)
- {
- txn->ninvalidations = nmsgs;
- txn->invalidations = (SharedInvalidationMessage *)
- palloc(sizeof(SharedInvalidationMessage) * nmsgs);
- memcpy(txn->invalidations, msgs,
- sizeof(SharedInvalidationMessage) * nmsgs);
- }
- else
- {
- txn->invalidations = (SharedInvalidationMessage *)
- repalloc(txn->invalidations, sizeof(SharedInvalidationMessage) *
- (txn->ninvalidations + nmsgs));
+ /* Accumulate invalidations. */
+ if (*ninvalidations == 0)
+ {
+ *ninvalidations = nmsgs;
+ *invalidations = (SharedInvalidationMessage *)
+ palloc(sizeof(SharedInvalidationMessage) * nmsgs);
+ memcpy(*invalidations, msgs, sizeof(SharedInvalidationMessage) * nmsgs);
+ }
+ else
+ {
+ *invalidations = (SharedInvalidationMessage *)
+ repalloc(*invalidations, sizeof(SharedInvalidationMessage) *
+ (*ninvalidations + nmsgs));
- memcpy(txn->invalidations + txn->ninvalidations, msgs,
- nmsgs * sizeof(SharedInvalidationMessage));
- txn->ninvalidations += nmsgs;
+ memcpy(*invalidations + *ninvalidations, msgs,
+ nmsgs * sizeof(SharedInvalidationMessage));
+ *ninvalidations += nmsgs;
+ }
}
change = ReorderBufferAllocChange(rb);
@@ -3489,6 +3516,80 @@ ReorderBufferAddInvalidations(ReorderBuffer *rb, TransactionId xid,
MemoryContextSwitchTo(oldcontext);
}
+/*
+ * Accumulate the invalidations for executing them later.
+ *
+ * This needs to be called for each XLOG_XACT_INVALIDATIONS message.
+ */
+void
+ReorderBufferAddInvalidations(ReorderBuffer *rb, TransactionId xid,
+ XLogRecPtr lsn, Size nmsgs,
+ SharedInvalidationMessage *msgs)
+{
+ ReorderBufferTXN *txn;
+
+ txn = ReorderBufferTXNByXid(rb, xid, true, NULL, lsn, true);
+
+ /*
+ * Collect all the invalidations under the top transaction, if available,
+ * so that we can execute them all together. See comments atop
+ * ReorderBufferAddInvalidationsCommon function.
+ */
+ txn = rbtxn_get_toptxn(txn);
+
+ Assert(nmsgs > 0);
+
+ ReorderBufferAddInvalidationsCommon(rb, xid, lsn, nmsgs, msgs, txn, false);
+}
+
+/*
+ * Accumulate the invalidations sent by committed transactions for executing
+ * them later.
+ *
+ * This needs to be called by committed transactions to distribute the
+ * invalidations to the in-progress transactions.
+ */
+void
+ReorderBufferAddDistributedInvalidtions(ReorderBuffer *rb, TransactionId xid,
+ XLogRecPtr lsn, Size nmsgs,
+ SharedInvalidationMessage *msgs)
+{
+ ReorderBufferTXN *txn;
+ Size req_mem_size;
+
+ txn = ReorderBufferTXNByXid(rb, xid, true, NULL, lsn, true);
+
+ /*
+ * Collect all the invalidations under the top transaction, if available,
+ * so that we can execute them all together. See comments
+ * ReorderBufferAddInvalidationsCommon ReorderBufferAddInvalidations.
+ */
+ txn = rbtxn_get_toptxn(txn);
+
+ Assert(nmsgs > 0);
+
+ req_mem_size = sizeof(SharedInvalidationMessage) * (txn->ninvalidations_distr + nmsgs);
+
+ /*
+ * If the number of invalidation messages is larger than 8MB, it's more
+ * efficient to invalidate the entire cache rather than processing each
+ * message individually.
+ */
+ if (req_mem_size > (8 * 1024 * 1024) || rbtxn_inval_all_cache(txn))
+ {
+ txn->txn_flags |= RBTXN_INVAL_ALL_CACHE;
+
+ if (txn->distributed_invalidations)
+ {
+ pfree(txn->distributed_invalidations);
+ txn->distributed_invalidations = NULL;
+ txn->ninvalidations_distr = 0;
+ }
+ }
+
+ ReorderBufferAddInvalidationsCommon(rb, xid, lsn, nmsgs, msgs, txn, true);
+}
+
/*
* Apply all invalidations we know. Possibly we only need parts at this point
* in the changestream but we don't know which those are.
diff --git a/src/backend/replication/logical/snapbuild.c b/src/backend/replication/logical/snapbuild.c
index 0d7bddbe4ed..a2d93f35a2f 100644
--- a/src/backend/replication/logical/snapbuild.c
+++ b/src/backend/replication/logical/snapbuild.c
@@ -807,8 +807,9 @@ SnapBuildDistributeSnapshotAndInval(SnapBuild *builder, XLogRecPtr lsn, Transact
{
Assert(msgs != NULL);
- ReorderBufferAddInvalidations(builder->reorder, txn->xid, lsn,
- ninvalidations, msgs);
+ ReorderBufferAddDistributedInvalidtions(builder->reorder,
+ txn->xid, lsn,
+ ninvalidations, msgs);
}
}
}
diff --git a/src/include/replication/reorderbuffer.h b/src/include/replication/reorderbuffer.h
index 24e88c409ba..a56c3e25d5a 100644
--- a/src/include/replication/reorderbuffer.h
+++ b/src/include/replication/reorderbuffer.h
@@ -176,6 +176,7 @@ typedef struct ReorderBufferChange
#define RBTXN_SENT_PREPARE 0x0200
#define RBTXN_IS_COMMITTED 0x0400
#define RBTXN_IS_ABORTED 0x0800
+#define RBTXN_INVAL_ALL_CACHE 0x1000
#define RBTXN_PREPARE_STATUS_MASK (RBTXN_IS_PREPARED | RBTXN_SKIPPED_PREPARE | RBTXN_SENT_PREPARE)
@@ -215,6 +216,12 @@ typedef struct ReorderBufferChange
((txn)->txn_flags & RBTXN_HAS_STREAMABLE_CHANGE) != 0 \
)
+/* Should the complete cache be invalidated? */
+#define rbtxn_inval_all_cache(txn) \
+( \
+ ((txn)->txn_flags & RBTXN_INVAL_ALL_CACHE) != 0 \
+)
+
/*
* Has this transaction been streamed to downstream?
*
@@ -422,6 +429,14 @@ typedef struct ReorderBufferTXN
uint32 ninvalidations;
SharedInvalidationMessage *invalidations;
+ /*
+ * Stores the cache invalidation messages distributed by the committing
+ * transaction.
+ */
+ uint32 ninvalidations_distr;
+ SharedInvalidationMessage *distributed_invalidations;
+
+
/* ---
* Position in one of two lists:
* * list of subtransactions if we are *known* to be subxact
@@ -738,6 +753,9 @@ extern void ReorderBufferAddNewTupleCids(ReorderBuffer *rb, TransactionId xid,
CommandId cmin, CommandId cmax, CommandId combocid);
extern void ReorderBufferAddInvalidations(ReorderBuffer *rb, TransactionId xid, XLogRecPtr lsn,
Size nmsgs, SharedInvalidationMessage *msgs);
+extern void ReorderBufferAddDistributedInvalidtions(ReorderBuffer *rb, TransactionId xid,
+ XLogRecPtr lsn, Size nmsgs,
+ SharedInvalidationMessage *msgs);
extern void ReorderBufferImmediateInvalidation(ReorderBuffer *rb, uint32 ninvalidations,
SharedInvalidationMessage *invalidations);
extern void ReorderBufferProcessXid(ReorderBuffer *rb, TransactionId xid, XLogRecPtr lsn);
--
2.43.0
[application/octet-stream] v8-PG16-0001-Fix-exponential-memory-allocation-issue-in-l.patch (11.9K, ../../CALDaNm2sGfKsZwm8rG132N_i0AvyxkYu8TL46kNcx7KwAUABxA@mail.gmail.com/7-v8-PG16-0001-Fix-exponential-memory-allocation-issue-in-l.patch)
download | inline diff:
From 0e39b2cc4cc2c716847a536abc944be4823f3e11 Mon Sep 17 00:00:00 2001
From: Vignesh <vignesh21@gmail.com>
Date: Mon, 2 Jun 2025 11:37:24 +0530
Subject: [PATCH v8-PG16] Fix exponential memory allocation issue in logical
decoding due to invalidation message distribution
Commit 9a2f8b4f01 added logic to distribute invalidation messages from
one transaction to all concurrent in-progress transactions. However,
since the source of each invalidation message was not tracked, transactions
could redundantly re-distribute already propagated messages. This led to an
exponential increase in invalidation messages and, ultimately, memory
allocation failures.
To fix this, a new field is introduced to track distributed invalidation
messages separately. This ensures that only transactions in the pre-existing
list receive the invalidations, preventing redundant propagation.
The accumulated distributed invalidations are now consumed at the end of
transaction processing in `ReorderBufferProcessTXN()`.
---
.../replication/logical/reorderbuffer.c | 175 ++++++++++++++----
src/backend/replication/logical/snapbuild.c | 5 +-
src/include/replication/reorderbuffer.h | 17 ++
3 files changed, 158 insertions(+), 39 deletions(-)
diff --git a/src/backend/replication/logical/reorderbuffer.c b/src/backend/replication/logical/reorderbuffer.c
index fa04e829cc9..43b5583b1f8 100644
--- a/src/backend/replication/logical/reorderbuffer.c
+++ b/src/backend/replication/logical/reorderbuffer.c
@@ -103,6 +103,7 @@
#include "storage/sinval.h"
#include "utils/builtins.h"
#include "utils/combocid.h"
+#include "utils/inval.h"
#include "utils/memdebug.h"
#include "utils/memutils.h"
#include "utils/rel.h"
@@ -461,6 +462,12 @@ ReorderBufferReturnTXN(ReorderBuffer *rb, ReorderBufferTXN *txn)
txn->invalidations = NULL;
}
+ if (txn->distributed_invalidations)
+ {
+ pfree(txn->distributed_invalidations);
+ txn->distributed_invalidations = NULL;
+ }
+
/* Reset the toast hash */
ReorderBufferToastReset(rb, txn);
@@ -2539,7 +2546,14 @@ ReorderBufferProcessTXN(ReorderBuffer *rb, ReorderBufferTXN *txn,
AbortCurrentTransaction();
/* make sure there's no cache pollution */
- ReorderBufferExecuteInvalidations(txn->ninvalidations, txn->invalidations);
+ if (rbtxn_inval_all_cache(txn))
+ InvalidateSystemCaches();
+ else
+ {
+ ReorderBufferExecuteInvalidations(txn->ninvalidations, txn->invalidations);
+ ReorderBufferExecuteInvalidations(txn->ninvalidations_distr,
+ txn->distributed_invalidations);
+ }
if (using_subtxn)
RollbackAndReleaseCurrentSubTransaction();
@@ -2615,6 +2629,20 @@ ReorderBufferProcessTXN(ReorderBuffer *rb, ReorderBufferTXN *txn,
errdata = NULL;
curtxn->concurrent_abort = true;
+ /*
+ * Make sure there's no cache pollution. Unlike the PG_TRY section,
+ * this must be done unconditionally, since processing may fail
+ * before we reach the invalidation messages. This isn't needed
+ * in non-rollback cases, as the system cache is cleared either
+ * by the API's error handling or when the walsender exits.
+ */
+ if (rbtxn_inval_all_cache(txn))
+ InvalidateSystemCaches();
+ else
+ ReorderBufferExecuteInvalidations(txn->ninvalidations_distr,
+ txn->distributed_invalidations);
+
+
/* Reset the TXN so that it is allowed to stream remaining data. */
ReorderBufferResetTXN(rb, txn, snapshot_now,
command_id, prev_lsn,
@@ -3258,58 +3286,57 @@ ReorderBufferAddNewTupleCids(ReorderBuffer *rb, TransactionId xid,
}
/*
- * Accumulate the invalidations for executing them later.
+ * Helper function to store invalidation messages and queue them as a
+ * change in the reorder buffer. Handles both regular and distributed
+ * invalidations.
*
- * This needs to be called for each XLOG_XACT_INVALIDATIONS message and
- * accumulates all the invalidation messages in the toplevel transaction, if
- * available, otherwise in the current transaction, as well as in the form of
- * change in reorder buffer. We require to record it in form of the change
+ * This accumulates all the invalidation messages in the toplevel transaction,
+ * if available, otherwise in the current transaction, as well as in the form
+ * of change in reorder buffer. We require to record it in form of the change
* so that we can execute only the required invalidations instead of executing
* all the invalidations on each CommandId increment. We also need to
* accumulate these in the txn buffer because in some cases where we skip
* processing the transaction (see ReorderBufferForget), we need to execute
* all the invalidations together.
*/
-void
-ReorderBufferAddInvalidations(ReorderBuffer *rb, TransactionId xid,
- XLogRecPtr lsn, Size nmsgs,
- SharedInvalidationMessage *msgs)
+static void
+ReorderBufferAddInvalidationsCommon(ReorderBuffer *rb, TransactionId xid,
+ XLogRecPtr lsn, Size nmsgs,
+ SharedInvalidationMessage *msgs,
+ ReorderBufferTXN *txn,
+ bool for_inval)
{
- ReorderBufferTXN *txn;
MemoryContext oldcontext;
ReorderBufferChange *change;
- txn = ReorderBufferTXNByXid(rb, xid, true, NULL, lsn, true);
-
oldcontext = MemoryContextSwitchTo(rb->context);
- /*
- * Collect all the invalidations under the top transaction, if available,
- * so that we can execute them all together. See comments atop this
- * function.
- */
- txn = rbtxn_get_toptxn(txn);
+ if (!for_inval || (for_inval && !rbtxn_inval_all_cache(txn)))
+ {
+ uint32 *ninvalidations;
+ SharedInvalidationMessage **invalidations;
- Assert(nmsgs > 0);
+ ninvalidations = for_inval ? &txn->ninvalidations_distr : &txn->ninvalidations;
+ invalidations = for_inval ? &txn->distributed_invalidations : &txn->invalidations;
- /* Accumulate invalidations. */
- if (txn->ninvalidations == 0)
- {
- txn->ninvalidations = nmsgs;
- txn->invalidations = (SharedInvalidationMessage *)
- palloc(sizeof(SharedInvalidationMessage) * nmsgs);
- memcpy(txn->invalidations, msgs,
- sizeof(SharedInvalidationMessage) * nmsgs);
- }
- else
- {
- txn->invalidations = (SharedInvalidationMessage *)
- repalloc(txn->invalidations, sizeof(SharedInvalidationMessage) *
- (txn->ninvalidations + nmsgs));
+ /* Accumulate invalidations. */
+ if (*ninvalidations == 0)
+ {
+ *ninvalidations = nmsgs;
+ *invalidations = (SharedInvalidationMessage *)
+ palloc(sizeof(SharedInvalidationMessage) * nmsgs);
+ memcpy(*invalidations, msgs, sizeof(SharedInvalidationMessage) * nmsgs);
+ }
+ else
+ {
+ *invalidations = (SharedInvalidationMessage *)
+ repalloc(*invalidations, sizeof(SharedInvalidationMessage) *
+ (*ninvalidations + nmsgs));
- memcpy(txn->invalidations + txn->ninvalidations, msgs,
- nmsgs * sizeof(SharedInvalidationMessage));
- txn->ninvalidations += nmsgs;
+ memcpy(*invalidations + *ninvalidations, msgs,
+ nmsgs * sizeof(SharedInvalidationMessage));
+ *ninvalidations += nmsgs;
+ }
}
change = ReorderBufferGetChange(rb);
@@ -3325,6 +3352,80 @@ ReorderBufferAddInvalidations(ReorderBuffer *rb, TransactionId xid,
MemoryContextSwitchTo(oldcontext);
}
+/*
+ * Accumulate the invalidations for executing them later.
+ *
+ * This needs to be called for each XLOG_XACT_INVALIDATIONS message.
+ */
+void
+ReorderBufferAddInvalidations(ReorderBuffer *rb, TransactionId xid,
+ XLogRecPtr lsn, Size nmsgs,
+ SharedInvalidationMessage *msgs)
+{
+ ReorderBufferTXN *txn;
+
+ txn = ReorderBufferTXNByXid(rb, xid, true, NULL, lsn, true);
+
+ /*
+ * Collect all the invalidations under the top transaction, if available,
+ * so that we can execute them all together. See comments atop
+ * ReorderBufferAddInvalidationsCommon function.
+ */
+ txn = rbtxn_get_toptxn(txn);
+
+ Assert(nmsgs > 0);
+
+ ReorderBufferAddInvalidationsCommon(rb, xid, lsn, nmsgs, msgs, txn, false);
+}
+
+/*
+ * Accumulate the invalidations sent by committed transactions for executing
+ * them later.
+ *
+ * This needs to be called by committed transactions to distribute the
+ * invalidations to the in-progress transactions.
+ */
+void
+ReorderBufferAddDistributedInvalidtions(ReorderBuffer *rb, TransactionId xid,
+ XLogRecPtr lsn, Size nmsgs,
+ SharedInvalidationMessage *msgs)
+{
+ ReorderBufferTXN *txn;
+ Size req_mem_size;
+
+ txn = ReorderBufferTXNByXid(rb, xid, true, NULL, lsn, true);
+
+ /*
+ * Collect all the invalidations under the top transaction, if available,
+ * so that we can execute them all together. See comments
+ * ReorderBufferAddInvalidationsCommon ReorderBufferAddInvalidations.
+ */
+ txn = rbtxn_get_toptxn(txn);
+
+ Assert(nmsgs > 0);
+
+ req_mem_size = sizeof(SharedInvalidationMessage) * (txn->ninvalidations_distr + nmsgs);
+
+ /*
+ * If the number of invalidation messages is larger than 8MB, it's more
+ * efficient to invalidate the entire cache rather than processing each
+ * message individually.
+ */
+ if (req_mem_size > (8 * 1024 * 1024) || rbtxn_inval_all_cache(txn))
+ {
+ txn->txn_flags |= RBTXN_INVAL_ALL_CACHE;
+
+ if (txn->distributed_invalidations)
+ {
+ pfree(txn->distributed_invalidations);
+ txn->distributed_invalidations = NULL;
+ txn->ninvalidations_distr = 0;
+ }
+ }
+
+ ReorderBufferAddInvalidationsCommon(rb, xid, lsn, nmsgs, msgs, txn, true);
+}
+
/*
* Apply all invalidations we know. Possibly we only need parts at this point
* in the changestream but we don't know which those are.
diff --git a/src/backend/replication/logical/snapbuild.c b/src/backend/replication/logical/snapbuild.c
index d48ebb8337b..5f218ffaa84 100644
--- a/src/backend/replication/logical/snapbuild.c
+++ b/src/backend/replication/logical/snapbuild.c
@@ -931,8 +931,9 @@ SnapBuildDistributeSnapshotAndInval(SnapBuild *builder, XLogRecPtr lsn, Transact
{
Assert(msgs != NULL);
- ReorderBufferAddInvalidations(builder->reorder, txn->xid, lsn,
- ninvalidations, msgs);
+ ReorderBufferAddDistributedInvalidtions(builder->reorder,
+ txn->xid, lsn,
+ ninvalidations, msgs);
}
}
}
diff --git a/src/include/replication/reorderbuffer.h b/src/include/replication/reorderbuffer.h
index 8a32bbe28df..4049ad4bf4e 100644
--- a/src/include/replication/reorderbuffer.h
+++ b/src/include/replication/reorderbuffer.h
@@ -186,6 +186,7 @@ typedef struct ReorderBufferChange
#define RBTXN_PREPARE 0x0040
#define RBTXN_SKIPPED_PREPARE 0x0080
#define RBTXN_HAS_STREAMABLE_CHANGE 0x0100
+#define RBTXN_INVAL_ALL_CACHE 0x0200
/* Does the transaction have catalog changes? */
#define rbtxn_has_catalog_changes(txn) \
@@ -223,6 +224,12 @@ typedef struct ReorderBufferChange
((txn)->txn_flags & RBTXN_HAS_STREAMABLE_CHANGE) != 0 \
)
+/* Should the complete cache be invalidated? */
+#define rbtxn_inval_all_cache(txn) \
+( \
+ ((txn)->txn_flags & RBTXN_INVAL_ALL_CACHE) != 0 \
+)
+
/*
* Has this transaction been streamed to downstream?
*
@@ -435,6 +442,13 @@ typedef struct ReorderBufferTXN
* Private data pointer of the output plugin.
*/
void *output_plugin_private;
+
+ /*
+ * Stores the cache invalidation messages distributed by the committing
+ * transaction.
+ */
+ uint32 ninvalidations_distr;
+ SharedInvalidationMessage *distributed_invalidations;
} ReorderBufferTXN;
/* so we can define the callbacks used inside struct ReorderBuffer itself */
@@ -728,6 +742,9 @@ extern void ReorderBufferAddNewTupleCids(ReorderBuffer *rb, TransactionId xid,
CommandId cmin, CommandId cmax, CommandId combocid);
extern void ReorderBufferAddInvalidations(ReorderBuffer *rb, TransactionId xid, XLogRecPtr lsn,
Size nmsgs, SharedInvalidationMessage *msgs);
+extern void ReorderBufferAddDistributedInvalidtions(ReorderBuffer *rb, TransactionId xid,
+ XLogRecPtr lsn, Size nmsgs,
+ SharedInvalidationMessage *msgs);
extern void ReorderBufferImmediateInvalidation(ReorderBuffer *rb, uint32 ninvalidations,
SharedInvalidationMessage *invalidations);
extern void ReorderBufferProcessXid(ReorderBuffer *rb, TransactionId xid, XLogRecPtr lsn);
--
2.43.0
^ permalink raw reply [nested|flat] 94+ messages in thread
* RE: Logical replication 'invalid memory alloc request size 1585837200' after upgrading to 17.5
@ 2025-06-02 11:35 Hayato Kuroda (Fujitsu) <kuroda.hayato@fujitsu.com>
parent: vignesh C <vignesh21@gmail.com>
1 sibling, 1 reply; 94+ messages in thread
From: Hayato Kuroda (Fujitsu) @ 2025-06-02 11:35 UTC (permalink / raw)
To: pgsql-bugs@lists.postgresql.org <pgsql-bugs@lists.postgresql.org>; +Cc: Amit Kapila <amit.kapila16@gmail.com>; Duncan Sands <duncan.sands@deepbluecap.com>; vignesh C <vignesh21@gmail.com>; Masahiko Sawada <sawada.mshk@gmail.com>
Dear Amit, Duncan, Vignesh, Sawada-san,
I searched on GitHub and confirmed that no extensions use sizeof(ReorderBufferTXN).
Based on that, can we proceed with the fix for backbenches?
Note that I checked only repositories that are opened on GitHub. Proprietary
ones and hosting on other services cannot be guaranteed.
How I search
===========
I searched on github with "sizeof(ReorderBufferTXN)", and 259 files were found.
All of them were either "reorderbuffer.c" or "reorderbuffer.cpp". reorderbuffer.c
came from the forked postgres repos, and "reorderbuffer.cpp" came from the
openGauss project, which tries to enhance postgres. In both cases, they are
server-side code and will rebase community's commits when we update the header
file and sizeof(ReorderBufferTXN).
In this check, I could not find extensions that refer to the size; only
server-side codes were found. Based on that, we could extend the struct
ReorderBufferTXN.
[1]: https://github.com/search?q=sizeof%28ReorderBufferTXN%29&type=code
[2]: https://github.com/opengauss-mirror/openGauss-server/blob/master/src/gausskernel/storage/replication...
Best regards,
Hayato Kuroda
FUJITSU LIMITED
^ permalink raw reply [nested|flat] 94+ messages in thread
* Re: Logical replication 'invalid memory alloc request size 1585837200' after upgrading to 17.5
@ 2025-06-02 16:51 Masahiko Sawada <sawada.mshk@gmail.com>
parent: vignesh C <vignesh21@gmail.com>
0 siblings, 1 reply; 94+ messages in thread
From: Masahiko Sawada @ 2025-06-02 16:51 UTC (permalink / raw)
To: vignesh C <vignesh21@gmail.com>; +Cc: Amit Kapila <amit.kapila16@gmail.com>; Hayato Kuroda (Fujitsu) <kuroda.hayato@fujitsu.com>; Duncan Sands <duncan.sands@deepbluecap.com>; pgsql-bugs@lists.postgresql.org <pgsql-bugs@lists.postgresql.org>
On Sun, Jun 1, 2025 at 8:13 PM vignesh C <vignesh21@gmail.com> wrote:
>
> On Sat, 31 May 2025 at 10:20, Amit Kapila <amit.kapila16@gmail.com> wrote:
> >
> > On Fri, May 30, 2025 at 11:00 PM Masahiko Sawada <sawada.mshk@gmail.com> wrote:
> > >
> > > Thank you for updating the patch. Here are some review comments:
> > >
> > > ---
> > > + /*
> > > + * Make sure there's no cache pollution. Unlike the PG_TRY part,
> > > + * this must be done unconditionally because the processing might
> > > + * fail before we reach invalidation messages.
> > > + */
> > > + if (rbtxn_inval_all_cache(txn))
> > > + InvalidateSystemCaches();
> > > + else
> > > + ReorderBufferExecuteInvalidations(txn->ninvalidations_distr,
> > > +
> > > txn->distributed_invalidations);
> > > +
> > >
> > > If we don't need to execute the distributed inval message in an error
> > > path other than detecting concurrent abort, we should describe the
> > > reason.
> > >
> >
> > The concurrent abort handling is done for streaming and prepared
> > transactions, where we send the transaction changes to the client
> > before we read COMMIT/COMMIT PREPARED/ROLLBACK/ROLLBACK PREPARED. Now,
> > among these COMMIT/ROLLBACK PREPARED cases are handled as part of a
> > prepared transaction case. For ROLLBACK, we will never perform any
> > changes from the current transaction, so we don't need distributed
> > invalidations to be executed. For COMMIT, if we encounter any errors
> > while processing changes (this is when we reach the ERROR path, which
> > is not a concurrent abort), then we will reprocess all changes and, at
> > the end, execute both the current transaction and distributed
> > invalidations. Now, one possibility is that if, after ERROR, the
> > caller does slot_advance to skip the ERROR, then we will probably miss
> > executing the distributed invalidations, leading to data loss
> > afterwards. If the above theory is correct, then it is better to
> > execute distributed invalidation even in non-concurrent-abort cases in
> > the ERROR path.
This theory seems to be correct to me. If users re-try the logical
decoding with smaller logical_decoding_work_mem the process could end
up streaming the transaction with the wrong relcache before sending
the committed transaction?
> One possible reason this scenario may not occur is that
> pg_logical_slot_get_changes_guts uses a PG_CATCH block to handle
> exceptions, during which it calls InvalidateSystemCaches to clear the
> system cache. Because of this, I believe the scenario might not
> actually happen.
> @Sawada-san / others — Are there any other cases where this could still occur?
I think that we cannot confine use cases to walsender and
pg_logical_slot_get_changes_guts() given that we expose logical
decoding API at C level.
Regards,
--
Masahiko Sawada
Amazon Web Services: https://aws.amazon.com
^ permalink raw reply [nested|flat] 94+ messages in thread
* Re: Logical replication 'invalid memory alloc request size 1585837200' after upgrading to 17.5
@ 2025-06-02 17:19 Masahiko Sawada <sawada.mshk@gmail.com>
parent: vignesh C <vignesh21@gmail.com>
1 sibling, 1 reply; 94+ messages in thread
From: Masahiko Sawada @ 2025-06-02 17:19 UTC (permalink / raw)
To: vignesh C <vignesh21@gmail.com>; +Cc: Amit Kapila <amit.kapila16@gmail.com>; Hayato Kuroda (Fujitsu) <kuroda.hayato@fujitsu.com>; Duncan Sands <duncan.sands@deepbluecap.com>; pgsql-bugs@lists.postgresql.org <pgsql-bugs@lists.postgresql.org>
On Mon, Jun 2, 2025 at 3:22 AM vignesh C <vignesh21@gmail.com> wrote:
>
> On Sat, 31 May 2025 at 13:27, vignesh C <vignesh21@gmail.com> wrote:
> >
> > On Fri, 30 May 2025 at 23:00, Masahiko Sawada <sawada.mshk@gmail.com> wrote:
> > >
> > > Thank you for updating the patch. Here are some review comments:
> > >
> > > @@ -3439,9 +3464,27 @@ ReorderBufferAddInvalidations(ReorderBuffer
> > > *rb, TransactionId xid,
> > > XLogRecPtr lsn, Size nmsgs,
> > > SharedInvalidationMessage *msgs)
> > > {
> > > - ReorderBufferTXN *txn;
> > > + ReorderBufferAddInvalidationsExtended(rb, xid, lsn, nmsgs, msgs, false);
> > > +}
> > >
> > > If the patch is the changes for master do we need to have an extended
> > > version of ReorderBufferAddInvalidation()?
> >
> > This has been removed now and ReorderBufferAddDistributedInvalidtions
> > has been added
> >
> > > ---
> > > + /*
> > > + * Make sure there's no cache pollution. Unlike the PG_TRY part,
> > > + * this must be done unconditionally because the processing might
> > > + * fail before we reach invalidation messages.
> > > + */
> > > + if (rbtxn_inval_all_cache(txn))
> > > + InvalidateSystemCaches();
> > > + else
> > > + ReorderBufferExecuteInvalidations(txn->ninvalidations_distr,
> > > +
> > > txn->distributed_invalidations);
> > > +
> > >
> > > If we don't need to execute the distributed inval message in an error
> > > path other than detecting concurrent abort, we should describe the
> > > reason.
> >
> > Removed it to keep it in the common error path
> >
> > > ---
> > > Given that we don't account the memory usage of both
> > > txn->invalidations and txn->distributed_invalidations, probably we can
> > > have a lower limit, say 8MB (or lower?), to avoid memory exhaustion.
> >
> > Modified
> >
> > > ---
> > > + if ((for_inval && !AllocSizeIsValid(req_mem_size)) ||
> > > + rbtxn_inval_all_cache(txn))
> > > {
> > > - txn->ninvalidations = nmsgs;
> > > - txn->invalidations = (SharedInvalidationMessage *)
> > > - palloc(sizeof(SharedInvalidationMessage) * nmsgs);
> > > - memcpy(txn->invalidations, msgs,
> > > - sizeof(SharedInvalidationMessage) * nmsgs);
> > > + txn->txn_flags |= RBTXN_INVAL_ALL_CACHE;
> > > +
> > > + if (*invalidations)
> > > + {
> > > + pfree(*invalidations);
> > > + *invalidations = NULL;
> > > + *ninvalidations = 0;
> > > + }
> > >
> > > RBTXN_INVAL_ALL_CACHE seems to have an effect only on the distributed
> > > inval messages. One question is do we need to care about the overflow
> > > of txn->invalidations as well? If no, does it make sense to have a
> > > separate function like ReorderBufferAddDistributedInvalidtions()
> > > instead of having an extended version of
> > > ReorderBufferAddInvalidations()? Some common routines can also be
> > > declared as a static function if needed.
> >
> > Modified
> >
> > The attached v7 version patch has the changes for the same.
>
> Here is the patch, including updates for the back branches.
> The main difference from master is that the newly added structure
> members are appended at the end in the back branches to preserve
> compatibility. The invalidation addition logic remains consistent with
> master: a new function, ReorderBufferAddDistributedInvalidations, has
> been introduced to handle distributed invalidations. Shared logic
> between ReorderBufferAddInvalidations and
> ReorderBufferAddDistributedInvalidations has been factored out into a
> common helper, ReorderBufferAddInvalidationsCommon. This approach
> simplifies future merges and, in my assessment, does not introduce any
> backward compatibility issues.
>
Thank you for updating the patch. Here are some review comments:
+ req_mem_size = sizeof(SharedInvalidationMessage) *
(txn->ninvalidations_distr + nmsgs);
+
+ /*
+ * If the number of invalidation messages is larger than 8MB, it's more
+ * efficient to invalidate the entire cache rather than processing each
+ * message individually.
+ */
+ if (req_mem_size > (8 * 1024 * 1024) || rbtxn_inval_all_cache(txn))
It's better to define the maximum number of distributed inval messages
per transaction as a macro instead of calculating the memory size
every time.
---
+static void
+ReorderBufferAddInvalidationsCommon(ReorderBuffer *rb, TransactionId xid,
+ XLogRecPtr lsn, Size nmsgs,
+ SharedInvalidationMessage *msgs,
+ ReorderBufferTXN *txn,
+ bool for_inval)
This function is quite confusing to me. For instance,
ReorderBufferAddDistributedInvalidations() needs to call this function
with for_inval=false in spite of adding inval messages actually. Also,
the following condition seems not intuisive but there is no comment:
if (!for_inval || (for_inval && !rbtxn_inval_all_cache(txn)))
Instead of having ReorderBufferAddInvalidationsCommon(), I think we
can have a function say ReorderBufferQueueInvalidations() where we
enqueue the given inval messages as a
REORDER_BUFFER_CHANGE_INVALIDATION change.
ReorderBufferAddInvalidations() adds inval messages to
txn->invalidations and calls that function, while
ReorderBufferQueueInvalidations() adds inval messages to
txn->distributed_ivnalidations and calls that function if the array is
not full.
BTW if we need to invalidate all accumulated caches at the end of
transaction replay anyway, we don't need to add inval messages to
txn->invalidations once txn->distributed_invalidations gets full?
Regards,
--
Masahiko Sawada
Amazon Web Services: https://aws.amazon.com
^ permalink raw reply [nested|flat] 94+ messages in thread
* Re: Logical replication 'invalid memory alloc request size 1585837200' after upgrading to 17.5
@ 2025-06-02 18:08 Masahiko Sawada <sawada.mshk@gmail.com>
parent: Hayato Kuroda (Fujitsu) <kuroda.hayato@fujitsu.com>
0 siblings, 1 reply; 94+ messages in thread
From: Masahiko Sawada @ 2025-06-02 18:08 UTC (permalink / raw)
To: Hayato Kuroda (Fujitsu) <kuroda.hayato@fujitsu.com>; +Cc: pgsql-bugs@lists.postgresql.org <pgsql-bugs@lists.postgresql.org>; Amit Kapila <amit.kapila16@gmail.com>; Duncan Sands <duncan.sands@deepbluecap.com>; vignesh C <vignesh21@gmail.com>
On Mon, Jun 2, 2025 at 4:35 AM Hayato Kuroda (Fujitsu)
<kuroda.hayato@fujitsu.com> wrote:
>
> Dear Amit, Duncan, Vignesh, Sawada-san,
>
> I searched on GitHub and confirmed that no extensions use sizeof(ReorderBufferTXN).
> Based on that, can we proceed with the fix for backbenches?
> Note that I checked only repositories that are opened on GitHub. Proprietary
> ones and hosting on other services cannot be guaranteed.
>
> How I search
> ===========
> I searched on github with "sizeof(ReorderBufferTXN)", and 259 files were found.
> All of them were either "reorderbuffer.c" or "reorderbuffer.cpp". reorderbuffer.c
> came from the forked postgres repos, and "reorderbuffer.cpp" came from the
> openGauss project, which tries to enhance postgres. In both cases, they are
> server-side code and will rebase community's commits when we update the header
> file and sizeof(ReorderBufferTXN).
> In this check, I could not find extensions that refer to the size; only
> server-side codes were found. Based on that, we could extend the struct
> ReorderBufferTXN.
Thank you for checking the source codes on Github. I personally think
the chance that extensions depend on the ReorderBufferTXN size is low.
Without that, we would need more complex logic to store inval messages
per-transaction, which introduce a risk. So I agree with the current
solution.
Regards,
--
Masahiko Sawada
Amazon Web Services: https://aws.amazon.com
^ permalink raw reply [nested|flat] 94+ messages in thread
* Re: Logical replication 'invalid memory alloc request size 1585837200' after upgrading to 17.5
@ 2025-06-03 03:09 Amit Kapila <amit.kapila16@gmail.com>
parent: Masahiko Sawada <sawada.mshk@gmail.com>
0 siblings, 0 replies; 94+ messages in thread
From: Amit Kapila @ 2025-06-03 03:09 UTC (permalink / raw)
To: Masahiko Sawada <sawada.mshk@gmail.com>; +Cc: Hayato Kuroda (Fujitsu) <kuroda.hayato@fujitsu.com>; pgsql-bugs@lists.postgresql.org <pgsql-bugs@lists.postgresql.org>; Duncan Sands <duncan.sands@deepbluecap.com>; vignesh C <vignesh21@gmail.com>
On Mon, Jun 2, 2025 at 11:39 PM Masahiko Sawada <sawada.mshk@gmail.com> wrote:
>
> On Mon, Jun 2, 2025 at 4:35 AM Hayato Kuroda (Fujitsu)
> <kuroda.hayato@fujitsu.com> wrote:
> >
> > Dear Amit, Duncan, Vignesh, Sawada-san,
> >
> > I searched on GitHub and confirmed that no extensions use sizeof(ReorderBufferTXN).
> > Based on that, can we proceed with the fix for backbenches?
> > Note that I checked only repositories that are opened on GitHub. Proprietary
> > ones and hosting on other services cannot be guaranteed.
> >
> > How I search
> > ===========
> > I searched on github with "sizeof(ReorderBufferTXN)", and 259 files were found.
> > All of them were either "reorderbuffer.c" or "reorderbuffer.cpp". reorderbuffer.c
> > came from the forked postgres repos, and "reorderbuffer.cpp" came from the
> > openGauss project, which tries to enhance postgres. In both cases, they are
> > server-side code and will rebase community's commits when we update the header
> > file and sizeof(ReorderBufferTXN).
> > In this check, I could not find extensions that refer to the size; only
> > server-side codes were found. Based on that, we could extend the struct
> > ReorderBufferTXN.
>
> Thank you for checking the source codes on Github. I personally think
> the chance that extensions depend on the ReorderBufferTXN size is low.
> Without that, we would need more complex logic to store inval messages
> per-transaction, which introduce a risk. So I agree with the current
> solution.
>
It is difficult to predict whether proprietary extensions rely on the
sizeof ReorderBufferTXN, but I can't think of a better fix (where we
add new members at the end of ReorderBufferTXN) for backbranches. I
think we should explicitly mention this in the commit message, and in
the worst case, we need to request extension owners (that rely on
sizeof(ReorderBufferTXN)) to rebuild their extension.
--
With Regards,
Amit Kapila.
^ permalink raw reply [nested|flat] 94+ messages in thread
* Re: Logical replication 'invalid memory alloc request size 1585837200' after upgrading to 17.5
@ 2025-06-03 03:38 Amit Kapila <amit.kapila16@gmail.com>
parent: Masahiko Sawada <sawada.mshk@gmail.com>
0 siblings, 0 replies; 94+ messages in thread
From: Amit Kapila @ 2025-06-03 03:38 UTC (permalink / raw)
To: Masahiko Sawada <sawada.mshk@gmail.com>; +Cc: vignesh C <vignesh21@gmail.com>; Hayato Kuroda (Fujitsu) <kuroda.hayato@fujitsu.com>; Duncan Sands <duncan.sands@deepbluecap.com>; pgsql-bugs@lists.postgresql.org <pgsql-bugs@lists.postgresql.org>
On Mon, Jun 2, 2025 at 10:21 PM Masahiko Sawada <sawada.mshk@gmail.com> wrote:
>
> On Sun, Jun 1, 2025 at 8:13 PM vignesh C <vignesh21@gmail.com> wrote:
> >
> > On Sat, 31 May 2025 at 10:20, Amit Kapila <amit.kapila16@gmail.com> wrote:
> > >
> > >
> > > The concurrent abort handling is done for streaming and prepared
> > > transactions, where we send the transaction changes to the client
> > > before we read COMMIT/COMMIT PREPARED/ROLLBACK/ROLLBACK PREPARED. Now,
> > > among these COMMIT/ROLLBACK PREPARED cases are handled as part of a
> > > prepared transaction case. For ROLLBACK, we will never perform any
> > > changes from the current transaction, so we don't need distributed
> > > invalidations to be executed. For COMMIT, if we encounter any errors
> > > while processing changes (this is when we reach the ERROR path, which
> > > is not a concurrent abort), then we will reprocess all changes and, at
> > > the end, execute both the current transaction and distributed
> > > invalidations. Now, one possibility is that if, after ERROR, the
> > > caller does slot_advance to skip the ERROR, then we will probably miss
> > > executing the distributed invalidations, leading to data loss
> > > afterwards. If the above theory is correct, then it is better to
> > > execute distributed invalidation even in non-concurrent-abort cases in
> > > the ERROR path.
>
> This theory seems to be correct to me. If users re-try the logical
> decoding with smaller logical_decoding_work_mem the process could end
> up streaming the transaction with the wrong relcache before sending
> the committed transaction?
>
But will it give a problem? I think if a different transaction is
chosen for streaming after restart, then it should also have
distributed invalidation change queued for it at the appropriate
place, so that helps in making sure the cache is appropriate.
> > One possible reason this scenario may not occur is that
> > pg_logical_slot_get_changes_guts uses a PG_CATCH block to handle
> > exceptions, during which it calls InvalidateSystemCaches to clear the
> > system cache. Because of this, I believe the scenario might not
> > actually happen.
> > @Sawada-san / others — Are there any other cases where this could still occur?
>
> I think that we cannot confine use cases to walsender and
> pg_logical_slot_get_changes_guts() given that we expose logical
> decoding API at C level.
>
Hmm, yeah, that could be risky. Also, if we have to rely on the
InvalidateSystemCaches performed by the caller, then we don't even
need to execute the non-distributed invalidations. I think for the
sake of consistency and for the reason you mentioned, we can execute
distributed invalidations in the non-concurrent-abort ERROR path.
--
With Regards,
Amit Kapila.
^ permalink raw reply [nested|flat] 94+ messages in thread
* Re: Logical replication 'invalid memory alloc request size 1585837200' after upgrading to 17.5
@ 2025-06-03 07:07 vignesh C <vignesh21@gmail.com>
parent: Masahiko Sawada <sawada.mshk@gmail.com>
0 siblings, 2 replies; 94+ messages in thread
From: vignesh C @ 2025-06-03 07:07 UTC (permalink / raw)
To: Masahiko Sawada <sawada.mshk@gmail.com>; +Cc: Amit Kapila <amit.kapila16@gmail.com>; Hayato Kuroda (Fujitsu) <kuroda.hayato@fujitsu.com>; Duncan Sands <duncan.sands@deepbluecap.com>; pgsql-bugs@lists.postgresql.org <pgsql-bugs@lists.postgresql.org>
On Mon, 2 Jun 2025 at 22:49, Masahiko Sawada <sawada.mshk@gmail.com> wrote:
>
> Thank you for updating the patch. Here are some review comments:
>
> + req_mem_size = sizeof(SharedInvalidationMessage) *
> (txn->ninvalidations_distr + nmsgs);
> +
> + /*
> + * If the number of invalidation messages is larger than 8MB, it's more
> + * efficient to invalidate the entire cache rather than processing each
> + * message individually.
> + */
> + if (req_mem_size > (8 * 1024 * 1024) || rbtxn_inval_all_cache(txn))
>
> It's better to define the maximum number of distributed inval messages
> per transaction as a macro instead of calculating the memory size
> every time.
Modified
> ---
> +static void
> +ReorderBufferAddInvalidationsCommon(ReorderBuffer *rb, TransactionId xid,
> + XLogRecPtr lsn, Size nmsgs,
> + SharedInvalidationMessage *msgs,
> + ReorderBufferTXN *txn,
> + bool for_inval)
>
> This function is quite confusing to me. For instance,
> ReorderBufferAddDistributedInvalidations() needs to call this function
> with for_inval=false in spite of adding inval messages actually. Also,
> the following condition seems not intuisive but there is no comment:
>
> if (!for_inval || (for_inval && !rbtxn_inval_all_cache(txn)))
>
> Instead of having ReorderBufferAddInvalidationsCommon(), I think we
> can have a function say ReorderBufferQueueInvalidations() where we
> enqueue the given inval messages as a
> REORDER_BUFFER_CHANGE_INVALIDATION change.
> ReorderBufferAddInvalidations() adds inval messages to
> txn->invalidations and calls that function, while
> ReorderBufferQueueInvalidations() adds inval messages to
> txn->distributed_ivnalidations and calls that function if the array is
> not full.
Modified
> BTW if we need to invalidate all accumulated caches at the end of
> transaction replay anyway, we don't need to add inval messages to
> txn->invalidations once txn->distributed_invalidations gets full?
yes, no need to add invalidation messages to txn->invalidation once
RBTXN_INVAL_ALL_CACHE is set. This is handled now.
The attached v9 version patch has the changes for the same.
Regards,
Vignesh
Attachments:
[text/x-patch] v9-master-0001-Fix-exponential-memory-allocation-issue-in.patch (11.2K, ../../CALDaNm21OWoL66qpgVEsv0ZvgDLaiVkjtcU-YDVqQZVvo3NNPA@mail.gmail.com/2-v9-master-0001-Fix-exponential-memory-allocation-issue-in.patch)
download | inline diff:
From 4517d67fc14b95b2761381781949c2931cead70e Mon Sep 17 00:00:00 2001
From: Vignesh <vignesh21@gmail.com>
Date: Fri, 30 May 2025 12:06:07 +0530
Subject: [PATCH v9-master] Fix exponential memory allocation issue in logical
decoding due to invalidation message distribution
Commit 4909b38af0 added logic to distribute invalidation messages from
one transaction to all concurrent in-progress transactions. However,
since the source of each invalidation message was not tracked, transactions
could redundantly re-distribute already propagated messages. This led to an
exponential increase in invalidation messages and, ultimately, memory
allocation failures.
To fix this, a new field is introduced to track distributed invalidation
messages separately. This ensures that only transactions in the pre-existing
list receive the invalidations, preventing redundant propagation.
The accumulated distributed invalidations are now consumed at the end of
transaction processing in `ReorderBufferProcessTXN()`.
---
.../replication/logical/reorderbuffer.c | 174 +++++++++++++++---
src/backend/replication/logical/snapbuild.c | 5 +-
src/include/replication/reorderbuffer.h | 18 ++
3 files changed, 166 insertions(+), 31 deletions(-)
diff --git a/src/backend/replication/logical/reorderbuffer.c b/src/backend/replication/logical/reorderbuffer.c
index 67655111875..860e1d4b0bb 100644
--- a/src/backend/replication/logical/reorderbuffer.c
+++ b/src/backend/replication/logical/reorderbuffer.c
@@ -109,10 +109,18 @@
#include "storage/procarray.h"
#include "storage/sinval.h"
#include "utils/builtins.h"
+#include "utils/inval.h"
#include "utils/memutils.h"
#include "utils/rel.h"
#include "utils/relfilenumbermap.h"
+/*
+ * Maximum number of distributed invalidation messages per transaction.
+ * Each message is ~16 bytes, this allows up to 8 MB of invalidation
+ * message data.
+ */
+#define MAX_DISTR_INVAL_MSG_PER_TXN 524288
+
/* entry for a hash table we use to map from xid to our transaction state */
typedef struct ReorderBufferTXNByIdEnt
{
@@ -472,6 +480,12 @@ ReorderBufferFreeTXN(ReorderBuffer *rb, ReorderBufferTXN *txn)
txn->invalidations = NULL;
}
+ if (txn->invalidations_distributed)
+ {
+ pfree(txn->invalidations_distributed);
+ txn->invalidations_distributed = NULL;
+ }
+
/* Reset the toast hash */
ReorderBufferToastReset(rb, txn);
@@ -2661,7 +2675,14 @@ ReorderBufferProcessTXN(ReorderBuffer *rb, ReorderBufferTXN *txn,
AbortCurrentTransaction();
/* make sure there's no cache pollution */
- ReorderBufferExecuteInvalidations(txn->ninvalidations, txn->invalidations);
+ if (rbtxn_inval_all_cache(txn))
+ InvalidateSystemCaches();
+ else
+ {
+ ReorderBufferExecuteInvalidations(txn->ninvalidations, txn->invalidations);
+ ReorderBufferExecuteInvalidations(txn->ninvalidations_distributed,
+ txn->invalidations_distributed);
+ }
if (using_subtxn)
RollbackAndReleaseCurrentSubTransaction();
@@ -2710,8 +2731,14 @@ ReorderBufferProcessTXN(ReorderBuffer *rb, ReorderBufferTXN *txn,
AbortCurrentTransaction();
/* make sure there's no cache pollution */
- ReorderBufferExecuteInvalidations(txn->ninvalidations,
- txn->invalidations);
+ if (rbtxn_inval_all_cache(txn))
+ InvalidateSystemCaches();
+ else
+ {
+ ReorderBufferExecuteInvalidations(txn->ninvalidations, txn->invalidations);
+ ReorderBufferExecuteInvalidations(txn->ninvalidations_distributed,
+ txn->invalidations_distributed);
+ }
if (using_subtxn)
RollbackAndReleaseCurrentSubTransaction();
@@ -3421,6 +3448,55 @@ ReorderBufferAddNewTupleCids(ReorderBuffer *rb, TransactionId xid,
txn->ntuplecids++;
}
+/*
+ * Add a new invalidation message to the reorder buffer queue.
+ */
+static void
+ReorderBufferQueueInvalidations(ReorderBuffer *rb, TransactionId xid,
+ XLogRecPtr lsn, Size nmsgs,
+ SharedInvalidationMessage *msgs)
+{
+ ReorderBufferChange *change;
+
+ change = ReorderBufferAllocChange(rb);
+ change->action = REORDER_BUFFER_CHANGE_INVALIDATION;
+ change->data.inval.ninvalidations = nmsgs;
+ change->data.inval.invalidations = (SharedInvalidationMessage *)
+ palloc(sizeof(SharedInvalidationMessage) * nmsgs);
+ memcpy(change->data.inval.invalidations, msgs,
+ sizeof(SharedInvalidationMessage) * nmsgs);
+
+ ReorderBufferQueueChange(rb, xid, lsn, change, false);
+}
+
+/*
+ * Accumulate the invalidations.
+ */
+static void
+ReorderBufferAccumulateInvalidations(uint32 *ninvalidations,
+ SharedInvalidationMessage **invalidations,
+ Size nmsgs,
+ SharedInvalidationMessage *msgs)
+{
+ if (*ninvalidations == 0)
+ {
+ *ninvalidations = nmsgs;
+ *invalidations = (SharedInvalidationMessage *)
+ palloc(sizeof(SharedInvalidationMessage) * nmsgs);
+ memcpy(*invalidations, msgs, sizeof(SharedInvalidationMessage) * nmsgs);
+ }
+ else
+ {
+ *invalidations = (SharedInvalidationMessage *)
+ repalloc(*invalidations, sizeof(SharedInvalidationMessage) *
+ (*ninvalidations + nmsgs));
+
+ memcpy(*invalidations + *ninvalidations, msgs,
+ nmsgs * sizeof(SharedInvalidationMessage));
+ *ninvalidations += nmsgs;
+ }
+}
+
/*
* Accumulate the invalidations for executing them later.
*
@@ -3441,7 +3517,6 @@ ReorderBufferAddInvalidations(ReorderBuffer *rb, TransactionId xid,
{
ReorderBufferTXN *txn;
MemoryContext oldcontext;
- ReorderBufferChange *change;
txn = ReorderBufferTXNByXid(rb, xid, true, NULL, lsn, true);
@@ -3456,35 +3531,76 @@ ReorderBufferAddInvalidations(ReorderBuffer *rb, TransactionId xid,
Assert(nmsgs > 0);
- /* Accumulate invalidations. */
- if (txn->ninvalidations == 0)
- {
- txn->ninvalidations = nmsgs;
- txn->invalidations = (SharedInvalidationMessage *)
- palloc(sizeof(SharedInvalidationMessage) * nmsgs);
- memcpy(txn->invalidations, msgs,
- sizeof(SharedInvalidationMessage) * nmsgs);
- }
- else
+ /*
+ * If the complete cache will be invalidated, we don't need to accumulate
+ * the invalidations.
+ */
+ if (!rbtxn_inval_all_cache(txn))
+ ReorderBufferAccumulateInvalidations(&txn->ninvalidations,
+ &txn->invalidations, nmsgs, msgs);
+
+ ReorderBufferQueueInvalidations(rb, xid, lsn, nmsgs, msgs);
+
+ MemoryContextSwitchTo(oldcontext);
+}
+
+/*
+ * Accumulate the invalidations sent by committed transactions for executing
+ * them later.
+ *
+ * This needs to be called by committed transactions to distribute the
+ * invalidations to the in-progress transactions.
+ */
+void
+ReorderBufferAddDistributedInvalidations(ReorderBuffer *rb, TransactionId xid,
+ XLogRecPtr lsn, Size nmsgs,
+ SharedInvalidationMessage *msgs)
+{
+ ReorderBufferTXN *txn;
+ MemoryContext oldcontext;
+
+ txn = ReorderBufferTXNByXid(rb, xid, true, NULL, lsn, true);
+
+ oldcontext = MemoryContextSwitchTo(rb->context);
+
+ /*
+ * Collect all the invalidations under the top transaction, if available,
+ * so that we can execute them all together. See comments
+ * ReorderBufferAddInvalidations.
+ */
+ txn = rbtxn_get_toptxn(txn);
+
+ Assert(nmsgs > 0);
+
+ /*
+ * If the number of invalidation messages is high, performing a full cache
+ * invalidation is more efficient than handling each message separately.
+ */
+ if (((nmsgs + txn->ninvalidations_distributed) > MAX_DISTR_INVAL_MSG_PER_TXN) ||
+ rbtxn_inval_all_cache(txn))
{
- txn->invalidations = (SharedInvalidationMessage *)
- repalloc(txn->invalidations, sizeof(SharedInvalidationMessage) *
- (txn->ninvalidations + nmsgs));
+ txn->txn_flags |= RBTXN_INVAL_ALL_CACHE;
- memcpy(txn->invalidations + txn->ninvalidations, msgs,
- nmsgs * sizeof(SharedInvalidationMessage));
- txn->ninvalidations += nmsgs;
- }
+ if (txn->invalidations_distributed)
+ {
+ pfree(txn->invalidations_distributed);
+ txn->invalidations_distributed = NULL;
+ txn->ninvalidations_distributed = 0;
+ }
- change = ReorderBufferAllocChange(rb);
- change->action = REORDER_BUFFER_CHANGE_INVALIDATION;
- change->data.inval.ninvalidations = nmsgs;
- change->data.inval.invalidations = (SharedInvalidationMessage *)
- palloc(sizeof(SharedInvalidationMessage) * nmsgs);
- memcpy(change->data.inval.invalidations, msgs,
- sizeof(SharedInvalidationMessage) * nmsgs);
+ if (txn->invalidations)
+ {
+ pfree(txn->invalidations);
+ txn->invalidations = NULL;
+ txn->ninvalidations = 0;
+ }
+ }
+ else
+ ReorderBufferAccumulateInvalidations(&txn->ninvalidations_distributed,
+ &txn->invalidations_distributed,
+ nmsgs, msgs);
- ReorderBufferQueueChange(rb, xid, lsn, change, false);
+ ReorderBufferQueueInvalidations(rb, xid, lsn, nmsgs, msgs);
MemoryContextSwitchTo(oldcontext);
}
diff --git a/src/backend/replication/logical/snapbuild.c b/src/backend/replication/logical/snapbuild.c
index 0d7bddbe4ed..3e052ea1d58 100644
--- a/src/backend/replication/logical/snapbuild.c
+++ b/src/backend/replication/logical/snapbuild.c
@@ -807,8 +807,9 @@ SnapBuildDistributeSnapshotAndInval(SnapBuild *builder, XLogRecPtr lsn, Transact
{
Assert(msgs != NULL);
- ReorderBufferAddInvalidations(builder->reorder, txn->xid, lsn,
- ninvalidations, msgs);
+ ReorderBufferAddDistributedInvalidations(builder->reorder,
+ txn->xid, lsn,
+ ninvalidations, msgs);
}
}
}
diff --git a/src/include/replication/reorderbuffer.h b/src/include/replication/reorderbuffer.h
index 24e88c409ba..15d839f3d90 100644
--- a/src/include/replication/reorderbuffer.h
+++ b/src/include/replication/reorderbuffer.h
@@ -176,6 +176,7 @@ typedef struct ReorderBufferChange
#define RBTXN_SENT_PREPARE 0x0200
#define RBTXN_IS_COMMITTED 0x0400
#define RBTXN_IS_ABORTED 0x0800
+#define RBTXN_INVAL_ALL_CACHE 0x1000
#define RBTXN_PREPARE_STATUS_MASK (RBTXN_IS_PREPARED | RBTXN_SKIPPED_PREPARE | RBTXN_SENT_PREPARE)
@@ -265,6 +266,12 @@ typedef struct ReorderBufferChange
((txn)->txn_flags & RBTXN_SKIPPED_PREPARE) != 0 \
)
+/* Should the complete cache be invalidated? */
+#define rbtxn_inval_all_cache(txn) \
+( \
+ ((txn)->txn_flags & RBTXN_INVAL_ALL_CACHE) != 0 \
+)
+
/* Is this a top-level transaction? */
#define rbtxn_is_toptxn(txn) \
( \
@@ -422,6 +429,14 @@ typedef struct ReorderBufferTXN
uint32 ninvalidations;
SharedInvalidationMessage *invalidations;
+ /*
+ * Stores the cache invalidation messages distributed by the committing
+ * transaction.
+ */
+ uint32 ninvalidations_distributed;
+ SharedInvalidationMessage *invalidations_distributed;
+
+
/* ---
* Position in one of two lists:
* * list of subtransactions if we are *known* to be subxact
@@ -738,6 +753,9 @@ extern void ReorderBufferAddNewTupleCids(ReorderBuffer *rb, TransactionId xid,
CommandId cmin, CommandId cmax, CommandId combocid);
extern void ReorderBufferAddInvalidations(ReorderBuffer *rb, TransactionId xid, XLogRecPtr lsn,
Size nmsgs, SharedInvalidationMessage *msgs);
+extern void ReorderBufferAddDistributedInvalidations(ReorderBuffer *rb, TransactionId xid,
+ XLogRecPtr lsn, Size nmsgs,
+ SharedInvalidationMessage *msgs);
extern void ReorderBufferImmediateInvalidation(ReorderBuffer *rb, uint32 ninvalidations,
SharedInvalidationMessage *invalidations);
extern void ReorderBufferProcessXid(ReorderBuffer *rb, TransactionId xid, XLogRecPtr lsn);
--
2.43.0
^ permalink raw reply [nested|flat] 94+ messages in thread
* Re: Logical replication 'invalid memory alloc request size 1585837200' after upgrading to 17.5
@ 2025-06-03 12:24 vignesh C <vignesh21@gmail.com>
parent: vignesh C <vignesh21@gmail.com>
1 sibling, 0 replies; 94+ messages in thread
From: vignesh C @ 2025-06-03 12:24 UTC (permalink / raw)
To: Masahiko Sawada <sawada.mshk@gmail.com>; +Cc: Amit Kapila <amit.kapila16@gmail.com>; Hayato Kuroda (Fujitsu) <kuroda.hayato@fujitsu.com>; Duncan Sands <duncan.sands@deepbluecap.com>; pgsql-bugs@lists.postgresql.org <pgsql-bugs@lists.postgresql.org>
On Tue, 3 Jun 2025 at 12:37, vignesh C <vignesh21@gmail.com> wrote:
>
> On Mon, 2 Jun 2025 at 22:49, Masahiko Sawada <sawada.mshk@gmail.com> wrote:
> >
> > Thank you for updating the patch. Here are some review comments:
> >
> > + req_mem_size = sizeof(SharedInvalidationMessage) *
> > (txn->ninvalidations_distr + nmsgs);
> > +
> > + /*
> > + * If the number of invalidation messages is larger than 8MB, it's more
> > + * efficient to invalidate the entire cache rather than processing each
> > + * message individually.
> > + */
> > + if (req_mem_size > (8 * 1024 * 1024) || rbtxn_inval_all_cache(txn))
> >
> > It's better to define the maximum number of distributed inval messages
> > per transaction as a macro instead of calculating the memory size
> > every time.
>
> Modified
>
> > ---
> > +static void
> > +ReorderBufferAddInvalidationsCommon(ReorderBuffer *rb, TransactionId xid,
> > + XLogRecPtr lsn, Size nmsgs,
> > + SharedInvalidationMessage *msgs,
> > + ReorderBufferTXN *txn,
> > + bool for_inval)
> >
> > This function is quite confusing to me. For instance,
> > ReorderBufferAddDistributedInvalidations() needs to call this function
> > with for_inval=false in spite of adding inval messages actually. Also,
> > the following condition seems not intuisive but there is no comment:
> >
> > if (!for_inval || (for_inval && !rbtxn_inval_all_cache(txn)))
> >
> > Instead of having ReorderBufferAddInvalidationsCommon(), I think we
> > can have a function say ReorderBufferQueueInvalidations() where we
> > enqueue the given inval messages as a
> > REORDER_BUFFER_CHANGE_INVALIDATION change.
> > ReorderBufferAddInvalidations() adds inval messages to
> > txn->invalidations and calls that function, while
> > ReorderBufferQueueInvalidations() adds inval messages to
> > txn->distributed_ivnalidations and calls that function if the array is
> > not full.
>
> Modified
>
> > BTW if we need to invalidate all accumulated caches at the end of
> > transaction replay anyway, we don't need to add inval messages to
> > txn->invalidations once txn->distributed_invalidations gets full?
>
> yes, no need to add invalidation messages to txn->invalidation once
> RBTXN_INVAL_ALL_CACHE is set. This is handled now.
>
> The attached v9 version patch has the changes for the same.
I've posted the patch for the master branch only. I'll submit the
back-branch patches once this patch is in a committable state.
Regards,
Vignesh
^ permalink raw reply [nested|flat] 94+ messages in thread
* Re: Logical replication 'invalid memory alloc request size 1585837200' after upgrading to 17.5
@ 2025-06-03 19:43 Masahiko Sawada <sawada.mshk@gmail.com>
parent: vignesh C <vignesh21@gmail.com>
1 sibling, 2 replies; 94+ messages in thread
From: Masahiko Sawada @ 2025-06-03 19:43 UTC (permalink / raw)
To: vignesh C <vignesh21@gmail.com>; +Cc: Amit Kapila <amit.kapila16@gmail.com>; Hayato Kuroda (Fujitsu) <kuroda.hayato@fujitsu.com>; Duncan Sands <duncan.sands@deepbluecap.com>; pgsql-bugs@lists.postgresql.org <pgsql-bugs@lists.postgresql.org>
On Tue, Jun 3, 2025 at 12:07 AM vignesh C <vignesh21@gmail.com> wrote:
>
> On Mon, 2 Jun 2025 at 22:49, Masahiko Sawada <sawada.mshk@gmail.com> wrote:
> >
> > Thank you for updating the patch. Here are some review comments:
> >
> > + req_mem_size = sizeof(SharedInvalidationMessage) *
> > (txn->ninvalidations_distr + nmsgs);
> > +
> > + /*
> > + * If the number of invalidation messages is larger than 8MB, it's more
> > + * efficient to invalidate the entire cache rather than processing each
> > + * message individually.
> > + */
> > + if (req_mem_size > (8 * 1024 * 1024) || rbtxn_inval_all_cache(txn))
> >
> > It's better to define the maximum number of distributed inval messages
> > per transaction as a macro instead of calculating the memory size
> > every time.
>
> Modified
>
> > ---
> > +static void
> > +ReorderBufferAddInvalidationsCommon(ReorderBuffer *rb, TransactionId xid,
> > + XLogRecPtr lsn, Size nmsgs,
> > + SharedInvalidationMessage *msgs,
> > + ReorderBufferTXN *txn,
> > + bool for_inval)
> >
> > This function is quite confusing to me. For instance,
> > ReorderBufferAddDistributedInvalidations() needs to call this function
> > with for_inval=false in spite of adding inval messages actually. Also,
> > the following condition seems not intuisive but there is no comment:
> >
> > if (!for_inval || (for_inval && !rbtxn_inval_all_cache(txn)))
> >
> > Instead of having ReorderBufferAddInvalidationsCommon(), I think we
> > can have a function say ReorderBufferQueueInvalidations() where we
> > enqueue the given inval messages as a
> > REORDER_BUFFER_CHANGE_INVALIDATION change.
> > ReorderBufferAddInvalidations() adds inval messages to
> > txn->invalidations and calls that function, while
> > ReorderBufferQueueInvalidations() adds inval messages to
> > txn->distributed_ivnalidations and calls that function if the array is
> > not full.
>
> Modified
>
> > BTW if we need to invalidate all accumulated caches at the end of
> > transaction replay anyway, we don't need to add inval messages to
> > txn->invalidations once txn->distributed_invalidations gets full?
>
> yes, no need to add invalidation messages to txn->invalidation once
> RBTXN_INVAL_ALL_CACHE is set. This is handled now.
>
> The attached v9 version patch has the changes for the same.
Thank you for updating the patch. Here are review comments on v9 patch:
+/*
+ * Maximum number of distributed invalidation messages per transaction.
+ * Each message is ~16 bytes, this allows up to 8 MB of invalidation
+ * message data.
+ */
+#define MAX_DISTR_INVAL_MSG_PER_TXN 524288
The size of SharedInvalidationMessage could change in the future so we
should calculate it at compile time.
---
+ /*
+ * If the complete cache will be invalidated, we don't need to accumulate
+ * the invalidations.
+ */
+ if (!rbtxn_inval_all_cache(txn))
+ ReorderBufferAccumulateInvalidations(&txn->ninvalidations,
+ &txn->invalidations, nmsgs, msgs);
We need to explain why we don't check the number of invalidation
messages for txn->invalidations and mark it as inval-all-cache, unlike
ReorderBufferAddDistributedInvalidations().
---
+ /*
+ * If the number of invalidation messages is high, performing a full cache
+ * invalidation is more efficient than handling each message separately.
+ */
+ if (((nmsgs + txn->ninvalidations_distributed) >
MAX_DISTR_INVAL_MSG_PER_TXN) ||
+ rbtxn_inval_all_cache(txn))
{
- txn->invalidations = (SharedInvalidationMessage *)
- repalloc(txn->invalidations, sizeof(SharedInvalidationMessage) *
- (txn->ninvalidations + nmsgs));
+ txn->txn_flags |= RBTXN_INVAL_ALL_CACHE;
I think we don't need to mark the transaction as RBTXN_INVAL_ALL_CACHE
again. I'd rewrite the logic as follows:
if (txn->ninvalidations_distributed + nmsgs >= MAX_DISTR_INVAL_MSG_PER_TXN)
{
/* mark the txn as inval-all-cache */
....
/* free the accumulated inval msgs */
....
}
if (!rbtxn_inval_all_cache(txn))
ReorderBufferAccumulateInvalidations(...);
---
- ReorderBufferAddInvalidations(builder->reorder, txn->xid, lsn,
- ninvalidations, msgs);
+ ReorderBufferAddDistributedInvalidations(builder->reorder,
+ txn->xid, lsn,
+ ninvalidations, msgs);
I think we need some comments here to explain why we need to
distribute only inval messages coming from the current transaction.
---
+/* Should the complete cache be invalidated? */
+#define rbtxn_inval_all_cache(txn) \
+( \
+ ((txn)->txn_flags & RBTXN_INVAL_ALL_CACHE) != 0 \
+)
I find that if we rename the flag to something like
RBTXN_INVAL_OVERFLOWED, it would explain the state of the transaction
clearer.
---
Can we have a reasonable test case that covers the inval message overflow cases?
I've attached a patch for some changes and adding more comments (note
that it still has XXX comments). Please include these changes that you
agreed with in the next version patch.
Regards,
--
Masahiko Sawada
Amazon Web Services: https://aws.amazon.com
Attachments:
[application/octet-stream] change_v9_masahiko.patch (8.8K, ../../CAD21AoBhDBqavSkxr+0GCxom1Q7P7guY5Ees20Wda=YZLFVfCA@mail.gmail.com/2-change_v9_masahiko.patch)
download | inline diff:
diff --git a/src/backend/replication/logical/reorderbuffer.c b/src/backend/replication/logical/reorderbuffer.c
index 860e1d4b0bb..7a500e30c74 100644
--- a/src/backend/replication/logical/reorderbuffer.c
+++ b/src/backend/replication/logical/reorderbuffer.c
@@ -115,11 +115,14 @@
#include "utils/relfilenumbermap.h"
/*
- * Maximum number of distributed invalidation messages per transaction.
- * Each message is ~16 bytes, this allows up to 8 MB of invalidation
- * message data.
+ * Each transaction is limited to a maximum of 8MV of inval messages distributed
+ * from other transaction. Once the number of distributed inval messages reach
+ * this threshold, the transaction is marked as RBTXN_INVAL_OVERFLOWED,
+ * invalidating all caches instead as we have lost some inval messages and hence
+ * don't know what needs to be invalidated.
*/
-#define MAX_DISTR_INVAL_MSG_PER_TXN 524288
+#define MAX_DISTR_INVAL_MSG_PER_TXN \
+ ((8 * 1024 * 1024) / sizeof(SharedInvalidationMessage))
/* entry for a hash table we use to map from xid to our transaction state */
typedef struct ReorderBufferTXNByIdEnt
@@ -2675,8 +2678,11 @@ ReorderBufferProcessTXN(ReorderBuffer *rb, ReorderBufferTXN *txn,
AbortCurrentTransaction();
/* make sure there's no cache pollution */
- if (rbtxn_inval_all_cache(txn))
+ if (rbtxn_inval_overflowed(txn))
+ {
+ Assert(txn->ninvalidations == 0 && txn->ninvalidations_distributed == 0);
InvalidateSystemCaches();
+ }
else
{
ReorderBufferExecuteInvalidations(txn->ninvalidations, txn->invalidations);
@@ -2731,8 +2737,11 @@ ReorderBufferProcessTXN(ReorderBuffer *rb, ReorderBufferTXN *txn,
AbortCurrentTransaction();
/* make sure there's no cache pollution */
- if (rbtxn_inval_all_cache(txn))
+ if (rbtxn_inval_overflowed(txn))
+ {
+ Assert(txn->ninvalidations == 0 && txn->ninvalidations_distributed == 0);
InvalidateSystemCaches();
+ }
else
{
ReorderBufferExecuteInvalidations(txn->ninvalidations, txn->invalidations);
@@ -3470,30 +3479,32 @@ ReorderBufferQueueInvalidations(ReorderBuffer *rb, TransactionId xid,
}
/*
- * Accumulate the invalidations.
+ * A helper function for ReorderBufferAddInvalidations() and
+ * ReorderBufferAddDistributedInvalidations() to accumulate the invalidation
+ * messages to the **invals_out.
*/
static void
-ReorderBufferAccumulateInvalidations(uint32 *ninvalidations,
- SharedInvalidationMessage **invalidations,
- Size nmsgs,
- SharedInvalidationMessage *msgs)
+ReorderBufferAccumulateInvalidations(SharedInvalidationMessage **invals_out,
+ uint32 *ninvals_out,
+ SharedInvalidationMessage *msgs_new,
+ Size nmsgs_new)
{
- if (*ninvalidations == 0)
+ if (*ninvals_out == 0)
{
- *ninvalidations = nmsgs;
- *invalidations = (SharedInvalidationMessage *)
- palloc(sizeof(SharedInvalidationMessage) * nmsgs);
- memcpy(*invalidations, msgs, sizeof(SharedInvalidationMessage) * nmsgs);
+ *ninvals_out = nmsgs_new;
+ *invals_out = (SharedInvalidationMessage *)
+ palloc(sizeof(SharedInvalidationMessage) * nmsgs_new);
+ memcpy(*invals_out, msgs_new, sizeof(SharedInvalidationMessage) * nmsgs_new);
}
else
{
- *invalidations = (SharedInvalidationMessage *)
- repalloc(*invalidations, sizeof(SharedInvalidationMessage) *
- (*ninvalidations + nmsgs));
-
- memcpy(*invalidations + *ninvalidations, msgs,
- nmsgs * sizeof(SharedInvalidationMessage));
- *ninvalidations += nmsgs;
+ /* Enlarge the array of inval messages */
+ *invals_out = (SharedInvalidationMessage *)
+ repalloc(*invals_out, sizeof(SharedInvalidationMessage) *
+ (*ninvals_out + nmsgs_new));
+ memcpy(*invals_out + *ninvals_out, msgs_new,
+ nmsgs_new * sizeof(SharedInvalidationMessage));
+ *ninvals_out += nmsgs_new;
}
}
@@ -3532,12 +3543,15 @@ ReorderBufferAddInvalidations(ReorderBuffer *rb, TransactionId xid,
Assert(nmsgs > 0);
/*
- * If the complete cache will be invalidated, we don't need to accumulate
- * the invalidations.
+ * Accumulate the invalidation messages unless it's overflowed.
+ *
+ * XXX: need to explain here why we don't try to mark it as overflowed
+ * unlike ReorderBufferAddDistributedInvalidations().
*/
- if (!rbtxn_inval_all_cache(txn))
- ReorderBufferAccumulateInvalidations(&txn->ninvalidations,
- &txn->invalidations, nmsgs, msgs);
+ if (!rbtxn_inval_overflowed(txn))
+ ReorderBufferAccumulateInvalidations(&txn->invalidations,
+ &txn->ninvalidations,
+ msgs, nmsgs);
ReorderBufferQueueInvalidations(rb, xid, lsn, nmsgs, msgs);
@@ -3545,11 +3559,15 @@ ReorderBufferAddInvalidations(ReorderBuffer *rb, TransactionId xid,
}
/*
- * Accumulate the invalidations sent by committed transactions for executing
- * them later.
+ * Accumulate the invalidations distributed by other committed transactions
+ * for executing them later.
*
- * This needs to be called by committed transactions to distribute the
- * invalidations to the in-progress transactions.
+ * This function is similar to ReorderBufferAddInvalidations() but stores
+ * the given inval messages to the txn->invalidations_distributed with the
+ * overflow check.
+ *
+ * This needs to be called by committed transactions to distribute their
+ * inval messages to in-progress transactions.
*/
void
ReorderBufferAddDistributedInvalidations(ReorderBuffer *rb, TransactionId xid,
@@ -3573,13 +3591,16 @@ ReorderBufferAddDistributedInvalidations(ReorderBuffer *rb, TransactionId xid,
Assert(nmsgs > 0);
/*
- * If the number of invalidation messages is high, performing a full cache
- * invalidation is more efficient than handling each message separately.
+ * Check the transaction has enough space for storing distributed
+ * invalidation messages.
*/
- if (((nmsgs + txn->ninvalidations_distributed) > MAX_DISTR_INVAL_MSG_PER_TXN) ||
- rbtxn_inval_all_cache(txn))
+ if (txn->ninvalidations_distributed + nmsgs >= MAX_DISTR_INVAL_MSG_PER_TXN)
{
- txn->txn_flags |= RBTXN_INVAL_ALL_CACHE;
+ /*
+ * Mark the invalidation message as overflowed and free up the
+ * messages accumulated so far.
+ */
+ txn->txn_flags |= RBTXN_INVAL_OVERFLOWED;
if (txn->invalidations_distributed)
{
@@ -3595,11 +3616,13 @@ ReorderBufferAddDistributedInvalidations(ReorderBuffer *rb, TransactionId xid,
txn->ninvalidations = 0;
}
}
- else
- ReorderBufferAccumulateInvalidations(&txn->ninvalidations_distributed,
- &txn->invalidations_distributed,
- nmsgs, msgs);
+ if (!rbtxn_inval_overflowed(txn))
+ ReorderBufferAccumulateInvalidations(&txn->invalidations_distributed,
+ &txn->ninvalidations_distributed,
+ msgs, nmsgs);
+
+ /* Queue the invalidation messages into the transaction */
ReorderBufferQueueInvalidations(rb, xid, lsn, nmsgs, msgs);
MemoryContextSwitchTo(oldcontext);
diff --git a/src/backend/replication/logical/snapbuild.c b/src/backend/replication/logical/snapbuild.c
index 3e052ea1d58..c334e550342 100644
--- a/src/backend/replication/logical/snapbuild.c
+++ b/src/backend/replication/logical/snapbuild.c
@@ -794,6 +794,9 @@ SnapBuildDistributeSnapshotAndInval(SnapBuild *builder, XLogRecPtr lsn, Transact
* contents built by the current transaction even after its decoding,
* which should have been invalidated due to concurrent catalog
* changing transaction.
+ *
+ * XXX: need to explain here why we need to distribute only inval
+ * messages coming from the current committed transactions.
*/
if (txn->xid != xid)
{
diff --git a/src/include/replication/reorderbuffer.h b/src/include/replication/reorderbuffer.h
index 15d839f3d90..d4fbe5ff709 100644
--- a/src/include/replication/reorderbuffer.h
+++ b/src/include/replication/reorderbuffer.h
@@ -176,7 +176,7 @@ typedef struct ReorderBufferChange
#define RBTXN_SENT_PREPARE 0x0200
#define RBTXN_IS_COMMITTED 0x0400
#define RBTXN_IS_ABORTED 0x0800
-#define RBTXN_INVAL_ALL_CACHE 0x1000
+#define RBTXN_INVAL_OVERFLOWED 0x1000
#define RBTXN_PREPARE_STATUS_MASK (RBTXN_IS_PREPARED | RBTXN_SKIPPED_PREPARE | RBTXN_SENT_PREPARE)
@@ -266,10 +266,10 @@ typedef struct ReorderBufferChange
((txn)->txn_flags & RBTXN_SKIPPED_PREPARE) != 0 \
)
-/* Should the complete cache be invalidated? */
-#define rbtxn_inval_all_cache(txn) \
+/* Is the array of inval message overflowed? */
+#define rbtxn_inval_overflowed(txn) \
( \
- ((txn)->txn_flags & RBTXN_INVAL_ALL_CACHE) != 0 \
+ ((txn)->txn_flags & RBTXN_INVAL_OVERFLOWED) != 0 \
)
/* Is this a top-level transaction? */
@@ -430,8 +430,7 @@ typedef struct ReorderBufferTXN
SharedInvalidationMessage *invalidations;
/*
- * Stores the cache invalidation messages distributed by the committing
- * transaction.
+ * Stores cache invalidation messages distributed by other transactions.
*/
uint32 ninvalidations_distributed;
SharedInvalidationMessage *invalidations_distributed;
^ permalink raw reply [nested|flat] 94+ messages in thread
* Re: Logical replication 'invalid memory alloc request size 1585837200' after upgrading to 17.5
@ 2025-06-04 06:47 vignesh C <vignesh21@gmail.com>
parent: Masahiko Sawada <sawada.mshk@gmail.com>
1 sibling, 1 reply; 94+ messages in thread
From: vignesh C @ 2025-06-04 06:47 UTC (permalink / raw)
To: Masahiko Sawada <sawada.mshk@gmail.com>; +Cc: Amit Kapila <amit.kapila16@gmail.com>; Hayato Kuroda (Fujitsu) <kuroda.hayato@fujitsu.com>; Duncan Sands <duncan.sands@deepbluecap.com>; pgsql-bugs@lists.postgresql.org <pgsql-bugs@lists.postgresql.org>
On Wed, 4 Jun 2025 at 01:14, Masahiko Sawada <sawada.mshk@gmail.com> wrote:
>
> On Tue, Jun 3, 2025 at 12:07 AM vignesh C <vignesh21@gmail.com> wrote:
> >
> > On Mon, 2 Jun 2025 at 22:49, Masahiko Sawada <sawada.mshk@gmail.com> wrote:
> > >
> > > Thank you for updating the patch. Here are some review comments:
> > >
> > > + req_mem_size = sizeof(SharedInvalidationMessage) *
> > > (txn->ninvalidations_distr + nmsgs);
> > > +
> > > + /*
> > > + * If the number of invalidation messages is larger than 8MB, it's more
> > > + * efficient to invalidate the entire cache rather than processing each
> > > + * message individually.
> > > + */
> > > + if (req_mem_size > (8 * 1024 * 1024) || rbtxn_inval_all_cache(txn))
> > >
> > > It's better to define the maximum number of distributed inval messages
> > > per transaction as a macro instead of calculating the memory size
> > > every time.
> >
> > Modified
> >
> > > ---
> > > +static void
> > > +ReorderBufferAddInvalidationsCommon(ReorderBuffer *rb, TransactionId xid,
> > > + XLogRecPtr lsn, Size nmsgs,
> > > + SharedInvalidationMessage *msgs,
> > > + ReorderBufferTXN *txn,
> > > + bool for_inval)
> > >
> > > This function is quite confusing to me. For instance,
> > > ReorderBufferAddDistributedInvalidations() needs to call this function
> > > with for_inval=false in spite of adding inval messages actually. Also,
> > > the following condition seems not intuisive but there is no comment:
> > >
> > > if (!for_inval || (for_inval && !rbtxn_inval_all_cache(txn)))
> > >
> > > Instead of having ReorderBufferAddInvalidationsCommon(), I think we
> > > can have a function say ReorderBufferQueueInvalidations() where we
> > > enqueue the given inval messages as a
> > > REORDER_BUFFER_CHANGE_INVALIDATION change.
> > > ReorderBufferAddInvalidations() adds inval messages to
> > > txn->invalidations and calls that function, while
> > > ReorderBufferQueueInvalidations() adds inval messages to
> > > txn->distributed_ivnalidations and calls that function if the array is
> > > not full.
> >
> > Modified
> >
> > > BTW if we need to invalidate all accumulated caches at the end of
> > > transaction replay anyway, we don't need to add inval messages to
> > > txn->invalidations once txn->distributed_invalidations gets full?
> >
> > yes, no need to add invalidation messages to txn->invalidation once
> > RBTXN_INVAL_ALL_CACHE is set. This is handled now.
> >
> > The attached v9 version patch has the changes for the same.
>
> Thank you for updating the patch. Here are review comments on v9 patch:
>
> +/*
> + * Maximum number of distributed invalidation messages per transaction.
> + * Each message is ~16 bytes, this allows up to 8 MB of invalidation
> + * message data.
> + */
> +#define MAX_DISTR_INVAL_MSG_PER_TXN 524288
>
> The size of SharedInvalidationMessage could change in the future so we
> should calculate it at compile time.
Modified
> ---
> + /*
> + * If the complete cache will be invalidated, we don't need to accumulate
> + * the invalidations.
> + */
> + if (!rbtxn_inval_all_cache(txn))
> + ReorderBufferAccumulateInvalidations(&txn->ninvalidations,
> + &txn->invalidations, nmsgs, msgs);
>
> We need to explain why we don't check the number of invalidation
> messages for txn->invalidations and mark it as inval-all-cache, unlike
> ReorderBufferAddDistributedInvalidations().
Added comments
> ---
> + /*
> + * If the number of invalidation messages is high, performing a full cache
> + * invalidation is more efficient than handling each message separately.
> + */
> + if (((nmsgs + txn->ninvalidations_distributed) >
> MAX_DISTR_INVAL_MSG_PER_TXN) ||
> + rbtxn_inval_all_cache(txn))
> {
> - txn->invalidations = (SharedInvalidationMessage *)
> - repalloc(txn->invalidations, sizeof(SharedInvalidationMessage) *
> - (txn->ninvalidations + nmsgs));
> + txn->txn_flags |= RBTXN_INVAL_ALL_CACHE;
>
> I think we don't need to mark the transaction as RBTXN_INVAL_ALL_CACHE
> again. I'd rewrite the logic as follows:
>
> if (txn->ninvalidations_distributed + nmsgs >= MAX_DISTR_INVAL_MSG_PER_TXN)
> {
> /* mark the txn as inval-all-cache */
> ....
> /* free the accumulated inval msgs */
> ....
> }
>
> if (!rbtxn_inval_all_cache(txn))
> ReorderBufferAccumulateInvalidations(...);
Modified
> ---
> - ReorderBufferAddInvalidations(builder->reorder, txn->xid, lsn,
> - ninvalidations, msgs);
> + ReorderBufferAddDistributedInvalidations(builder->reorder,
> + txn->xid, lsn,
> + ninvalidations, msgs);
>
> I think we need some comments here to explain why we need to
> distribute only inval messages coming from the current transaction.
Added comments
> ---
> +/* Should the complete cache be invalidated? */
> +#define rbtxn_inval_all_cache(txn) \
> +( \
> + ((txn)->txn_flags & RBTXN_INVAL_ALL_CACHE) != 0 \
> +)
>
> I find that if we rename the flag to something like
> RBTXN_INVAL_OVERFLOWED, it would explain the state of the transaction
> clearer.
Modified
> Can we have a reasonable test case that covers the inval message overflow cases?
One of us will work on this and post a separate patch
> I've attached a patch for some changes and adding more comments (note
> that it still has XXX comments). Please include these changes that you
> agreed with in the next version patch.
Thanks for the comments, I merged it.
The attached v10 version patch has the changes for the same.
Regards,
Vignesh
Attachments:
[application/octet-stream] v10-master-0001-Fix-exponential-memory-allocation-issue-i.patch (13.0K, ../../CALDaNm1MMafe_Xr3RFc0t3ds82W4CrR4=Ewi1Vh3uscpZ-rW0Q@mail.gmail.com/2-v10-master-0001-Fix-exponential-memory-allocation-issue-i.patch)
download | inline diff:
From 67c08b93ba188559b2c2aafc0f9f0d62ca8db78f Mon Sep 17 00:00:00 2001
From: Vignesh <vignesh21@gmail.com>
Date: Fri, 30 May 2025 12:06:07 +0530
Subject: [PATCH v10-master] Fix exponential memory allocation issue in logical
decoding due to invalidation message distribution
Commit 4909b38af0 added logic to distribute invalidation messages from
one transaction to all concurrent in-progress transactions. However,
since the source of each invalidation message was not tracked, transactions
could redundantly re-distribute already propagated messages. This led to an
exponential increase in invalidation messages and, ultimately, memory
allocation failures.
To fix this, a new field is introduced to track distributed invalidation
messages separately. This ensures that only transactions in the pre-existing
list receive the invalidations, preventing redundant propagation.
The accumulated distributed invalidations are now consumed at the end of
transaction processing in `ReorderBufferProcessTXN()`.
---
.../replication/logical/reorderbuffer.c | 197 +++++++++++++++---
src/backend/replication/logical/snapbuild.c | 12 +-
src/include/replication/reorderbuffer.h | 17 ++
3 files changed, 196 insertions(+), 30 deletions(-)
diff --git a/src/backend/replication/logical/reorderbuffer.c b/src/backend/replication/logical/reorderbuffer.c
index 67655111875..e05259bf693 100644
--- a/src/backend/replication/logical/reorderbuffer.c
+++ b/src/backend/replication/logical/reorderbuffer.c
@@ -109,10 +109,21 @@
#include "storage/procarray.h"
#include "storage/sinval.h"
#include "utils/builtins.h"
+#include "utils/inval.h"
#include "utils/memutils.h"
#include "utils/rel.h"
#include "utils/relfilenumbermap.h"
+/*
+ * Each transaction is limited to a maximum of 8MB of inval messages distributed
+ * from other transaction. Once the number of distributed inval messages reach
+ * this threshold, the transaction is marked as RBTXN_INVAL_OVERFLOWED,
+ * invalidating all caches instead as we have lost some inval messages and hence
+ * don't know what needs to be invalidated.
+ */
+#define MAX_DISTR_INVAL_MSG_PER_TXN \
+ ((8 * 1024 * 1024) / sizeof(SharedInvalidationMessage))
+
/* entry for a hash table we use to map from xid to our transaction state */
typedef struct ReorderBufferTXNByIdEnt
{
@@ -472,6 +483,12 @@ ReorderBufferFreeTXN(ReorderBuffer *rb, ReorderBufferTXN *txn)
txn->invalidations = NULL;
}
+ if (txn->invalidations_distributed)
+ {
+ pfree(txn->invalidations_distributed);
+ txn->invalidations_distributed = NULL;
+ }
+
/* Reset the toast hash */
ReorderBufferToastReset(rb, txn);
@@ -2661,7 +2678,17 @@ ReorderBufferProcessTXN(ReorderBuffer *rb, ReorderBufferTXN *txn,
AbortCurrentTransaction();
/* make sure there's no cache pollution */
- ReorderBufferExecuteInvalidations(txn->ninvalidations, txn->invalidations);
+ if (rbtxn_inval_overflowed(txn))
+ {
+ Assert(txn->ninvalidations == 0 && txn->ninvalidations_distributed == 0);
+ InvalidateSystemCaches();
+ }
+ else
+ {
+ ReorderBufferExecuteInvalidations(txn->ninvalidations, txn->invalidations);
+ ReorderBufferExecuteInvalidations(txn->ninvalidations_distributed,
+ txn->invalidations_distributed);
+ }
if (using_subtxn)
RollbackAndReleaseCurrentSubTransaction();
@@ -2710,8 +2737,17 @@ ReorderBufferProcessTXN(ReorderBuffer *rb, ReorderBufferTXN *txn,
AbortCurrentTransaction();
/* make sure there's no cache pollution */
- ReorderBufferExecuteInvalidations(txn->ninvalidations,
- txn->invalidations);
+ if (rbtxn_inval_overflowed(txn))
+ {
+ Assert(txn->ninvalidations == 0 && txn->ninvalidations_distributed == 0);
+ InvalidateSystemCaches();
+ }
+ else
+ {
+ ReorderBufferExecuteInvalidations(txn->ninvalidations, txn->invalidations);
+ ReorderBufferExecuteInvalidations(txn->ninvalidations_distributed,
+ txn->invalidations_distributed);
+ }
if (using_subtxn)
RollbackAndReleaseCurrentSubTransaction();
@@ -3421,6 +3457,57 @@ ReorderBufferAddNewTupleCids(ReorderBuffer *rb, TransactionId xid,
txn->ntuplecids++;
}
+/*
+ * Add a new invalidation message to the reorder buffer queue.
+ */
+static void
+ReorderBufferQueueInvalidations(ReorderBuffer *rb, TransactionId xid,
+ XLogRecPtr lsn, Size nmsgs,
+ SharedInvalidationMessage *msgs)
+{
+ ReorderBufferChange *change;
+
+ change = ReorderBufferAllocChange(rb);
+ change->action = REORDER_BUFFER_CHANGE_INVALIDATION;
+ change->data.inval.ninvalidations = nmsgs;
+ change->data.inval.invalidations = (SharedInvalidationMessage *)
+ palloc(sizeof(SharedInvalidationMessage) * nmsgs);
+ memcpy(change->data.inval.invalidations, msgs,
+ sizeof(SharedInvalidationMessage) * nmsgs);
+
+ ReorderBufferQueueChange(rb, xid, lsn, change, false);
+}
+
+/*
+ * A helper function for ReorderBufferAddInvalidations() and
+ * ReorderBufferAddDistributedInvalidations() to accumulate the invalidation
+ * messages to the **invals_out.
+ */
+static void
+ReorderBufferAccumulateInvalidations(SharedInvalidationMessage **invals_out,
+ uint32 *ninvals_out,
+ SharedInvalidationMessage *msgs_new,
+ Size nmsgs_new)
+{
+ if (*ninvals_out == 0)
+ {
+ *ninvals_out = nmsgs_new;
+ *invals_out = (SharedInvalidationMessage *)
+ palloc(sizeof(SharedInvalidationMessage) * nmsgs_new);
+ memcpy(*invals_out, msgs_new, sizeof(SharedInvalidationMessage) * nmsgs_new);
+ }
+ else
+ {
+ /* Enlarge the array of inval messages */
+ *invals_out = (SharedInvalidationMessage *)
+ repalloc(*invals_out, sizeof(SharedInvalidationMessage) *
+ (*ninvals_out + nmsgs_new));
+ memcpy(*invals_out + *ninvals_out, msgs_new,
+ nmsgs_new * sizeof(SharedInvalidationMessage));
+ *ninvals_out += nmsgs_new;
+ }
+}
+
/*
* Accumulate the invalidations for executing them later.
*
@@ -3441,7 +3528,6 @@ ReorderBufferAddInvalidations(ReorderBuffer *rb, TransactionId xid,
{
ReorderBufferTXN *txn;
MemoryContext oldcontext;
- ReorderBufferChange *change;
txn = ReorderBufferTXNByXid(rb, xid, true, NULL, lsn, true);
@@ -3456,35 +3542,90 @@ ReorderBufferAddInvalidations(ReorderBuffer *rb, TransactionId xid,
Assert(nmsgs > 0);
- /* Accumulate invalidations. */
- if (txn->ninvalidations == 0)
- {
- txn->ninvalidations = nmsgs;
- txn->invalidations = (SharedInvalidationMessage *)
- palloc(sizeof(SharedInvalidationMessage) * nmsgs);
- memcpy(txn->invalidations, msgs,
- sizeof(SharedInvalidationMessage) * nmsgs);
- }
- else
+ /*
+ * Accumulate the invalidation messages unless it's overflowed.
+ *
+ * Unlike 'invalidations_distributed', which handle messages from multiple
+ * transactions, this mechanism accumulates invalidations from a single
+ * transaction. Therefore, the likelihood of the message buffer
+ * overflowing is low, and we do not impose a strict size limit.
+ */
+ if (!rbtxn_inval_overflowed(txn))
+ ReorderBufferAccumulateInvalidations(&txn->invalidations,
+ &txn->ninvalidations,
+ msgs, nmsgs);
+
+ ReorderBufferQueueInvalidations(rb, xid, lsn, nmsgs, msgs);
+
+ MemoryContextSwitchTo(oldcontext);
+}
+
+/*
+ * Accumulate the invalidations distributed by other committed transactions
+ * for executing them later.
+ *
+ * This function is similar to ReorderBufferAddInvalidations() but stores
+ * the given inval messages to the txn->invalidations_distributed with the
+ * overflow check.
+ *
+ * This needs to be called by committed transactions to distribute their
+ * inval messages to in-progress transactions.
+ */
+void
+ReorderBufferAddDistributedInvalidations(ReorderBuffer *rb, TransactionId xid,
+ XLogRecPtr lsn, Size nmsgs,
+ SharedInvalidationMessage *msgs)
+{
+ ReorderBufferTXN *txn;
+ MemoryContext oldcontext;
+
+ txn = ReorderBufferTXNByXid(rb, xid, true, NULL, lsn, true);
+
+ oldcontext = MemoryContextSwitchTo(rb->context);
+
+ /*
+ * Collect all the invalidations under the top transaction, if available,
+ * so that we can execute them all together. See comments
+ * ReorderBufferAddInvalidations.
+ */
+ txn = rbtxn_get_toptxn(txn);
+
+ Assert(nmsgs > 0);
+
+ /*
+ * Check the transaction has enough space for storing distributed
+ * invalidation messages.
+ */
+ if (txn->ninvalidations_distributed + nmsgs >= MAX_DISTR_INVAL_MSG_PER_TXN)
{
- txn->invalidations = (SharedInvalidationMessage *)
- repalloc(txn->invalidations, sizeof(SharedInvalidationMessage) *
- (txn->ninvalidations + nmsgs));
+ /*
+ * Mark the invalidation message as overflowed and free up the
+ * messages accumulated so far.
+ */
+ txn->txn_flags |= RBTXN_INVAL_OVERFLOWED;
- memcpy(txn->invalidations + txn->ninvalidations, msgs,
- nmsgs * sizeof(SharedInvalidationMessage));
- txn->ninvalidations += nmsgs;
+ if (txn->invalidations_distributed)
+ {
+ pfree(txn->invalidations_distributed);
+ txn->invalidations_distributed = NULL;
+ txn->ninvalidations_distributed = 0;
+ }
+
+ if (txn->invalidations)
+ {
+ pfree(txn->invalidations);
+ txn->invalidations = NULL;
+ txn->ninvalidations = 0;
+ }
}
- change = ReorderBufferAllocChange(rb);
- change->action = REORDER_BUFFER_CHANGE_INVALIDATION;
- change->data.inval.ninvalidations = nmsgs;
- change->data.inval.invalidations = (SharedInvalidationMessage *)
- palloc(sizeof(SharedInvalidationMessage) * nmsgs);
- memcpy(change->data.inval.invalidations, msgs,
- sizeof(SharedInvalidationMessage) * nmsgs);
+ if (!rbtxn_inval_overflowed(txn))
+ ReorderBufferAccumulateInvalidations(&txn->invalidations_distributed,
+ &txn->ninvalidations_distributed,
+ msgs, nmsgs);
- ReorderBufferQueueChange(rb, xid, lsn, change, false);
+ /* Queue the invalidation messages into the transaction */
+ ReorderBufferQueueInvalidations(rb, xid, lsn, nmsgs, msgs);
MemoryContextSwitchTo(oldcontext);
}
diff --git a/src/backend/replication/logical/snapbuild.c b/src/backend/replication/logical/snapbuild.c
index 0d7bddbe4ed..adf18c397db 100644
--- a/src/backend/replication/logical/snapbuild.c
+++ b/src/backend/replication/logical/snapbuild.c
@@ -794,6 +794,13 @@ SnapBuildDistributeSnapshotAndInval(SnapBuild *builder, XLogRecPtr lsn, Transact
* contents built by the current transaction even after its decoding,
* which should have been invalidated due to concurrent catalog
* changing transaction.
+ *
+ * Distribute only the invalidation messages generated by the current
+ * committed transaction. Invalidation messages received from other
+ * transactions would have already been propagated to the relevant
+ * in-progress transactions. This transaction would have processed
+ * those invalidations, ensuring that subsequent transactions observe
+ * a consistent cache state.
*/
if (txn->xid != xid)
{
@@ -807,8 +814,9 @@ SnapBuildDistributeSnapshotAndInval(SnapBuild *builder, XLogRecPtr lsn, Transact
{
Assert(msgs != NULL);
- ReorderBufferAddInvalidations(builder->reorder, txn->xid, lsn,
- ninvalidations, msgs);
+ ReorderBufferAddDistributedInvalidations(builder->reorder,
+ txn->xid, lsn,
+ ninvalidations, msgs);
}
}
}
diff --git a/src/include/replication/reorderbuffer.h b/src/include/replication/reorderbuffer.h
index 24e88c409ba..d4fbe5ff709 100644
--- a/src/include/replication/reorderbuffer.h
+++ b/src/include/replication/reorderbuffer.h
@@ -176,6 +176,7 @@ typedef struct ReorderBufferChange
#define RBTXN_SENT_PREPARE 0x0200
#define RBTXN_IS_COMMITTED 0x0400
#define RBTXN_IS_ABORTED 0x0800
+#define RBTXN_INVAL_OVERFLOWED 0x1000
#define RBTXN_PREPARE_STATUS_MASK (RBTXN_IS_PREPARED | RBTXN_SKIPPED_PREPARE | RBTXN_SENT_PREPARE)
@@ -265,6 +266,12 @@ typedef struct ReorderBufferChange
((txn)->txn_flags & RBTXN_SKIPPED_PREPARE) != 0 \
)
+/* Is the array of inval message overflowed? */
+#define rbtxn_inval_overflowed(txn) \
+( \
+ ((txn)->txn_flags & RBTXN_INVAL_OVERFLOWED) != 0 \
+)
+
/* Is this a top-level transaction? */
#define rbtxn_is_toptxn(txn) \
( \
@@ -422,6 +429,13 @@ typedef struct ReorderBufferTXN
uint32 ninvalidations;
SharedInvalidationMessage *invalidations;
+ /*
+ * Stores cache invalidation messages distributed by other transactions.
+ */
+ uint32 ninvalidations_distributed;
+ SharedInvalidationMessage *invalidations_distributed;
+
+
/* ---
* Position in one of two lists:
* * list of subtransactions if we are *known* to be subxact
@@ -738,6 +752,9 @@ extern void ReorderBufferAddNewTupleCids(ReorderBuffer *rb, TransactionId xid,
CommandId cmin, CommandId cmax, CommandId combocid);
extern void ReorderBufferAddInvalidations(ReorderBuffer *rb, TransactionId xid, XLogRecPtr lsn,
Size nmsgs, SharedInvalidationMessage *msgs);
+extern void ReorderBufferAddDistributedInvalidations(ReorderBuffer *rb, TransactionId xid,
+ XLogRecPtr lsn, Size nmsgs,
+ SharedInvalidationMessage *msgs);
extern void ReorderBufferImmediateInvalidation(ReorderBuffer *rb, uint32 ninvalidations,
SharedInvalidationMessage *invalidations);
extern void ReorderBufferProcessXid(ReorderBuffer *rb, TransactionId xid, XLogRecPtr lsn);
--
2.43.0
^ permalink raw reply [nested|flat] 94+ messages in thread
* RE: Logical replication 'invalid memory alloc request size 1585837200' after upgrading to 17.5
@ 2025-06-04 09:20 Hayato Kuroda (Fujitsu) <kuroda.hayato@fujitsu.com>
parent: Masahiko Sawada <sawada.mshk@gmail.com>
1 sibling, 1 reply; 94+ messages in thread
From: Hayato Kuroda (Fujitsu) @ 2025-06-04 09:20 UTC (permalink / raw)
To: Masahiko Sawada <sawada.mshk@gmail.com>; +Cc: Amit Kapila <amit.kapila16@gmail.com>; Duncan Sands <duncan.sands@deepbluecap.com>; pgsql-bugs@lists.postgresql.org <pgsql-bugs@lists.postgresql.org>; vignesh C <vignesh21@gmail.com>
Dear Sawada-san,
> Can we have a reasonable test case that covers the inval message overflow
> cases?
I have been considering how we add tests, but it needs lots of invalidation
messages and consumes resources so much. Instead of that, how do you feel
to use injection_points? If it is enabled, the threshold for overflow is much
smaller than usual. Attached patch implemented the idea.
Best regards,
Hayato Kuroda
FUJITSU LIMITED
From b5df911769d7c56c95ece58da22267677b23d66a Mon Sep 17 00:00:00 2001
From: Hayato Kuroda <kuroda.hayato@fujitsu.com>
Date: Wed, 4 Jun 2025 15:42:09 +0900
Subject: [PATCH] add regression test
---
contrib/test_decoding/Makefile | 4 +
contrib/test_decoding/meson.build | 4 +
contrib/test_decoding/t/002_inval.pl | 88 +++++++++++++++++++
.../replication/logical/reorderbuffer.c | 15 +++-
4 files changed, 110 insertions(+), 1 deletion(-)
create mode 100644 contrib/test_decoding/t/002_inval.pl
diff --git a/contrib/test_decoding/Makefile b/contrib/test_decoding/Makefile
index 02e961f4d31..a93a3fa78cb 100644
--- a/contrib/test_decoding/Makefile
+++ b/contrib/test_decoding/Makefile
@@ -20,6 +20,10 @@ NO_INSTALLCHECK = 1
TAP_TESTS = 1
+EXTRA_INSTALL=src/test/modules/injection_points
+
+export enable_injection_points
+
ifdef USE_PGXS
PG_CONFIG = pg_config
PGXS := $(shell $(PG_CONFIG) --pgxs)
diff --git a/contrib/test_decoding/meson.build b/contrib/test_decoding/meson.build
index 25f6b8a9082..a1f2498ff1c 100644
--- a/contrib/test_decoding/meson.build
+++ b/contrib/test_decoding/meson.build
@@ -72,8 +72,12 @@ tests += {
'runningcheck': false,
},
'tap': {
+ 'env': {
+ 'enable_injection_points': get_option('injection_points') ? 'yes' : 'no',
+ },
'tests': [
't/001_repl_stats.pl',
+ 't/002_inval.pl',
],
},
}
diff --git a/contrib/test_decoding/t/002_inval.pl b/contrib/test_decoding/t/002_inval.pl
new file mode 100644
index 00000000000..0558b917b7a
--- /dev/null
+++ b/contrib/test_decoding/t/002_inval.pl
@@ -0,0 +1,88 @@
+# Copyright (c) 2025, PostgreSQL Global Development Group
+
+# Tests for various bugs found over time
+use strict;
+use warnings FATAL => 'all';
+use PostgreSQL::Test::Cluster;
+use PostgreSQL::Test::Utils;
+use Test::More;
+
+if ($ENV{enable_injection_points} ne 'yes')
+{
+ plan skip_all => 'Injection points not supported by this build';
+}
+
+my $node = PostgreSQL::Test::Cluster->new('publisher');
+$node->init(allows_streaming => 'logical');
+$node->start;
+
+# Check if the extension injection_points is available, as it may be
+# possible that this script is run with installcheck, where the module
+# would not be installed by default.
+if (!$node->check_extension('injection_points'))
+{
+ plan skip_all => 'Extension injection_points not installed';
+}
+
+# Define database objects
+$node->safe_psql("postgres", "CREATE TABLE tab (id int);");
+$node->safe_psql("postgres", "SELECT pg_create_logical_replication_slot('test', 'test_decoding');");
+
+# Bump the query timeout to avoid false negatives on slow test systems.
+my $psql_timeout_secs = 4 * $PostgreSQL::Test::Utils::timeout_default;
+
+my @sessions = ();
+my $num_sessions = 4;
+
+# Start background sessions and push to the array once
+for my $i (1..$num_sessions)
+{
+ push(@sessions, $node->background_psql('postgres',
+ timeout => $psql_timeout_secs));
+}
+
+# Start transactions on sessions. Each transactions have invalidation messages
+for my $i (0.. $#sessions)
+{
+ $sessions[$i]->query_safe(
+ qq[
+ BEGIN;
+ INSERT INTO tab VALUES (1);
+ CREATE TABLE tmp_$i (id int);
+ DROP TABLE tmp_$i;
+ ]);
+}
+
+# Commit all transactions and exit
+foreach my $session (@sessions)
+{
+ $session->query_safe(
+ qq[
+ COMMIT;
+ ]);
+ $session->quit;
+}
+
+# Confirms all transactions were committed
+is($node->safe_psql('postgres', "SELECT count(*) FROM tab;"), $num_sessions);
+
+# Decodes changes and confirms whether all caches were invalidated due to the
+# cache overflow.
+$node->safe_psql('postgres', 'CREATE EXTENSION injection_points');
+$node->safe_psql('postgres',
+ "SELECT injection_points_attach('inval-message-overflow', 'error');");
+
+# Increase the log level and set offset because the evidence is output as DEBUG1.
+$node->append_conf('postgresql.conf', 'log_min_messages = debug1');
+$node->reload();
+my $log_location = -s $node->logfile;
+
+# Decodes changes from parallel transactions
+$node->safe_psql(
+ 'postgres', "SELECT data FROM pg_logical_slot_get_binary_changes('test', NULL, NULL);"
+);
+
+# Confirms the debug log has been output
+$node->wait_for_log(qr/RBTXN_INVAL_OVERFLOWED flag was set to the transaction/, $log_location);
+
+done_testing();
diff --git a/src/backend/replication/logical/reorderbuffer.c b/src/backend/replication/logical/reorderbuffer.c
index e05259bf693..e189e5b9e6f 100644
--- a/src/backend/replication/logical/reorderbuffer.c
+++ b/src/backend/replication/logical/reorderbuffer.c
@@ -109,6 +109,7 @@
#include "storage/procarray.h"
#include "storage/sinval.h"
#include "utils/builtins.h"
+#include "utils/injection_point.h"
#include "utils/inval.h"
#include "utils/memutils.h"
#include "utils/rel.h"
@@ -3578,6 +3579,7 @@ ReorderBufferAddDistributedInvalidations(ReorderBuffer *rb, TransactionId xid,
{
ReorderBufferTXN *txn;
MemoryContext oldcontext;
+ Size maximum;
txn = ReorderBufferTXNByXid(rb, xid, true, NULL, lsn, true);
@@ -3592,11 +3594,20 @@ ReorderBufferAddDistributedInvalidations(ReorderBuffer *rb, TransactionId xid,
Assert(nmsgs > 0);
+#ifdef USE_INJECTION_POINTS
+ if (IS_INJECTION_POINT_ATTACHED("inval-message-overflow"))
+ maximum = 100;
+ else
+ maximum = MAX_DISTR_INVAL_MSG_PER_TXN;
+#else
+ maximum = MAX_DISTR_INVAL_MSG_PER_TXN;
+#endif
+
/*
* Check the transaction has enough space for storing distributed
* invalidation messages.
*/
- if (txn->ninvalidations_distributed + nmsgs >= MAX_DISTR_INVAL_MSG_PER_TXN)
+ if (txn->ninvalidations_distributed + nmsgs >= maximum)
{
/*
* Mark the invalidation message as overflowed and free up the
@@ -3604,6 +3615,8 @@ ReorderBufferAddDistributedInvalidations(ReorderBuffer *rb, TransactionId xid,
*/
txn->txn_flags |= RBTXN_INVAL_OVERFLOWED;
+ elog(DEBUG1, "RBTXN_INVAL_OVERFLOWED flag was set to the transaction");
+
if (txn->invalidations_distributed)
{
pfree(txn->invalidations_distributed);
--
2.47.1
Attachments:
[text/plain] 0001-add-regression-test.txt (5.9K, ../../OSCPR01MB149664746A441DE1F9F56203DF56CA@OSCPR01MB14966.jpnprd01.prod.outlook.com/2-0001-add-regression-test.txt)
download | inline diff:
From b5df911769d7c56c95ece58da22267677b23d66a Mon Sep 17 00:00:00 2001
From: Hayato Kuroda <kuroda.hayato@fujitsu.com>
Date: Wed, 4 Jun 2025 15:42:09 +0900
Subject: [PATCH] add regression test
---
contrib/test_decoding/Makefile | 4 +
contrib/test_decoding/meson.build | 4 +
contrib/test_decoding/t/002_inval.pl | 88 +++++++++++++++++++
.../replication/logical/reorderbuffer.c | 15 +++-
4 files changed, 110 insertions(+), 1 deletion(-)
create mode 100644 contrib/test_decoding/t/002_inval.pl
diff --git a/contrib/test_decoding/Makefile b/contrib/test_decoding/Makefile
index 02e961f4d31..a93a3fa78cb 100644
--- a/contrib/test_decoding/Makefile
+++ b/contrib/test_decoding/Makefile
@@ -20,6 +20,10 @@ NO_INSTALLCHECK = 1
TAP_TESTS = 1
+EXTRA_INSTALL=src/test/modules/injection_points
+
+export enable_injection_points
+
ifdef USE_PGXS
PG_CONFIG = pg_config
PGXS := $(shell $(PG_CONFIG) --pgxs)
diff --git a/contrib/test_decoding/meson.build b/contrib/test_decoding/meson.build
index 25f6b8a9082..a1f2498ff1c 100644
--- a/contrib/test_decoding/meson.build
+++ b/contrib/test_decoding/meson.build
@@ -72,8 +72,12 @@ tests += {
'runningcheck': false,
},
'tap': {
+ 'env': {
+ 'enable_injection_points': get_option('injection_points') ? 'yes' : 'no',
+ },
'tests': [
't/001_repl_stats.pl',
+ 't/002_inval.pl',
],
},
}
diff --git a/contrib/test_decoding/t/002_inval.pl b/contrib/test_decoding/t/002_inval.pl
new file mode 100644
index 00000000000..0558b917b7a
--- /dev/null
+++ b/contrib/test_decoding/t/002_inval.pl
@@ -0,0 +1,88 @@
+# Copyright (c) 2025, PostgreSQL Global Development Group
+
+# Tests for various bugs found over time
+use strict;
+use warnings FATAL => 'all';
+use PostgreSQL::Test::Cluster;
+use PostgreSQL::Test::Utils;
+use Test::More;
+
+if ($ENV{enable_injection_points} ne 'yes')
+{
+ plan skip_all => 'Injection points not supported by this build';
+}
+
+my $node = PostgreSQL::Test::Cluster->new('publisher');
+$node->init(allows_streaming => 'logical');
+$node->start;
+
+# Check if the extension injection_points is available, as it may be
+# possible that this script is run with installcheck, where the module
+# would not be installed by default.
+if (!$node->check_extension('injection_points'))
+{
+ plan skip_all => 'Extension injection_points not installed';
+}
+
+# Define database objects
+$node->safe_psql("postgres", "CREATE TABLE tab (id int);");
+$node->safe_psql("postgres", "SELECT pg_create_logical_replication_slot('test', 'test_decoding');");
+
+# Bump the query timeout to avoid false negatives on slow test systems.
+my $psql_timeout_secs = 4 * $PostgreSQL::Test::Utils::timeout_default;
+
+my @sessions = ();
+my $num_sessions = 4;
+
+# Start background sessions and push to the array once
+for my $i (1..$num_sessions)
+{
+ push(@sessions, $node->background_psql('postgres',
+ timeout => $psql_timeout_secs));
+}
+
+# Start transactions on sessions. Each transactions have invalidation messages
+for my $i (0.. $#sessions)
+{
+ $sessions[$i]->query_safe(
+ qq[
+ BEGIN;
+ INSERT INTO tab VALUES (1);
+ CREATE TABLE tmp_$i (id int);
+ DROP TABLE tmp_$i;
+ ]);
+}
+
+# Commit all transactions and exit
+foreach my $session (@sessions)
+{
+ $session->query_safe(
+ qq[
+ COMMIT;
+ ]);
+ $session->quit;
+}
+
+# Confirms all transactions were committed
+is($node->safe_psql('postgres', "SELECT count(*) FROM tab;"), $num_sessions);
+
+# Decodes changes and confirms whether all caches were invalidated due to the
+# cache overflow.
+$node->safe_psql('postgres', 'CREATE EXTENSION injection_points');
+$node->safe_psql('postgres',
+ "SELECT injection_points_attach('inval-message-overflow', 'error');");
+
+# Increase the log level and set offset because the evidence is output as DEBUG1.
+$node->append_conf('postgresql.conf', 'log_min_messages = debug1');
+$node->reload();
+my $log_location = -s $node->logfile;
+
+# Decodes changes from parallel transactions
+$node->safe_psql(
+ 'postgres', "SELECT data FROM pg_logical_slot_get_binary_changes('test', NULL, NULL);"
+);
+
+# Confirms the debug log has been output
+$node->wait_for_log(qr/RBTXN_INVAL_OVERFLOWED flag was set to the transaction/, $log_location);
+
+done_testing();
diff --git a/src/backend/replication/logical/reorderbuffer.c b/src/backend/replication/logical/reorderbuffer.c
index e05259bf693..e189e5b9e6f 100644
--- a/src/backend/replication/logical/reorderbuffer.c
+++ b/src/backend/replication/logical/reorderbuffer.c
@@ -109,6 +109,7 @@
#include "storage/procarray.h"
#include "storage/sinval.h"
#include "utils/builtins.h"
+#include "utils/injection_point.h"
#include "utils/inval.h"
#include "utils/memutils.h"
#include "utils/rel.h"
@@ -3578,6 +3579,7 @@ ReorderBufferAddDistributedInvalidations(ReorderBuffer *rb, TransactionId xid,
{
ReorderBufferTXN *txn;
MemoryContext oldcontext;
+ Size maximum;
txn = ReorderBufferTXNByXid(rb, xid, true, NULL, lsn, true);
@@ -3592,11 +3594,20 @@ ReorderBufferAddDistributedInvalidations(ReorderBuffer *rb, TransactionId xid,
Assert(nmsgs > 0);
+#ifdef USE_INJECTION_POINTS
+ if (IS_INJECTION_POINT_ATTACHED("inval-message-overflow"))
+ maximum = 100;
+ else
+ maximum = MAX_DISTR_INVAL_MSG_PER_TXN;
+#else
+ maximum = MAX_DISTR_INVAL_MSG_PER_TXN;
+#endif
+
/*
* Check the transaction has enough space for storing distributed
* invalidation messages.
*/
- if (txn->ninvalidations_distributed + nmsgs >= MAX_DISTR_INVAL_MSG_PER_TXN)
+ if (txn->ninvalidations_distributed + nmsgs >= maximum)
{
/*
* Mark the invalidation message as overflowed and free up the
@@ -3604,6 +3615,8 @@ ReorderBufferAddDistributedInvalidations(ReorderBuffer *rb, TransactionId xid,
*/
txn->txn_flags |= RBTXN_INVAL_OVERFLOWED;
+ elog(DEBUG1, "RBTXN_INVAL_OVERFLOWED flag was set to the transaction");
+
if (txn->invalidations_distributed)
{
pfree(txn->invalidations_distributed);
--
2.47.1
^ permalink raw reply [nested|flat] 94+ messages in thread
* Re: Logical replication 'invalid memory alloc request size 1585837200' after upgrading to 17.5
@ 2025-06-04 21:48 Masahiko Sawada <sawada.mshk@gmail.com>
parent: vignesh C <vignesh21@gmail.com>
0 siblings, 2 replies; 94+ messages in thread
From: Masahiko Sawada @ 2025-06-04 21:48 UTC (permalink / raw)
To: vignesh C <vignesh21@gmail.com>; +Cc: Amit Kapila <amit.kapila16@gmail.com>; Hayato Kuroda (Fujitsu) <kuroda.hayato@fujitsu.com>; Duncan Sands <duncan.sands@deepbluecap.com>; pgsql-bugs@lists.postgresql.org <pgsql-bugs@lists.postgresql.org>
On Tue, Jun 3, 2025 at 11:48 PM vignesh C <vignesh21@gmail.com> wrote:
>
> On Wed, 4 Jun 2025 at 01:14, Masahiko Sawada <sawada.mshk@gmail.com> wrote:
> >
> > On Tue, Jun 3, 2025 at 12:07 AM vignesh C <vignesh21@gmail.com> wrote:
> > >
> > > On Mon, 2 Jun 2025 at 22:49, Masahiko Sawada <sawada.mshk@gmail.com> wrote:
> > > >
> > > > Thank you for updating the patch. Here are some review comments:
> > > >
> > > > + req_mem_size = sizeof(SharedInvalidationMessage) *
> > > > (txn->ninvalidations_distr + nmsgs);
> > > > +
> > > > + /*
> > > > + * If the number of invalidation messages is larger than 8MB, it's more
> > > > + * efficient to invalidate the entire cache rather than processing each
> > > > + * message individually.
> > > > + */
> > > > + if (req_mem_size > (8 * 1024 * 1024) || rbtxn_inval_all_cache(txn))
> > > >
> > > > It's better to define the maximum number of distributed inval messages
> > > > per transaction as a macro instead of calculating the memory size
> > > > every time.
> > >
> > > Modified
> > >
> > > > ---
> > > > +static void
> > > > +ReorderBufferAddInvalidationsCommon(ReorderBuffer *rb, TransactionId xid,
> > > > + XLogRecPtr lsn, Size nmsgs,
> > > > + SharedInvalidationMessage *msgs,
> > > > + ReorderBufferTXN *txn,
> > > > + bool for_inval)
> > > >
> > > > This function is quite confusing to me. For instance,
> > > > ReorderBufferAddDistributedInvalidations() needs to call this function
> > > > with for_inval=false in spite of adding inval messages actually. Also,
> > > > the following condition seems not intuisive but there is no comment:
> > > >
> > > > if (!for_inval || (for_inval && !rbtxn_inval_all_cache(txn)))
> > > >
> > > > Instead of having ReorderBufferAddInvalidationsCommon(), I think we
> > > > can have a function say ReorderBufferQueueInvalidations() where we
> > > > enqueue the given inval messages as a
> > > > REORDER_BUFFER_CHANGE_INVALIDATION change.
> > > > ReorderBufferAddInvalidations() adds inval messages to
> > > > txn->invalidations and calls that function, while
> > > > ReorderBufferQueueInvalidations() adds inval messages to
> > > > txn->distributed_ivnalidations and calls that function if the array is
> > > > not full.
> > >
> > > Modified
> > >
> > > > BTW if we need to invalidate all accumulated caches at the end of
> > > > transaction replay anyway, we don't need to add inval messages to
> > > > txn->invalidations once txn->distributed_invalidations gets full?
> > >
> > > yes, no need to add invalidation messages to txn->invalidation once
> > > RBTXN_INVAL_ALL_CACHE is set. This is handled now.
> > >
> > > The attached v9 version patch has the changes for the same.
> >
> > Thank you for updating the patch. Here are review comments on v9 patch:
> >
> > +/*
> > + * Maximum number of distributed invalidation messages per transaction.
> > + * Each message is ~16 bytes, this allows up to 8 MB of invalidation
> > + * message data.
> > + */
> > +#define MAX_DISTR_INVAL_MSG_PER_TXN 524288
> >
> > The size of SharedInvalidationMessage could change in the future so we
> > should calculate it at compile time.
>
> Modified
>
> > ---
> > + /*
> > + * If the complete cache will be invalidated, we don't need to accumulate
> > + * the invalidations.
> > + */
> > + if (!rbtxn_inval_all_cache(txn))
> > + ReorderBufferAccumulateInvalidations(&txn->ninvalidations,
> > + &txn->invalidations, nmsgs, msgs);
> >
> > We need to explain why we don't check the number of invalidation
> > messages for txn->invalidations and mark it as inval-all-cache, unlike
> > ReorderBufferAddDistributedInvalidations().
>
> Added comments
>
> > ---
> > + /*
> > + * If the number of invalidation messages is high, performing a full cache
> > + * invalidation is more efficient than handling each message separately.
> > + */
> > + if (((nmsgs + txn->ninvalidations_distributed) >
> > MAX_DISTR_INVAL_MSG_PER_TXN) ||
> > + rbtxn_inval_all_cache(txn))
> > {
> > - txn->invalidations = (SharedInvalidationMessage *)
> > - repalloc(txn->invalidations, sizeof(SharedInvalidationMessage) *
> > - (txn->ninvalidations + nmsgs));
> > + txn->txn_flags |= RBTXN_INVAL_ALL_CACHE;
> >
> > I think we don't need to mark the transaction as RBTXN_INVAL_ALL_CACHE
> > again. I'd rewrite the logic as follows:
> >
> > if (txn->ninvalidations_distributed + nmsgs >= MAX_DISTR_INVAL_MSG_PER_TXN)
> > {
> > /* mark the txn as inval-all-cache */
> > ....
> > /* free the accumulated inval msgs */
> > ....
> > }
> >
> > if (!rbtxn_inval_all_cache(txn))
> > ReorderBufferAccumulateInvalidations(...);
>
> Modified
>
> > ---
> > - ReorderBufferAddInvalidations(builder->reorder, txn->xid, lsn,
> > - ninvalidations, msgs);
> > + ReorderBufferAddDistributedInvalidations(builder->reorder,
> > + txn->xid, lsn,
> > + ninvalidations, msgs);
> >
> > I think we need some comments here to explain why we need to
> > distribute only inval messages coming from the current transaction.
>
> Added comments
>
> > ---
> > +/* Should the complete cache be invalidated? */
> > +#define rbtxn_inval_all_cache(txn) \
> > +( \
> > + ((txn)->txn_flags & RBTXN_INVAL_ALL_CACHE) != 0 \
> > +)
> >
> > I find that if we rename the flag to something like
> > RBTXN_INVAL_OVERFLOWED, it would explain the state of the transaction
> > clearer.
>
> Modified
>
> > Can we have a reasonable test case that covers the inval message overflow cases?
> One of us will work on this and post a separate patch
>
> > I've attached a patch for some changes and adding more comments (note
> > that it still has XXX comments). Please include these changes that you
> > agreed with in the next version patch.
>
> Thanks for the comments, I merged it.
>
> The attached v10 version patch has the changes for the same.
Thank you for updating the patch. I have some comments and questions:
In ReorderBufferAbort():
/*
* We might have decoded changes for this transaction that could load
* the cache as per the current transaction's view (consider DDL's
* happened in this transaction). We don't want the decoding of future
* transactions to use those cache entries so execute invalidations.
*/
if (txn->ninvalidations > 0)
ReorderBufferImmediateInvalidation(rb, txn->ninvalidations,
txn->invalidations);
I think that if the txn->invalidations_distributed is overflowed, we
would miss executing the txn->invalidations here. Probably the same is
true for ReorderBufferForget() and ReorderBufferInvalidate().
---
I'd like to make it clear again which case we need to execute
txn->invalidations as well as txn->invalidations_distributed (like in
ReorderBufferProcessTXN()) and which case we need to execute only
txn->invalidations (like in ReorderBufferForget() and
ReorderBufferAbort()). I think it might be worth putting some comments
about overall strategy somewhere.
---
BTW for back branches, a simple fix without ABI breakage would be to
introduce the RBTXN_INVAL_OVERFLOWED flag to limit the size of
txn->invalidations. That is, we accumulate inval messages both coming
from the current transaction and distributed by other transactions but
once the size reaches the threshold we invalidate all caches. Is it
worth considering for back branches?
Regards,
--
Masahiko Sawada
Amazon Web Services: https://aws.amazon.com
^ permalink raw reply [nested|flat] 94+ messages in thread
* Re: Logical replication 'invalid memory alloc request size 1585837200' after upgrading to 17.5
@ 2025-06-04 22:06 Masahiko Sawada <sawada.mshk@gmail.com>
parent: Hayato Kuroda (Fujitsu) <kuroda.hayato@fujitsu.com>
0 siblings, 1 reply; 94+ messages in thread
From: Masahiko Sawada @ 2025-06-04 22:06 UTC (permalink / raw)
To: Hayato Kuroda (Fujitsu) <kuroda.hayato@fujitsu.com>; +Cc: Amit Kapila <amit.kapila16@gmail.com>; Duncan Sands <duncan.sands@deepbluecap.com>; pgsql-bugs@lists.postgresql.org <pgsql-bugs@lists.postgresql.org>; vignesh C <vignesh21@gmail.com>
On Wed, Jun 4, 2025 at 2:21 AM Hayato Kuroda (Fujitsu)
<kuroda.hayato@fujitsu.com> wrote:
>
> Dear Sawada-san,
>
> > Can we have a reasonable test case that covers the inval message overflow
> > cases?
>
> I have been considering how we add tests, but it needs lots of invalidation
> messages and consumes resources so much. Instead of that, how do you feel
> to use injection_points? If it is enabled, the threshold for overflow is much
> smaller than usual. Attached patch implemented the idea.
Alternative idea is to lower the constant value when using an
assertion build. That way, we don't need to rely on injection points
being enabled.
Regards,
--
Masahiko Sawada
Amazon Web Services: https://aws.amazon.com
^ permalink raw reply [nested|flat] 94+ messages in thread
* Re: Logical replication 'invalid memory alloc request size 1585837200' after upgrading to 17.5
@ 2025-06-05 06:20 Amit Kapila <amit.kapila16@gmail.com>
parent: Masahiko Sawada <sawada.mshk@gmail.com>
1 sibling, 2 replies; 94+ messages in thread
From: Amit Kapila @ 2025-06-05 06:20 UTC (permalink / raw)
To: Masahiko Sawada <sawada.mshk@gmail.com>; +Cc: vignesh C <vignesh21@gmail.com>; Hayato Kuroda (Fujitsu) <kuroda.hayato@fujitsu.com>; Duncan Sands <duncan.sands@deepbluecap.com>; pgsql-bugs@lists.postgresql.org <pgsql-bugs@lists.postgresql.org>
On Thu, Jun 5, 2025 at 3:19 AM Masahiko Sawada <sawada.mshk@gmail.com> wrote:
>
> On Tue, Jun 3, 2025 at 11:48 PM vignesh C <vignesh21@gmail.com> wrote:
> >
> > On Wed, 4 Jun 2025 at 01:14, Masahiko Sawada <sawada.mshk@gmail.com> wrote:
>
> Thank you for updating the patch. I have some comments and questions:
>
> In ReorderBufferAbort():
>
> /*
> * We might have decoded changes for this transaction that could load
> * the cache as per the current transaction's view (consider DDL's
> * happened in this transaction). We don't want the decoding of future
> * transactions to use those cache entries so execute invalidations.
> */
> if (txn->ninvalidations > 0)
> ReorderBufferImmediateInvalidation(rb, txn->ninvalidations,
> txn->invalidations);
>
> I think that if the txn->invalidations_distributed is overflowed, we
> would miss executing the txn->invalidations here. Probably the same is
> true for ReorderBufferForget() and ReorderBufferInvalidate().
>
This is because of the following check "if
(!rbtxn_inval_overflowed(txn))" in function
ReorderBufferAddInvalidations(). What is the need of such a check in
this function? We don't need to execute distributed invalidations in
cases like ReorderBufferForget() when we haven't decoded any changes.
> ---
> I'd like to make it clear again which case we need to execute
> txn->invalidations as well as txn->invalidations_distributed (like in
> ReorderBufferProcessTXN()) and which case we need to execute only
> txn->invalidations (like in ReorderBufferForget() and
> ReorderBufferAbort()). I think it might be worth putting some comments
> about overall strategy somewhere.
>
> ---
> BTW for back branches, a simple fix without ABI breakage would be to
> introduce the RBTXN_INVAL_OVERFLOWED flag to limit the size of
> txn->invalidations. That is, we accumulate inval messages both coming
> from the current transaction and distributed by other transactions but
> once the size reaches the threshold we invalidate all caches. Is it
> worth considering for back branches?
>
It should work and is worth considering. The main concern would be
that it will hit sooner than we expect in the field, seeing the recent
reports. So, such a change has the potential to degrade the
performance. I feel that the number of people impacted due to
performance would be more than the number of people impacted due to
such an ABI change (adding the new members at the end of
ReorderBufferTXN). However, if we think we want to go safe w.r.t
extensions that can rely on the sizeof ReorderBufferTXN then your
proposal makes sense.
--
With Regards,
Amit Kapila.
^ permalink raw reply [nested|flat] 94+ messages in thread
* RE: Logical replication 'invalid memory alloc request size 1585837200' after upgrading to 17.5
@ 2025-06-05 06:36 Hayato Kuroda (Fujitsu) <kuroda.hayato@fujitsu.com>
parent: Masahiko Sawada <sawada.mshk@gmail.com>
0 siblings, 1 reply; 94+ messages in thread
From: Hayato Kuroda (Fujitsu) @ 2025-06-05 06:36 UTC (permalink / raw)
To: Masahiko Sawada <sawada.mshk@gmail.com>; +Cc: Amit Kapila <amit.kapila16@gmail.com>; Duncan Sands <duncan.sands@deepbluecap.com>; pgsql-bugs@lists.postgresql.org <pgsql-bugs@lists.postgresql.org>; vignesh C <vignesh21@gmail.com>
Dear Sawada-san,
> Alternative idea is to lower the constant value when using an
> assertion build. That way, we don't need to rely on injection points
> being enabled.
Hmm, possible but I prefer current one. Two concerns:
1.
USE_ASSERT_CHECKING has not been used to change the value yet. The main usage is
to call debug functions in debug build.
2.
If we add tests which is usable only for debug build, it must be run only when it
is enabled. IIUC such test does not exist yet.
Best regards,
Hayato Kuroda
FUJITSU LIMITED
^ permalink raw reply [nested|flat] 94+ messages in thread
* Re: Logical replication 'invalid memory alloc request size 1585837200' after upgrading to 17.5
@ 2025-06-05 10:28 vignesh C <vignesh21@gmail.com>
parent: Masahiko Sawada <sawada.mshk@gmail.com>
1 sibling, 1 reply; 94+ messages in thread
From: vignesh C @ 2025-06-05 10:28 UTC (permalink / raw)
To: Masahiko Sawada <sawada.mshk@gmail.com>; +Cc: Amit Kapila <amit.kapila16@gmail.com>; Hayato Kuroda (Fujitsu) <kuroda.hayato@fujitsu.com>; Duncan Sands <duncan.sands@deepbluecap.com>; pgsql-bugs@lists.postgresql.org <pgsql-bugs@lists.postgresql.org>
On Thu, 5 Jun 2025 at 03:19, Masahiko Sawada <sawada.mshk@gmail.com> wrote:
>
> Thank you for updating the patch. I have some comments and questions:
>
> In ReorderBufferAbort():
>
> /*
> * We might have decoded changes for this transaction that could load
> * the cache as per the current transaction's view (consider DDL's
> * happened in this transaction). We don't want the decoding of future
> * transactions to use those cache entries so execute invalidations.
> */
> if (txn->ninvalidations > 0)
> ReorderBufferImmediateInvalidation(rb, txn->ninvalidations,
> txn->invalidations);
>
> I think that if the txn->invalidations_distributed is overflowed, we
> would miss executing the txn->invalidations here. Probably the same is
> true for ReorderBufferForget() and ReorderBufferInvalidate().
I'm accumulating the invalidations in txn->invalidations irrespective
of RBTXN_INVAL_OVERFLOWED txn.
> ---
> I'd like to make it clear again which case we need to execute
> txn->invalidations as well as txn->invalidations_distributed (like in
> ReorderBufferProcessTXN()) and which case we need to execute only
> txn->invalidations (like in ReorderBufferForget() and
> ReorderBufferAbort()). I think it might be worth putting some comments
> about overall strategy somewhere.
I have added comments for this, feel free to reword it if some
changes are required.
The attached v11 version patch has the changes for the same.
Regards,
Vignesh
Attachments:
[text/x-patch] v11-master-0001-Fix-exponential-memory-allocation-issue-i.patch (13.2K, ../../CALDaNm2E3ks06QTvEuQAHw3CxPhUrg6J6twXAwYO93S3xZPtvQ@mail.gmail.com/2-v11-master-0001-Fix-exponential-memory-allocation-issue-i.patch)
download | inline diff:
From 252e499d972fc6ffe4a3084923ac9fa7fc99f947 Mon Sep 17 00:00:00 2001
From: Vignesh <vignesh21@gmail.com>
Date: Fri, 30 May 2025 12:06:07 +0530
Subject: [PATCH v11-master] Fix exponential memory allocation issue in logical
decoding due to invalidation message distribution
Commit 4909b38af0 added logic to distribute invalidation messages from
one transaction to all concurrent in-progress transactions. However,
since the source of each invalidation message was not tracked, transactions
could redundantly re-distribute already propagated messages. This led to an
exponential increase in invalidation messages and, ultimately, memory
allocation failures.
To fix this, a new field is introduced to track distributed invalidation
messages separately. This ensures that only transactions in the pre-existing
list receive the invalidations, preventing redundant propagation.
The accumulated distributed invalidations are now consumed at the end of
transaction processing in `ReorderBufferProcessTXN()`.
---
.../replication/logical/reorderbuffer.c | 180 +++++++++++++++---
src/backend/replication/logical/snapbuild.c | 12 +-
src/include/replication/reorderbuffer.h | 30 +++
3 files changed, 192 insertions(+), 30 deletions(-)
diff --git a/src/backend/replication/logical/reorderbuffer.c b/src/backend/replication/logical/reorderbuffer.c
index 67655111875..606c94df7db 100644
--- a/src/backend/replication/logical/reorderbuffer.c
+++ b/src/backend/replication/logical/reorderbuffer.c
@@ -109,10 +109,20 @@
#include "storage/procarray.h"
#include "storage/sinval.h"
#include "utils/builtins.h"
+#include "utils/inval.h"
#include "utils/memutils.h"
#include "utils/rel.h"
#include "utils/relfilenumbermap.h"
+/*
+ * Each transaction is limited to a maximum of 8MB of inval messages distributed
+ * from other transaction. Once the number of distributed inval messages reach
+ * this threshold, the transaction is marked as RBTXN_INVAL_OVERFLOWED to
+ * invalidate the complete cache.
+ */
+#define MAX_DISTR_INVAL_MSG_PER_TXN \
+ ((8 * 1024 * 1024) / sizeof(SharedInvalidationMessage))
+
/* entry for a hash table we use to map from xid to our transaction state */
typedef struct ReorderBufferTXNByIdEnt
{
@@ -472,6 +482,12 @@ ReorderBufferFreeTXN(ReorderBuffer *rb, ReorderBufferTXN *txn)
txn->invalidations = NULL;
}
+ if (txn->invalidations_distributed)
+ {
+ pfree(txn->invalidations_distributed);
+ txn->invalidations_distributed = NULL;
+ }
+
/* Reset the toast hash */
ReorderBufferToastReset(rb, txn);
@@ -2661,7 +2677,17 @@ ReorderBufferProcessTXN(ReorderBuffer *rb, ReorderBufferTXN *txn,
AbortCurrentTransaction();
/* make sure there's no cache pollution */
- ReorderBufferExecuteInvalidations(txn->ninvalidations, txn->invalidations);
+ if (rbtxn_inval_overflowed(txn))
+ {
+ Assert(txn->ninvalidations_distributed == 0);
+ InvalidateSystemCaches();
+ }
+ else
+ {
+ ReorderBufferExecuteInvalidations(txn->ninvalidations, txn->invalidations);
+ ReorderBufferExecuteInvalidations(txn->ninvalidations_distributed,
+ txn->invalidations_distributed);
+ }
if (using_subtxn)
RollbackAndReleaseCurrentSubTransaction();
@@ -2710,8 +2736,17 @@ ReorderBufferProcessTXN(ReorderBuffer *rb, ReorderBufferTXN *txn,
AbortCurrentTransaction();
/* make sure there's no cache pollution */
- ReorderBufferExecuteInvalidations(txn->ninvalidations,
- txn->invalidations);
+ if (rbtxn_inval_overflowed(txn))
+ {
+ Assert(txn->ninvalidations_distributed == 0);
+ InvalidateSystemCaches();
+ }
+ else
+ {
+ ReorderBufferExecuteInvalidations(txn->ninvalidations, txn->invalidations);
+ ReorderBufferExecuteInvalidations(txn->ninvalidations_distributed,
+ txn->invalidations_distributed);
+ }
if (using_subtxn)
RollbackAndReleaseCurrentSubTransaction();
@@ -3421,6 +3456,57 @@ ReorderBufferAddNewTupleCids(ReorderBuffer *rb, TransactionId xid,
txn->ntuplecids++;
}
+/*
+ * Add a new invalidation message to the reorder buffer queue.
+ */
+static void
+ReorderBufferQueueInvalidations(ReorderBuffer *rb, TransactionId xid,
+ XLogRecPtr lsn, Size nmsgs,
+ SharedInvalidationMessage *msgs)
+{
+ ReorderBufferChange *change;
+
+ change = ReorderBufferAllocChange(rb);
+ change->action = REORDER_BUFFER_CHANGE_INVALIDATION;
+ change->data.inval.ninvalidations = nmsgs;
+ change->data.inval.invalidations = (SharedInvalidationMessage *)
+ palloc(sizeof(SharedInvalidationMessage) * nmsgs);
+ memcpy(change->data.inval.invalidations, msgs,
+ sizeof(SharedInvalidationMessage) * nmsgs);
+
+ ReorderBufferQueueChange(rb, xid, lsn, change, false);
+}
+
+/*
+ * A helper function for ReorderBufferAddInvalidations() and
+ * ReorderBufferAddDistributedInvalidations() to accumulate the invalidation
+ * messages to the **invals_out.
+ */
+static void
+ReorderBufferAccumulateInvalidations(SharedInvalidationMessage **invals_out,
+ uint32 *ninvals_out,
+ SharedInvalidationMessage *msgs_new,
+ Size nmsgs_new)
+{
+ if (*ninvals_out == 0)
+ {
+ *ninvals_out = nmsgs_new;
+ *invals_out = (SharedInvalidationMessage *)
+ palloc(sizeof(SharedInvalidationMessage) * nmsgs_new);
+ memcpy(*invals_out, msgs_new, sizeof(SharedInvalidationMessage) * nmsgs_new);
+ }
+ else
+ {
+ /* Enlarge the array of inval messages */
+ *invals_out = (SharedInvalidationMessage *)
+ repalloc(*invals_out, sizeof(SharedInvalidationMessage) *
+ (*ninvals_out + nmsgs_new));
+ memcpy(*invals_out + *ninvals_out, msgs_new,
+ nmsgs_new * sizeof(SharedInvalidationMessage));
+ *ninvals_out += nmsgs_new;
+ }
+}
+
/*
* Accumulate the invalidations for executing them later.
*
@@ -3441,7 +3527,6 @@ ReorderBufferAddInvalidations(ReorderBuffer *rb, TransactionId xid,
{
ReorderBufferTXN *txn;
MemoryContext oldcontext;
- ReorderBufferChange *change;
txn = ReorderBufferTXNByXid(rb, xid, true, NULL, lsn, true);
@@ -3456,35 +3541,74 @@ ReorderBufferAddInvalidations(ReorderBuffer *rb, TransactionId xid,
Assert(nmsgs > 0);
- /* Accumulate invalidations. */
- if (txn->ninvalidations == 0)
- {
- txn->ninvalidations = nmsgs;
- txn->invalidations = (SharedInvalidationMessage *)
- palloc(sizeof(SharedInvalidationMessage) * nmsgs);
- memcpy(txn->invalidations, msgs,
- sizeof(SharedInvalidationMessage) * nmsgs);
- }
- else
+ ReorderBufferAccumulateInvalidations(&txn->invalidations,
+ &txn->ninvalidations,
+ msgs, nmsgs);
+
+ ReorderBufferQueueInvalidations(rb, xid, lsn, nmsgs, msgs);
+
+ MemoryContextSwitchTo(oldcontext);
+}
+
+/*
+ * Accumulate the invalidations distributed by other committed transactions
+ * for executing them later.
+ *
+ * This function is similar to ReorderBufferAddInvalidations() but stores
+ * the given inval messages to the txn->invalidations_distributed with the
+ * overflow check.
+ *
+ * This needs to be called by committed transactions to distribute their
+ * inval messages to in-progress transactions.
+ */
+void
+ReorderBufferAddDistributedInvalidations(ReorderBuffer *rb, TransactionId xid,
+ XLogRecPtr lsn, Size nmsgs,
+ SharedInvalidationMessage *msgs)
+{
+ ReorderBufferTXN *txn;
+ MemoryContext oldcontext;
+
+ txn = ReorderBufferTXNByXid(rb, xid, true, NULL, lsn, true);
+
+ oldcontext = MemoryContextSwitchTo(rb->context);
+
+ /*
+ * Collect all the invalidations under the top transaction, if available,
+ * so that we can execute them all together. See comments
+ * ReorderBufferAddInvalidations.
+ */
+ txn = rbtxn_get_toptxn(txn);
+
+ Assert(nmsgs > 0);
+
+ /*
+ * Check the transaction has enough space for storing distributed
+ * invalidation messages.
+ */
+ if (txn->ninvalidations_distributed + nmsgs >= MAX_DISTR_INVAL_MSG_PER_TXN)
{
- txn->invalidations = (SharedInvalidationMessage *)
- repalloc(txn->invalidations, sizeof(SharedInvalidationMessage) *
- (txn->ninvalidations + nmsgs));
+ /*
+ * Mark the invalidation message as overflowed and free up the
+ * messages accumulated so far.
+ */
+ txn->txn_flags |= RBTXN_INVAL_OVERFLOWED;
- memcpy(txn->invalidations + txn->ninvalidations, msgs,
- nmsgs * sizeof(SharedInvalidationMessage));
- txn->ninvalidations += nmsgs;
+ if (txn->invalidations_distributed)
+ {
+ pfree(txn->invalidations_distributed);
+ txn->invalidations_distributed = NULL;
+ txn->ninvalidations_distributed = 0;
+ }
}
- change = ReorderBufferAllocChange(rb);
- change->action = REORDER_BUFFER_CHANGE_INVALIDATION;
- change->data.inval.ninvalidations = nmsgs;
- change->data.inval.invalidations = (SharedInvalidationMessage *)
- palloc(sizeof(SharedInvalidationMessage) * nmsgs);
- memcpy(change->data.inval.invalidations, msgs,
- sizeof(SharedInvalidationMessage) * nmsgs);
+ if (!rbtxn_inval_overflowed(txn))
+ ReorderBufferAccumulateInvalidations(&txn->invalidations_distributed,
+ &txn->ninvalidations_distributed,
+ msgs, nmsgs);
- ReorderBufferQueueChange(rb, xid, lsn, change, false);
+ /* Queue the invalidation messages into the transaction */
+ ReorderBufferQueueInvalidations(rb, xid, lsn, nmsgs, msgs);
MemoryContextSwitchTo(oldcontext);
}
diff --git a/src/backend/replication/logical/snapbuild.c b/src/backend/replication/logical/snapbuild.c
index 0d7bddbe4ed..adf18c397db 100644
--- a/src/backend/replication/logical/snapbuild.c
+++ b/src/backend/replication/logical/snapbuild.c
@@ -794,6 +794,13 @@ SnapBuildDistributeSnapshotAndInval(SnapBuild *builder, XLogRecPtr lsn, Transact
* contents built by the current transaction even after its decoding,
* which should have been invalidated due to concurrent catalog
* changing transaction.
+ *
+ * Distribute only the invalidation messages generated by the current
+ * committed transaction. Invalidation messages received from other
+ * transactions would have already been propagated to the relevant
+ * in-progress transactions. This transaction would have processed
+ * those invalidations, ensuring that subsequent transactions observe
+ * a consistent cache state.
*/
if (txn->xid != xid)
{
@@ -807,8 +814,9 @@ SnapBuildDistributeSnapshotAndInval(SnapBuild *builder, XLogRecPtr lsn, Transact
{
Assert(msgs != NULL);
- ReorderBufferAddInvalidations(builder->reorder, txn->xid, lsn,
- ninvalidations, msgs);
+ ReorderBufferAddDistributedInvalidations(builder->reorder,
+ txn->xid, lsn,
+ ninvalidations, msgs);
}
}
}
diff --git a/src/include/replication/reorderbuffer.h b/src/include/replication/reorderbuffer.h
index 24e88c409ba..6579bee3f27 100644
--- a/src/include/replication/reorderbuffer.h
+++ b/src/include/replication/reorderbuffer.h
@@ -176,6 +176,7 @@ typedef struct ReorderBufferChange
#define RBTXN_SENT_PREPARE 0x0200
#define RBTXN_IS_COMMITTED 0x0400
#define RBTXN_IS_ABORTED 0x0800
+#define RBTXN_INVAL_OVERFLOWED 0x1000
#define RBTXN_PREPARE_STATUS_MASK (RBTXN_IS_PREPARED | RBTXN_SKIPPED_PREPARE | RBTXN_SENT_PREPARE)
@@ -265,6 +266,12 @@ typedef struct ReorderBufferChange
((txn)->txn_flags & RBTXN_SKIPPED_PREPARE) != 0 \
)
+/* Is the array of inval message overflowed? */
+#define rbtxn_inval_overflowed(txn) \
+( \
+ ((txn)->txn_flags & RBTXN_INVAL_OVERFLOWED) != 0 \
+)
+
/* Is this a top-level transaction? */
#define rbtxn_is_toptxn(txn) \
( \
@@ -422,6 +429,26 @@ typedef struct ReorderBufferTXN
uint32 ninvalidations;
SharedInvalidationMessage *invalidations;
+ /*
+ * Stores cache invalidation messages distributed by other transactions.
+ *
+ * It is acceptable to skip invalidations received from concurrent
+ * transactions during ReorderBufferForget and ReorderBufferInvalidate,
+ * because the transaction being discarded wouldn't have loaded any shared
+ * catalog entries and invalidating in the source transaction is
+ * sufficient (see the comment at the caller of ReorderBufferForget). In
+ * the case of ReorderBufferAbort(), invalidations are processed only if
+ * the transaction has already streamed some changes. When streaming
+ * occurs, any concurrent invalidations that happened before the
+ * triggering statement would have already been processed. That should be
+ * sufficient to maintain consistency. Therefore, it's safe to skip
+ * processing invalidations_distributed for ReorderBufferForget,
+ * ReorderBufferInvalidate, and ReorderBufferAbort.
+ */
+ uint32 ninvalidations_distributed;
+ SharedInvalidationMessage *invalidations_distributed;
+
+
/* ---
* Position in one of two lists:
* * list of subtransactions if we are *known* to be subxact
@@ -738,6 +765,9 @@ extern void ReorderBufferAddNewTupleCids(ReorderBuffer *rb, TransactionId xid,
CommandId cmin, CommandId cmax, CommandId combocid);
extern void ReorderBufferAddInvalidations(ReorderBuffer *rb, TransactionId xid, XLogRecPtr lsn,
Size nmsgs, SharedInvalidationMessage *msgs);
+extern void ReorderBufferAddDistributedInvalidations(ReorderBuffer *rb, TransactionId xid,
+ XLogRecPtr lsn, Size nmsgs,
+ SharedInvalidationMessage *msgs);
extern void ReorderBufferImmediateInvalidation(ReorderBuffer *rb, uint32 ninvalidations,
SharedInvalidationMessage *invalidations);
extern void ReorderBufferProcessXid(ReorderBuffer *rb, TransactionId xid, XLogRecPtr lsn);
--
2.43.0
^ permalink raw reply [nested|flat] 94+ messages in thread
* RE: Logical replication 'invalid memory alloc request size 1585837200' after upgrading to 17.5
@ 2025-06-05 11:07 Hayato Kuroda (Fujitsu) <kuroda.hayato@fujitsu.com>
parent: Amit Kapila <amit.kapila16@gmail.com>
1 sibling, 2 replies; 94+ messages in thread
From: Hayato Kuroda (Fujitsu) @ 2025-06-05 11:07 UTC (permalink / raw)
To: Amit Kapila <amit.kapila16@gmail.com>; Masahiko Sawada <sawada.mshk@gmail.com>; +Cc: vignesh C <vignesh21@gmail.com>; Duncan Sands <duncan.sands@deepbluecap.com>; pgsql-bugs@lists.postgresql.org <pgsql-bugs@lists.postgresql.org>
Dear Amit,
> > ---
> > I'd like to make it clear again which case we need to execute
> > txn->invalidations as well as txn->invalidations_distributed (like in
> > ReorderBufferProcessTXN()) and which case we need to execute only
> > txn->invalidations (like in ReorderBufferForget() and
> > ReorderBufferAbort()). I think it might be worth putting some comments
> > about overall strategy somewhere.
> >
> > ---
> > BTW for back branches, a simple fix without ABI breakage would be to
> > introduce the RBTXN_INVAL_OVERFLOWED flag to limit the size of
> > txn->invalidations. That is, we accumulate inval messages both coming
> > from the current transaction and distributed by other transactions but
> > once the size reaches the threshold we invalidate all caches. Is it
> > worth considering for back branches?
> >
>
> It should work and is worth considering. The main concern would be
> that it will hit sooner than we expect in the field, seeing the recent
> reports. So, such a change has the potential to degrade the
> performance. I feel that the number of people impacted due to
> performance would be more than the number of people impacted due to
> such an ABI change (adding the new members at the end of
> ReorderBufferTXN). However, if we think we want to go safe w.r.t
> extensions that can rely on the sizeof ReorderBufferTXN then your
> proposal makes sense.
While considering the approach, I found a doubtful point. Consider the below
workload:
0. S1: CREATE TABLE d(data text not null);
1. S1: BEGIN;
2. S1: INSERT INTO d VALUES ('d1')
3. S2: BEGIN;
4. S2: INSERT INTO d VALUES ('d2')
5. S1: ALTER PUBLICATION pb ADD TABLE d;
6. S1: ... lots of DDLs so overflow happens
7. S1: COMMIT;
8. S2: INSERT INTO d VALUES ('d3');
9. S2: COMMIT;
10. S2: INSERT INTO d VALUES ('d4');
In this case, the inval message generated by step 5 is discarded at step 6. No
invalidation messages are distributed in the SnapBuildDistributeSnapshotAndInval().
While decoding S2, relcache cannot be discarded and tuples d3 and d4 won't be
replicated. Do you think this can happen?
Note that this won't happen for v11 patch. The patch won't discard txn->invalidations
in case of overflow, needed inval messages can be distributed.
Best regards,
Hayato Kuroda
FUJITSU LIMITED
^ permalink raw reply [nested|flat] 94+ messages in thread
* RE: Logical replication 'invalid memory alloc request size 1585837200' after upgrading to 17.5
@ 2025-06-05 12:14 Hayato Kuroda (Fujitsu) <kuroda.hayato@fujitsu.com>
parent: Hayato Kuroda (Fujitsu) <kuroda.hayato@fujitsu.com>
1 sibling, 0 replies; 94+ messages in thread
From: Hayato Kuroda (Fujitsu) @ 2025-06-05 12:14 UTC (permalink / raw)
To: Amit Kapila <amit.kapila16@gmail.com>; Masahiko Sawada <sawada.mshk@gmail.com>; +Cc: vignesh C <vignesh21@gmail.com>; Duncan Sands <duncan.sands@deepbluecap.com>; pgsql-bugs@lists.postgresql.org <pgsql-bugs@lists.postgresql.org>; Hayato Kuroda (Fujitsu) <kuroda.hayato@fujitsu.com>
>0. S1: CREATE TABLE d(data text not null);
>1. S1: BEGIN;
>2. S1: INSERT INTO d VALUES ('d1')
>3. S2: BEGIN;
>4. S2: INSERT INTO d VALUES ('d2')
>5. S1: ALTER PUBLICATION pb ADD TABLE d;
>6. S1: ... lots of DDLs so overflow happens
>7. S1: COMMIT;
>8. S2: INSERT INTO d VALUES ('d3');
>9. S2: COMMIT;
>10. S2: INSERT INTO d VALUES ('d4');
>
> In this case, the inval message generated by step 5 is discarded at step 6. No
> invalidation messages are distributed in the
> SnapBuildDistributeSnapshotAndInval().
> While decoding S2, relcache cannot be discarded and tuples d3 and d4 won't be
> replicated. Do you think this can happen?
Before running the workload, pg_recvlogical was run to check the output.
At step 6, I created and dropped the same table several times until the debug log was output:
```
[650552] LOG: RBTXN_INVAL_OVERFLOWED flag is set
[650552] STATEMENT: START_REPLICATION SLOT "test" LOGICAL 0/0 ("proto_version" '4', "publication_names" 'pb')
```
After that I executed till 10 but no output was done by pg_recvlogical
Best regards,
Hayato Kuroda
FUJITSU LIMITED
Attachments:
[application/octet-stream] 0001-implement-limitation-approach-for-PG17.patch (7.0K, ../../OSCPR01MB14966C8D376CB550AB5FA6040F56FA@OSCPR01MB14966.jpnprd01.prod.outlook.com/2-0001-implement-limitation-approach-for-PG17.patch)
download | inline diff:
From a853ad2474b01d08e6f392655b749be404c1d273 Mon Sep 17 00:00:00 2001
From: Hayato Kuroda <kuroda.hayato@fujitsu.com>
Date: Thu, 5 Jun 2025 21:00:09 +0900
Subject: [PATCH] implement limitation approach for PG17
---
.../replication/logical/reorderbuffer.c | 116 ++++++++++++++----
src/include/replication/reorderbuffer.h | 7 ++
2 files changed, 98 insertions(+), 25 deletions(-)
diff --git a/src/backend/replication/logical/reorderbuffer.c b/src/backend/replication/logical/reorderbuffer.c
index 03eb005c39d..a73e58e9188 100644
--- a/src/backend/replication/logical/reorderbuffer.c
+++ b/src/backend/replication/logical/reorderbuffer.c
@@ -109,9 +109,22 @@
#include "storage/sinval.h"
#include "utils/builtins.h"
#include "utils/memutils.h"
+#include "utils/inval.h"
#include "utils/rel.h"
#include "utils/relfilenumbermap.h"
+/*
+ * Each transaction is limited to a maximum of 512MB of inval messages. Once
+ * the number of distributed inval messages reach this threshold, the
+ * transaction is marked as RBTXN_INVAL_OVERFLOWED, invalidating all caches
+ * instead as we have lost some inval messages and hence
+ * don't know what needs to be invalidated.
+ *
+ * XXX set to very small value for testing purpose
+ */
+#define MAX_DISTR_INVAL_MSG_PER_TXN \
+ ((5 * 16) / sizeof(SharedInvalidationMessage))
+
/* entry for a hash table we use to map from xid to our transaction state */
typedef struct ReorderBufferTXNByIdEnt
{
@@ -2574,7 +2587,14 @@ ReorderBufferProcessTXN(ReorderBuffer *rb, ReorderBufferTXN *txn,
AbortCurrentTransaction();
/* make sure there's no cache pollution */
- ReorderBufferExecuteInvalidations(txn->ninvalidations, txn->invalidations);
+ if (rbtxn_inval_overflowed(txn))
+ {
+ Assert(txn->ninvalidations == 0);
+ InvalidateSystemCaches();
+ }
+ else
+ ReorderBufferExecuteInvalidations(txn->ninvalidations,
+ txn->invalidations);
if (using_subtxn)
RollbackAndReleaseCurrentSubTransaction();
@@ -2620,8 +2640,13 @@ ReorderBufferProcessTXN(ReorderBuffer *rb, ReorderBufferTXN *txn,
AbortCurrentTransaction();
/* make sure there's no cache pollution */
- ReorderBufferExecuteInvalidations(txn->ninvalidations,
- txn->invalidations);
+ if (rbtxn_inval_overflowed(txn))
+ {
+ Assert(txn->ninvalidations == 0);
+ InvalidateSystemCaches();
+ }
+ else
+ ReorderBufferExecuteInvalidations(txn->ninvalidations, txn->invalidations);
if (using_subtxn)
RollbackAndReleaseCurrentSubTransaction();
@@ -2910,8 +2935,15 @@ ReorderBufferFinishPrepared(ReorderBuffer *rb, TransactionId xid,
rb->rollback_prepared(rb, txn, prepare_end_lsn, prepare_time);
/* cleanup: make sure there's no cache pollution */
- ReorderBufferExecuteInvalidations(txn->ninvalidations,
- txn->invalidations);
+ if (rbtxn_inval_overflowed(txn))
+ {
+ Assert(txn->ninvalidations == 0);
+ InvalidateSystemCaches();
+ }
+ else
+ ReorderBufferExecuteInvalidations(txn->ninvalidations,
+ txn->invalidations);
+
ReorderBufferCleanupTXN(rb, txn);
}
@@ -2953,7 +2985,12 @@ ReorderBufferAbort(ReorderBuffer *rb, TransactionId xid, XLogRecPtr lsn,
* happened in this transaction). We don't want the decoding of future
* transactions to use those cache entries so execute invalidations.
*/
- if (txn->ninvalidations > 0)
+ if (rbtxn_inval_overflowed(txn))
+ {
+ Assert(txn->ninvalidations == 0);
+ InvalidateSystemCaches();
+ }
+ else if (txn->ninvalidations > 0)
ReorderBufferImmediateInvalidation(rb, txn->ninvalidations,
txn->invalidations);
}
@@ -3042,9 +3079,14 @@ ReorderBufferForget(ReorderBuffer *rb, TransactionId xid, XLogRecPtr lsn)
* interested in the transaction's contents, it could have manipulated the
* catalog and we need to update the caches according to that.
*/
- if (txn->base_snapshot != NULL && txn->ninvalidations > 0)
- ReorderBufferImmediateInvalidation(rb, txn->ninvalidations,
- txn->invalidations);
+ if (txn->base_snapshot != NULL)
+ {
+ if (rbtxn_inval_overflowed(txn))
+ InvalidateSystemCaches();
+ else if (txn->ninvalidations > 0)
+ ReorderBufferImmediateInvalidation(rb, txn->ninvalidations,
+ txn->invalidations);
+ }
else
Assert(txn->ninvalidations == 0);
@@ -3347,24 +3389,48 @@ ReorderBufferAddInvalidations(ReorderBuffer *rb, TransactionId xid,
Assert(nmsgs > 0);
- /* Accumulate invalidations. */
- if (txn->ninvalidations == 0)
+ /* Accumulate the invalidation messages unless it's overflowed */
+ if (!rbtxn_inval_overflowed(txn))
{
- txn->ninvalidations = nmsgs;
- txn->invalidations = (SharedInvalidationMessage *)
- palloc(sizeof(SharedInvalidationMessage) * nmsgs);
- memcpy(txn->invalidations, msgs,
- sizeof(SharedInvalidationMessage) * nmsgs);
- }
- else
- {
- txn->invalidations = (SharedInvalidationMessage *)
- repalloc(txn->invalidations, sizeof(SharedInvalidationMessage) *
- (txn->ninvalidations + nmsgs));
+ /*
+ * Check the transaction has enough space for storing invalidation
+ * messages.
+ */
+ if (txn->ninvalidations + nmsgs >= MAX_DISTR_INVAL_MSG_PER_TXN)
+ {
+ /*
+ * Mark the invalidation message as overflowed and free up the
+ * messages accumulated so far.
+ */
+ txn->txn_flags |= RBTXN_INVAL_OVERFLOWED;
+
+ elog(LOG, "RBTXN_INVAL_OVERFLOWED flag is set");
+
+ if (txn->invalidations)
+ {
+ pfree(txn->invalidations);
+ txn->invalidations = NULL;
+ txn->ninvalidations = 0;
+ }
+ }
+ else if (txn->ninvalidations == 0)
+ {
+ txn->ninvalidations = nmsgs;
+ txn->invalidations = (SharedInvalidationMessage *)
+ palloc(sizeof(SharedInvalidationMessage) * nmsgs);
+ memcpy(txn->invalidations, msgs,
+ sizeof(SharedInvalidationMessage) * nmsgs);
+ }
+ else
+ {
+ txn->invalidations = (SharedInvalidationMessage *)
+ repalloc(txn->invalidations, sizeof(SharedInvalidationMessage) *
+ (txn->ninvalidations + nmsgs));
- memcpy(txn->invalidations + txn->ninvalidations, msgs,
- nmsgs * sizeof(SharedInvalidationMessage));
- txn->ninvalidations += nmsgs;
+ memcpy(txn->invalidations + txn->ninvalidations, msgs,
+ nmsgs * sizeof(SharedInvalidationMessage));
+ txn->ninvalidations += nmsgs;
+ }
}
change = ReorderBufferGetChange(rb);
diff --git a/src/include/replication/reorderbuffer.h b/src/include/replication/reorderbuffer.h
index 4c56f219fd8..a23aa7fefb0 100644
--- a/src/include/replication/reorderbuffer.h
+++ b/src/include/replication/reorderbuffer.h
@@ -168,6 +168,7 @@ typedef struct ReorderBufferChange
#define RBTXN_PREPARE 0x0040
#define RBTXN_SKIPPED_PREPARE 0x0080
#define RBTXN_HAS_STREAMABLE_CHANGE 0x0100
+#define RBTXN_INVAL_OVERFLOWED 0x0200
/* Does the transaction have catalog changes? */
#define rbtxn_has_catalog_changes(txn) \
@@ -231,6 +232,12 @@ typedef struct ReorderBufferChange
((txn)->txn_flags & RBTXN_SKIPPED_PREPARE) != 0 \
)
+/* Is the array of inval message overflowed? */
+#define rbtxn_inval_overflowed(txn) \
+( \
+ ((txn)->txn_flags & RBTXN_INVAL_OVERFLOWED) != 0 \
+)
+
/* Is this a top-level transaction? */
#define rbtxn_is_toptxn(txn) \
( \
--
2.47.1
^ permalink raw reply [nested|flat] 94+ messages in thread
* Re: Logical replication 'invalid memory alloc request size 1585837200' after upgrading to 17.5
@ 2025-06-05 17:43 Masahiko Sawada <sawada.mshk@gmail.com>
parent: Hayato Kuroda (Fujitsu) <kuroda.hayato@fujitsu.com>
0 siblings, 2 replies; 94+ messages in thread
From: Masahiko Sawada @ 2025-06-05 17:43 UTC (permalink / raw)
To: Hayato Kuroda (Fujitsu) <kuroda.hayato@fujitsu.com>; +Cc: Amit Kapila <amit.kapila16@gmail.com>; Duncan Sands <duncan.sands@deepbluecap.com>; pgsql-bugs@lists.postgresql.org <pgsql-bugs@lists.postgresql.org>; vignesh C <vignesh21@gmail.com>
On Wed, Jun 4, 2025 at 11:36 PM Hayato Kuroda (Fujitsu)
<kuroda.hayato@fujitsu.com> wrote:
>
> Dear Sawada-san,
>
> > Alternative idea is to lower the constant value when using an
> > assertion build. That way, we don't need to rely on injection points
> > being enabled.
>
> Hmm, possible but I prefer current one. Two concerns:
>
> 1.
> USE_ASSERT_CHECKING has not been used to change the value yet. The main usage is
> to call debug functions in debug build.
I think we have a similar precedent such as MT_NRELS_HASH to improve
the test coverages.
> 2.
> If we add tests which is usable only for debug build, it must be run only when it
> is enabled. IIUC such test does not exist yet.
I think we need to test cases not to check if we reach a specific code
point but to check if we can get the correct results even if we've
executed various code paths. As for this bug, it is better to check
that it works properly in a variety of cases. That way, we can check
overflow cases and non-overflow cases also in test cases added in the
future, improving the test coverage more.
Regards,
--
Masahiko Sawada
Amazon Web Services: https://aws.amazon.com
^ permalink raw reply [nested|flat] 94+ messages in thread
* Re: Logical replication 'invalid memory alloc request size 1585837200' after upgrading to 17.5
@ 2025-06-05 18:49 Masahiko Sawada <sawada.mshk@gmail.com>
parent: Amit Kapila <amit.kapila16@gmail.com>
1 sibling, 0 replies; 94+ messages in thread
From: Masahiko Sawada @ 2025-06-05 18:49 UTC (permalink / raw)
To: Amit Kapila <amit.kapila16@gmail.com>; +Cc: vignesh C <vignesh21@gmail.com>; Hayato Kuroda (Fujitsu) <kuroda.hayato@fujitsu.com>; Duncan Sands <duncan.sands@deepbluecap.com>; pgsql-bugs@lists.postgresql.org <pgsql-bugs@lists.postgresql.org>
On Wed, Jun 4, 2025 at 11:20 PM Amit Kapila <amit.kapila16@gmail.com> wrote:
>
> On Thu, Jun 5, 2025 at 3:19 AM Masahiko Sawada <sawada.mshk@gmail.com> wrote:
> >
> > On Tue, Jun 3, 2025 at 11:48 PM vignesh C <vignesh21@gmail.com> wrote:
> > >
> > > On Wed, 4 Jun 2025 at 01:14, Masahiko Sawada <sawada.mshk@gmail.com> wrote:
> >
> > Thank you for updating the patch. I have some comments and questions:
> >
> > In ReorderBufferAbort():
> >
> > /*
> > * We might have decoded changes for this transaction that could load
> > * the cache as per the current transaction's view (consider DDL's
> > * happened in this transaction). We don't want the decoding of future
> > * transactions to use those cache entries so execute invalidations.
> > */
> > if (txn->ninvalidations > 0)
> > ReorderBufferImmediateInvalidation(rb, txn->ninvalidations,
> > txn->invalidations);
> >
> > I think that if the txn->invalidations_distributed is overflowed, we
> > would miss executing the txn->invalidations here. Probably the same is
> > true for ReorderBufferForget() and ReorderBufferInvalidate().
> >
>
> This is because of the following check "if
> (!rbtxn_inval_overflowed(txn))" in function
> ReorderBufferAddInvalidations(). What is the need of such a check in
> this function? We don't need to execute distributed invalidations in
> cases like ReorderBufferForget() when we haven't decoded any changes.
>
> > ---
> > I'd like to make it clear again which case we need to execute
> > txn->invalidations as well as txn->invalidations_distributed (like in
> > ReorderBufferProcessTXN()) and which case we need to execute only
> > txn->invalidations (like in ReorderBufferForget() and
> > ReorderBufferAbort()). I think it might be worth putting some comments
> > about overall strategy somewhere.
> >
> > ---
> > BTW for back branches, a simple fix without ABI breakage would be to
> > introduce the RBTXN_INVAL_OVERFLOWED flag to limit the size of
> > txn->invalidations. That is, we accumulate inval messages both coming
> > from the current transaction and distributed by other transactions but
> > once the size reaches the threshold we invalidate all caches. Is it
> > worth considering for back branches?
> >
>
> It should work and is worth considering. The main concern would be
> that it will hit sooner than we expect in the field, seeing the recent
> reports. So, such a change has the potential to degrade the
> performance. I feel that the number of people impacted due to
> performance would be more than the number of people impacted due to
> such an ABI change (adding the new members at the end of
> ReorderBufferTXN).
That's a fair point. I initially assumed that DDLs were not executed
often in practice, but analyzing this bug has made me realize this
assumption was misguided.
Regards,
--
Masahiko Sawada
Amazon Web Services: https://aws.amazon.com
^ permalink raw reply [nested|flat] 94+ messages in thread
* Re: Logical replication 'invalid memory alloc request size 1585837200' after upgrading to 17.5
@ 2025-06-05 19:21 Masahiko Sawada <sawada.mshk@gmail.com>
parent: Hayato Kuroda (Fujitsu) <kuroda.hayato@fujitsu.com>
1 sibling, 1 reply; 94+ messages in thread
From: Masahiko Sawada @ 2025-06-05 19:21 UTC (permalink / raw)
To: Hayato Kuroda (Fujitsu) <kuroda.hayato@fujitsu.com>; +Cc: Amit Kapila <amit.kapila16@gmail.com>; vignesh C <vignesh21@gmail.com>; Duncan Sands <duncan.sands@deepbluecap.com>; pgsql-bugs@lists.postgresql.org <pgsql-bugs@lists.postgresql.org>
On Thu, Jun 5, 2025 at 4:07 AM Hayato Kuroda (Fujitsu)
<kuroda.hayato@fujitsu.com> wrote:
>
> Dear Amit,
>
> > > ---
> > > I'd like to make it clear again which case we need to execute
> > > txn->invalidations as well as txn->invalidations_distributed (like in
> > > ReorderBufferProcessTXN()) and which case we need to execute only
> > > txn->invalidations (like in ReorderBufferForget() and
> > > ReorderBufferAbort()). I think it might be worth putting some comments
> > > about overall strategy somewhere.
> > >
> > > ---
> > > BTW for back branches, a simple fix without ABI breakage would be to
> > > introduce the RBTXN_INVAL_OVERFLOWED flag to limit the size of
> > > txn->invalidations. That is, we accumulate inval messages both coming
> > > from the current transaction and distributed by other transactions but
> > > once the size reaches the threshold we invalidate all caches. Is it
> > > worth considering for back branches?
> > >
> >
> > It should work and is worth considering. The main concern would be
> > that it will hit sooner than we expect in the field, seeing the recent
> > reports. So, such a change has the potential to degrade the
> > performance. I feel that the number of people impacted due to
> > performance would be more than the number of people impacted due to
> > such an ABI change (adding the new members at the end of
> > ReorderBufferTXN). However, if we think we want to go safe w.r.t
> > extensions that can rely on the sizeof ReorderBufferTXN then your
> > proposal makes sense.
>
> While considering the approach, I found a doubtful point. Consider the below
> workload:
>
> 0. S1: CREATE TABLE d(data text not null);
> 1. S1: BEGIN;
> 2. S1: INSERT INTO d VALUES ('d1')
> 3. S2: BEGIN;
> 4. S2: INSERT INTO d VALUES ('d2')
> 5. S1: ALTER PUBLICATION pb ADD TABLE d;
> 6. S1: ... lots of DDLs so overflow happens
> 7. S1: COMMIT;
> 8. S2: INSERT INTO d VALUES ('d3');
> 9. S2: COMMIT;
> 10. S2: INSERT INTO d VALUES ('d4');
>
> In this case, the inval message generated by step 5 is discarded at step 6. No
> invalidation messages are distributed in the SnapBuildDistributeSnapshotAndInval().
> While decoding S2, relcache cannot be discarded and tuples d3 and d4 won't be
> replicated. Do you think this can happen?
I think that once the S1's inval messages got overflowed, we should
mark other transactions as overflowed instead of distributing inval
messages.
Regards,
--
Masahiko Sawada
Amazon Web Services: https://aws.amazon.com
^ permalink raw reply [nested|flat] 94+ messages in thread
* Re: Logical replication 'invalid memory alloc request size 1585837200' after upgrading to 17.5
@ 2025-06-05 21:59 Masahiko Sawada <sawada.mshk@gmail.com>
parent: vignesh C <vignesh21@gmail.com>
0 siblings, 0 replies; 94+ messages in thread
From: Masahiko Sawada @ 2025-06-05 21:59 UTC (permalink / raw)
To: vignesh C <vignesh21@gmail.com>; +Cc: Amit Kapila <amit.kapila16@gmail.com>; Hayato Kuroda (Fujitsu) <kuroda.hayato@fujitsu.com>; Duncan Sands <duncan.sands@deepbluecap.com>; pgsql-bugs@lists.postgresql.org <pgsql-bugs@lists.postgresql.org>
On Thu, Jun 5, 2025 at 3:28 AM vignesh C <vignesh21@gmail.com> wrote:
>
> On Thu, 5 Jun 2025 at 03:19, Masahiko Sawada <sawada.mshk@gmail.com> wrote:
> >
> > Thank you for updating the patch. I have some comments and questions:
> >
> > In ReorderBufferAbort():
> >
> > /*
> > * We might have decoded changes for this transaction that could load
> > * the cache as per the current transaction's view (consider DDL's
> > * happened in this transaction). We don't want the decoding of future
> > * transactions to use those cache entries so execute invalidations.
> > */
> > if (txn->ninvalidations > 0)
> > ReorderBufferImmediateInvalidation(rb, txn->ninvalidations,
> > txn->invalidations);
> >
> > I think that if the txn->invalidations_distributed is overflowed, we
> > would miss executing the txn->invalidations here. Probably the same is
> > true for ReorderBufferForget() and ReorderBufferInvalidate().
>
> I'm accumulating the invalidations in txn->invalidations irrespective
> of RBTXN_INVAL_OVERFLOWED txn.
Agreed with this change.
>
> > ---
> > I'd like to make it clear again which case we need to execute
> > txn->invalidations as well as txn->invalidations_distributed (like in
> > ReorderBufferProcessTXN()) and which case we need to execute only
> > txn->invalidations (like in ReorderBufferForget() and
> > ReorderBufferAbort()). I think it might be worth putting some comments
> > about overall strategy somewhere.
>
> I have added comments for this, feel free to reword it if some
> changes are required.
>
> The attached v11 version patch has the changes for the same.
I think the patch is getting into good shape. I've attached a patch
that includes changes I recommend. For example, it's better to rename
RBTXN_INVAL_OVERFLOWED to RBTXN_DISTR_INVAL_OVERFLOWED, and it
includes some comment updates. Please review them.
Regards,
--
Masahiko Sawada
Amazon Web Services: https://aws.amazon.com
Attachments:
[application/octet-stream] change_v11_masahiko.patch (7.1K, ../../CAD21AoDaCL9X4E8VAe=fYa=zjqGTKRJW13dTPazqAuOAEEykOg@mail.gmail.com/2-change_v11_masahiko.patch)
download | inline diff:
diff --git a/src/backend/replication/logical/reorderbuffer.c b/src/backend/replication/logical/reorderbuffer.c
index 606c94df7db..c4299c76fb1 100644
--- a/src/backend/replication/logical/reorderbuffer.c
+++ b/src/backend/replication/logical/reorderbuffer.c
@@ -115,10 +115,12 @@
#include "utils/relfilenumbermap.h"
/*
- * Each transaction is limited to a maximum of 8MB of inval messages distributed
- * from other transaction. Once the number of distributed inval messages reach
- * this threshold, the transaction is marked as RBTXN_INVAL_OVERFLOWED to
- * invalidate the complete cache.
+ * Each transaction has an 8MB limit for invalidation messages distributed from
+ * other transactions. This limit is set considering scenarios with many
+ * concurrent logical decoding operations. When the distributed invalidation
+ * messages reach this threshold, the transaction is marked as
+ * RBTXN_DISTR_INVAL_OVERFLOWED to invalidate the complete cache as we have lost
+ * some inval messages and hence don't know what needs to be invalidated.
*/
#define MAX_DISTR_INVAL_MSG_PER_TXN \
((8 * 1024 * 1024) / sizeof(SharedInvalidationMessage))
@@ -2677,7 +2679,7 @@ ReorderBufferProcessTXN(ReorderBuffer *rb, ReorderBufferTXN *txn,
AbortCurrentTransaction();
/* make sure there's no cache pollution */
- if (rbtxn_inval_overflowed(txn))
+ if (rbtxn_distr_inval_overflowed(txn))
{
Assert(txn->ninvalidations_distributed == 0);
InvalidateSystemCaches();
@@ -2736,7 +2738,7 @@ ReorderBufferProcessTXN(ReorderBuffer *rb, ReorderBufferTXN *txn,
AbortCurrentTransaction();
/* make sure there's no cache pollution */
- if (rbtxn_inval_overflowed(txn))
+ if (rbtxn_distr_inval_overflowed(txn))
{
Assert(txn->ninvalidations_distributed == 0);
InvalidateSystemCaches();
@@ -3095,7 +3097,8 @@ ReorderBufferAbort(ReorderBuffer *rb, TransactionId xid, XLogRecPtr lsn,
* We might have decoded changes for this transaction that could load
* the cache as per the current transaction's view (consider DDL's
* happened in this transaction). We don't want the decoding of future
- * transactions to use those cache entries so execute invalidations.
+ * transactions to use those cache entries so execute only the inval
+ * messages in this transaction.
*/
if (txn->ninvalidations > 0)
ReorderBufferImmediateInvalidation(rb, txn->ninvalidations,
@@ -3182,9 +3185,10 @@ ReorderBufferForget(ReorderBuffer *rb, TransactionId xid, XLogRecPtr lsn)
txn->final_lsn = lsn;
/*
- * Process cache invalidation messages if there are any. Even if we're not
- * interested in the transaction's contents, it could have manipulated the
- * catalog and we need to update the caches according to that.
+ * Process only cache invalidation messages in this transaction if there
+ * are any. Even if we're not interested in the transaction's contents, it
+ * could have manipulated the catalog and we need to update the caches
+ * according to that.
*/
if (txn->base_snapshot != NULL && txn->ninvalidations > 0)
ReorderBufferImmediateInvalidation(rb, txn->ninvalidations,
@@ -3457,7 +3461,7 @@ ReorderBufferAddNewTupleCids(ReorderBuffer *rb, TransactionId xid,
}
/*
- * Add a new invalidation message to the reorder buffer queue.
+ * Add new invalidation messages to the reorder buffer queue.
*/
static void
ReorderBufferQueueInvalidations(ReorderBuffer *rb, TransactionId xid,
@@ -3582,31 +3586,33 @@ ReorderBufferAddDistributedInvalidations(ReorderBuffer *rb, TransactionId xid,
Assert(nmsgs > 0);
- /*
- * Check the transaction has enough space for storing distributed
- * invalidation messages.
- */
- if (txn->ninvalidations_distributed + nmsgs >= MAX_DISTR_INVAL_MSG_PER_TXN)
+ if (!rbtxn_distr_inval_overflowed(txn))
{
/*
- * Mark the invalidation message as overflowed and free up the
- * messages accumulated so far.
+ * Check the transaction has enough space for storing distributed
+ * invalidation messages.
*/
- txn->txn_flags |= RBTXN_INVAL_OVERFLOWED;
-
- if (txn->invalidations_distributed)
+ if (txn->ninvalidations_distributed + nmsgs >= MAX_DISTR_INVAL_MSG_PER_TXN)
{
- pfree(txn->invalidations_distributed);
- txn->invalidations_distributed = NULL;
- txn->ninvalidations_distributed = 0;
+ /*
+ * Mark the invalidation message as overflowed and free up the
+ * messages accumulated so far.
+ */
+ txn->txn_flags |= RBTXN_DISTR_INVAL_OVERFLOWED;
+
+ if (txn->invalidations_distributed)
+ {
+ pfree(txn->invalidations_distributed);
+ txn->invalidations_distributed = NULL;
+ txn->ninvalidations_distributed = 0;
+ }
}
+ else
+ ReorderBufferAccumulateInvalidations(&txn->invalidations_distributed,
+ &txn->ninvalidations_distributed,
+ msgs, nmsgs);
}
- if (!rbtxn_inval_overflowed(txn))
- ReorderBufferAccumulateInvalidations(&txn->invalidations_distributed,
- &txn->ninvalidations_distributed,
- msgs, nmsgs);
-
/* Queue the invalidation messages into the transaction */
ReorderBufferQueueInvalidations(rb, xid, lsn, nmsgs, msgs);
diff --git a/src/include/replication/reorderbuffer.h b/src/include/replication/reorderbuffer.h
index 6579bee3f27..ed92830e75f 100644
--- a/src/include/replication/reorderbuffer.h
+++ b/src/include/replication/reorderbuffer.h
@@ -176,7 +176,7 @@ typedef struct ReorderBufferChange
#define RBTXN_SENT_PREPARE 0x0200
#define RBTXN_IS_COMMITTED 0x0400
#define RBTXN_IS_ABORTED 0x0800
-#define RBTXN_INVAL_OVERFLOWED 0x1000
+#define RBTXN_DISTR_INVAL_OVERFLOWED 0x1000
#define RBTXN_PREPARE_STATUS_MASK (RBTXN_IS_PREPARED | RBTXN_SKIPPED_PREPARE | RBTXN_SENT_PREPARE)
@@ -266,10 +266,10 @@ typedef struct ReorderBufferChange
((txn)->txn_flags & RBTXN_SKIPPED_PREPARE) != 0 \
)
-/* Is the array of inval message overflowed? */
-#define rbtxn_inval_overflowed(txn) \
+/* Is the array of distributed inval messages overflowed? */
+#define rbtxn_distr_inval_overflowed(txn) \
( \
- ((txn)->txn_flags & RBTXN_INVAL_OVERFLOWED) != 0 \
+ ((txn)->txn_flags & RBTXN_DISTR_INVAL_OVERFLOWED) != 0 \
)
/* Is this a top-level transaction? */
@@ -431,19 +431,6 @@ typedef struct ReorderBufferTXN
/*
* Stores cache invalidation messages distributed by other transactions.
- *
- * It is acceptable to skip invalidations received from concurrent
- * transactions during ReorderBufferForget and ReorderBufferInvalidate,
- * because the transaction being discarded wouldn't have loaded any shared
- * catalog entries and invalidating in the source transaction is
- * sufficient (see the comment at the caller of ReorderBufferForget). In
- * the case of ReorderBufferAbort(), invalidations are processed only if
- * the transaction has already streamed some changes. When streaming
- * occurs, any concurrent invalidations that happened before the
- * triggering statement would have already been processed. That should be
- * sufficient to maintain consistency. Therefore, it's safe to skip
- * processing invalidations_distributed for ReorderBufferForget,
- * ReorderBufferInvalidate, and ReorderBufferAbort.
*/
uint32 ninvalidations_distributed;
SharedInvalidationMessage *invalidations_distributed;
^ permalink raw reply [nested|flat] 94+ messages in thread
* Re: Logical replication 'invalid memory alloc request size 1585837200' after upgrading to 17.5
@ 2025-06-06 03:21 Amit Kapila <amit.kapila16@gmail.com>
parent: Masahiko Sawada <sawada.mshk@gmail.com>
0 siblings, 1 reply; 94+ messages in thread
From: Amit Kapila @ 2025-06-06 03:21 UTC (permalink / raw)
To: Masahiko Sawada <sawada.mshk@gmail.com>; +Cc: Hayato Kuroda (Fujitsu) <kuroda.hayato@fujitsu.com>; vignesh C <vignesh21@gmail.com>; Duncan Sands <duncan.sands@deepbluecap.com>; pgsql-bugs@lists.postgresql.org <pgsql-bugs@lists.postgresql.org>
On Fri, Jun 6, 2025 at 12:51 AM Masahiko Sawada <sawada.mshk@gmail.com> wrote:
>
> On Thu, Jun 5, 2025 at 4:07 AM Hayato Kuroda (Fujitsu)
> <kuroda.hayato@fujitsu.com> wrote:
> >
> > Dear Amit,
> >
> > > > ---
> > > > I'd like to make it clear again which case we need to execute
> > > > txn->invalidations as well as txn->invalidations_distributed (like in
> > > > ReorderBufferProcessTXN()) and which case we need to execute only
> > > > txn->invalidations (like in ReorderBufferForget() and
> > > > ReorderBufferAbort()). I think it might be worth putting some comments
> > > > about overall strategy somewhere.
> > > >
> > > > ---
> > > > BTW for back branches, a simple fix without ABI breakage would be to
> > > > introduce the RBTXN_INVAL_OVERFLOWED flag to limit the size of
> > > > txn->invalidations. That is, we accumulate inval messages both coming
> > > > from the current transaction and distributed by other transactions but
> > > > once the size reaches the threshold we invalidate all caches. Is it
> > > > worth considering for back branches?
> > > >
> > >
> > > It should work and is worth considering. The main concern would be
> > > that it will hit sooner than we expect in the field, seeing the recent
> > > reports. So, such a change has the potential to degrade the
> > > performance. I feel that the number of people impacted due to
> > > performance would be more than the number of people impacted due to
> > > such an ABI change (adding the new members at the end of
> > > ReorderBufferTXN). However, if we think we want to go safe w.r.t
> > > extensions that can rely on the sizeof ReorderBufferTXN then your
> > > proposal makes sense.
> >
> > While considering the approach, I found a doubtful point. Consider the below
> > workload:
> >
> > 0. S1: CREATE TABLE d(data text not null);
> > 1. S1: BEGIN;
> > 2. S1: INSERT INTO d VALUES ('d1')
> > 3. S2: BEGIN;
> > 4. S2: INSERT INTO d VALUES ('d2')
> > 5. S1: ALTER PUBLICATION pb ADD TABLE d;
> > 6. S1: ... lots of DDLs so overflow happens
> > 7. S1: COMMIT;
> > 8. S2: INSERT INTO d VALUES ('d3');
> > 9. S2: COMMIT;
> > 10. S2: INSERT INTO d VALUES ('d4');
> >
> > In this case, the inval message generated by step 5 is discarded at step 6. No
> > invalidation messages are distributed in the SnapBuildDistributeSnapshotAndInval().
> > While decoding S2, relcache cannot be discarded and tuples d3 and d4 won't be
> > replicated. Do you think this can happen?
>
> I think that once the S1's inval messages got overflowed, we should
> mark other transactions as overflowed instead of distributing inval
> messages.
>
Yeah, this should work, but are you still advocating that we go with
this approach (marking txn->invalidations also as overflowed) for
backbranches? In the previous email, you seemed to agree with the
performance impact due to DDLs, so it is not clear which approach you
prefer.
--
With Regards,
Amit Kapila.
^ permalink raw reply [nested|flat] 94+ messages in thread
* Re: Logical replication 'invalid memory alloc request size 1585837200' after upgrading to 17.5
@ 2025-06-06 03:43 Amit Kapila <amit.kapila16@gmail.com>
parent: Masahiko Sawada <sawada.mshk@gmail.com>
1 sibling, 1 reply; 94+ messages in thread
From: Amit Kapila @ 2025-06-06 03:43 UTC (permalink / raw)
To: Masahiko Sawada <sawada.mshk@gmail.com>; +Cc: Hayato Kuroda (Fujitsu) <kuroda.hayato@fujitsu.com>; Duncan Sands <duncan.sands@deepbluecap.com>; pgsql-bugs@lists.postgresql.org <pgsql-bugs@lists.postgresql.org>; vignesh C <vignesh21@gmail.com>
On Thu, Jun 5, 2025 at 11:14 PM Masahiko Sawada <sawada.mshk@gmail.com> wrote:
>
> On Wed, Jun 4, 2025 at 11:36 PM Hayato Kuroda (Fujitsu)
> <kuroda.hayato@fujitsu.com> wrote:
> >
> > Dear Sawada-san,
> >
> > > Alternative idea is to lower the constant value when using an
> > > assertion build. That way, we don't need to rely on injection points
> > > being enabled.
> >
> > Hmm, possible but I prefer current one. Two concerns:
> >
> > 1.
> > USE_ASSERT_CHECKING has not been used to change the value yet. The main usage is
> > to call debug functions in debug build.
>
> I think we have a similar precedent such as MT_NRELS_HASH to improve
> the test coverages.
>
> > 2.
> > If we add tests which is usable only for debug build, it must be run only when it
> > is enabled. IIUC such test does not exist yet.
>
> I think we need to test cases not to check if we reach a specific code
> point but to check if we can get the correct results even if we've
> executed various code paths. As for this bug, it is better to check
> that it works properly in a variety of cases. That way, we can check
> overflow cases and non-overflow cases also in test cases added in the
> future, improving the test coverage more.
>
This makes sense, but we should see whether some existing tests cover
this code path after lowering the limit in the overflow code path. One
minor point to consider is that at the time, the value MT_NRELS_HASH
was used to cover cases in a debug build, but we didn't have the
injection_point framework.
BTW, I noticed that you removed the following comments in your suggestions:
/*
* Stores cache invalidation messages distributed by other transactions.
- *
- * It is acceptable to skip invalidations received from concurrent
- * transactions during ReorderBufferForget and ReorderBufferInvalidate,
- * because the transaction being discarded wouldn't have loaded any shared
IIUC, you only mentioned having some comments like this for ease of
understanding, and now you are suggesting to remove those.
--
With Regards,
Amit Kapila.
^ permalink raw reply [nested|flat] 94+ messages in thread
* RE: Logical replication 'invalid memory alloc request size 1585837200' after upgrading to 17.5
@ 2025-06-06 06:22 Hayato Kuroda (Fujitsu) <kuroda.hayato@fujitsu.com>
parent: Masahiko Sawada <sawada.mshk@gmail.com>
1 sibling, 1 reply; 94+ messages in thread
From: Hayato Kuroda (Fujitsu) @ 2025-06-06 06:22 UTC (permalink / raw)
To: Masahiko Sawada <sawada.mshk@gmail.com>; +Cc: Amit Kapila <amit.kapila16@gmail.com>; Duncan Sands <duncan.sands@deepbluecap.com>; pgsql-bugs@lists.postgresql.org <pgsql-bugs@lists.postgresql.org>; vignesh C <vignesh21@gmail.com>
Dear Sawada-san,
> > 1.
> > USE_ASSERT_CHECKING has not been used to change the value yet. The main
> usage is
> > to call debug functions in debug build.
>
> I think we have a similar precedent such as MT_NRELS_HASH to improve
> the test coverages.
Oh, good detection, it seems a typical way.
> > 2.
> > If we add tests which is usable only for debug build, it must be run only when it
> > is enabled. IIUC such test does not exist yet.
>
> I think we need to test cases not to check if we reach a specific code
> point but to check if we can get the correct results even if we've
> executed various code paths. As for this bug, it is better to check
> that it works properly in a variety of cases. That way, we can check
> overflow cases and non-overflow cases also in test cases added in the
> future, improving the test coverage more.
Best regards,
Hayato Kuroda
FUJITSU LIMITED
^ permalink raw reply [nested|flat] 94+ messages in thread
* RE: Logical replication 'invalid memory alloc request size 1585837200' after upgrading to 17.5
@ 2025-06-06 06:54 Hayato Kuroda (Fujitsu) <kuroda.hayato@fujitsu.com>
parent: Hayato Kuroda (Fujitsu) <kuroda.hayato@fujitsu.com>
0 siblings, 0 replies; 94+ messages in thread
From: Hayato Kuroda (Fujitsu) @ 2025-06-06 06:54 UTC (permalink / raw)
To: Masahiko Sawada <sawada.mshk@gmail.com>; +Cc: Amit Kapila <amit.kapila16@gmail.com>; Duncan Sands <duncan.sands@deepbluecap.com>; 'pgsql-bugs@lists.postgresql.org' <pgsql-bugs@lists.postgresql.org>; vignesh C <vignesh21@gmail.com>
Dear Sawada-san,
Sorry, I mistakenly sent partial part of the post. Let me continue the reply to 2.
> > 2.
> > If we add tests which is usable only for debug build, it must be run only when it
> > is enabled. IIUC such test does not exist yet.
>
> I think we need to test cases not to check if we reach a specific code
> point but to check if we can get the correct results even if we've
> executed various code paths. As for this bug, it is better to check
> that it works properly in a variety of cases. That way, we can check
> overflow cases and non-overflow cases also in test cases added in the
> future, improving the test coverage more.
You meant that 1) we do not have to ensure we reached the overflow part by seeing
the actual log output, and 2) it should be tested by existing ones.
Based on your advice, I updated the patch set.
0001 contains changes raised by [1]. I checked then and looked good.
0002 reduces the limitation to extremely lower value. I confirmed by adding debug
log and 7 cases can cause the overflow.
Appending
=======
Added debug log:
```
--- a/src/backend/replication/logical/reorderbuffer.c
+++ b/src/backend/replication/logical/reorderbuffer.c
@@ -3607,6 +3607,8 @@ ReorderBufferAddDistributedInvalidations(ReorderBuffer *rb, TransactionId xid,
*/
txn->txn_flags |= RBTXN_DISTR_INVAL_OVERFLOWED;
+ elog(LOG, "RBTXN_DISTR_INVAL_OVERFLOWED is set to the transaction")
```
Method how I count the testcases:
```
testrun$ grep -rI "RBTXN_DISTR_INVAL_OVERFLOWED" | awk '{print $1 $6}' | sort -u
subscription/100_bugs/log/100_bugs_twoways.log:2025-06-06LOG:
test_decoding/isolation/log/postmaster.log:2025-06-06isolation/catalog_change_snapshot/s1
test_decoding/isolation/log/postmaster.log:2025-06-06isolation/concurrent_ddl_dml/s2
test_decoding/isolation/log/postmaster.log:2025-06-06isolation/concurrent_stream/s1
test_decoding/isolation/log/postmaster.log:2025-06-06isolation/invalidation_distribution/s2
test_decoding/isolation/log/postmaster.log:2025-06-06isolation/oldest_xmin/s0
test_decoding/isolation/log/postmaster.log:2025-06-06isolation/snapshot_transfer/s0
```
[1]: https://www.postgresql.org/message-id/CAD21AoDaCL9X4E8VAe%3DfYa%3DzjqGTKRJW13dTPazqAuOAEEykOg%40mail...
Best regards,
Hayato Kuroda
FUJITSU LIMITED
Attachments:
[application/octet-stream] v12-0001-Fix-exponential-memory-allocation-issue-in-logic.patch (13.9K, ../../OSCPR01MB1496608B31715BB094C194B70F56EA@OSCPR01MB14966.jpnprd01.prod.outlook.com/2-v12-0001-Fix-exponential-memory-allocation-issue-in-logic.patch)
download | inline diff:
From 6b78846aa77035027dd9c4d157b695ffa0c50088 Mon Sep 17 00:00:00 2001
From: Vignesh <vignesh21@gmail.com>
Date: Fri, 30 May 2025 12:06:07 +0530
Subject: [PATCH v12 1/2] Fix exponential memory allocation issue in logical
decoding due to invalidation message distribution
Commit 4909b38af0 added logic to distribute invalidation messages from
one transaction to all concurrent in-progress transactions. However,
since the source of each invalidation message was not tracked, transactions
could redundantly re-distribute already propagated messages. This led to an
exponential increase in invalidation messages and, ultimately, memory
allocation failures.
To fix this, a new field is introduced to track distributed invalidation
messages separately. This ensures that only transactions in the pre-existing
list receive the invalidations, preventing redundant propagation.
The accumulated distributed invalidations are now consumed at the end of
transaction processing in `ReorderBufferProcessTXN()`.
---
.../replication/logical/reorderbuffer.c | 196 +++++++++++++++---
src/backend/replication/logical/snapbuild.c | 12 +-
src/include/replication/reorderbuffer.h | 16 ++
3 files changed, 189 insertions(+), 35 deletions(-)
diff --git a/src/backend/replication/logical/reorderbuffer.c b/src/backend/replication/logical/reorderbuffer.c
index 67655111875..c4299c76fb1 100644
--- a/src/backend/replication/logical/reorderbuffer.c
+++ b/src/backend/replication/logical/reorderbuffer.c
@@ -109,10 +109,22 @@
#include "storage/procarray.h"
#include "storage/sinval.h"
#include "utils/builtins.h"
+#include "utils/inval.h"
#include "utils/memutils.h"
#include "utils/rel.h"
#include "utils/relfilenumbermap.h"
+/*
+ * Each transaction has an 8MB limit for invalidation messages distributed from
+ * other transactions. This limit is set considering scenarios with many
+ * concurrent logical decoding operations. When the distributed invalidation
+ * messages reach this threshold, the transaction is marked as
+ * RBTXN_DISTR_INVAL_OVERFLOWED to invalidate the complete cache as we have lost
+ * some inval messages and hence don't know what needs to be invalidated.
+ */
+#define MAX_DISTR_INVAL_MSG_PER_TXN \
+ ((8 * 1024 * 1024) / sizeof(SharedInvalidationMessage))
+
/* entry for a hash table we use to map from xid to our transaction state */
typedef struct ReorderBufferTXNByIdEnt
{
@@ -472,6 +484,12 @@ ReorderBufferFreeTXN(ReorderBuffer *rb, ReorderBufferTXN *txn)
txn->invalidations = NULL;
}
+ if (txn->invalidations_distributed)
+ {
+ pfree(txn->invalidations_distributed);
+ txn->invalidations_distributed = NULL;
+ }
+
/* Reset the toast hash */
ReorderBufferToastReset(rb, txn);
@@ -2661,7 +2679,17 @@ ReorderBufferProcessTXN(ReorderBuffer *rb, ReorderBufferTXN *txn,
AbortCurrentTransaction();
/* make sure there's no cache pollution */
- ReorderBufferExecuteInvalidations(txn->ninvalidations, txn->invalidations);
+ if (rbtxn_distr_inval_overflowed(txn))
+ {
+ Assert(txn->ninvalidations_distributed == 0);
+ InvalidateSystemCaches();
+ }
+ else
+ {
+ ReorderBufferExecuteInvalidations(txn->ninvalidations, txn->invalidations);
+ ReorderBufferExecuteInvalidations(txn->ninvalidations_distributed,
+ txn->invalidations_distributed);
+ }
if (using_subtxn)
RollbackAndReleaseCurrentSubTransaction();
@@ -2710,8 +2738,17 @@ ReorderBufferProcessTXN(ReorderBuffer *rb, ReorderBufferTXN *txn,
AbortCurrentTransaction();
/* make sure there's no cache pollution */
- ReorderBufferExecuteInvalidations(txn->ninvalidations,
- txn->invalidations);
+ if (rbtxn_distr_inval_overflowed(txn))
+ {
+ Assert(txn->ninvalidations_distributed == 0);
+ InvalidateSystemCaches();
+ }
+ else
+ {
+ ReorderBufferExecuteInvalidations(txn->ninvalidations, txn->invalidations);
+ ReorderBufferExecuteInvalidations(txn->ninvalidations_distributed,
+ txn->invalidations_distributed);
+ }
if (using_subtxn)
RollbackAndReleaseCurrentSubTransaction();
@@ -3060,7 +3097,8 @@ ReorderBufferAbort(ReorderBuffer *rb, TransactionId xid, XLogRecPtr lsn,
* We might have decoded changes for this transaction that could load
* the cache as per the current transaction's view (consider DDL's
* happened in this transaction). We don't want the decoding of future
- * transactions to use those cache entries so execute invalidations.
+ * transactions to use those cache entries so execute only the inval
+ * messages in this transaction.
*/
if (txn->ninvalidations > 0)
ReorderBufferImmediateInvalidation(rb, txn->ninvalidations,
@@ -3147,9 +3185,10 @@ ReorderBufferForget(ReorderBuffer *rb, TransactionId xid, XLogRecPtr lsn)
txn->final_lsn = lsn;
/*
- * Process cache invalidation messages if there are any. Even if we're not
- * interested in the transaction's contents, it could have manipulated the
- * catalog and we need to update the caches according to that.
+ * Process only cache invalidation messages in this transaction if there
+ * are any. Even if we're not interested in the transaction's contents, it
+ * could have manipulated the catalog and we need to update the caches
+ * according to that.
*/
if (txn->base_snapshot != NULL && txn->ninvalidations > 0)
ReorderBufferImmediateInvalidation(rb, txn->ninvalidations,
@@ -3421,6 +3460,57 @@ ReorderBufferAddNewTupleCids(ReorderBuffer *rb, TransactionId xid,
txn->ntuplecids++;
}
+/*
+ * Add new invalidation messages to the reorder buffer queue.
+ */
+static void
+ReorderBufferQueueInvalidations(ReorderBuffer *rb, TransactionId xid,
+ XLogRecPtr lsn, Size nmsgs,
+ SharedInvalidationMessage *msgs)
+{
+ ReorderBufferChange *change;
+
+ change = ReorderBufferAllocChange(rb);
+ change->action = REORDER_BUFFER_CHANGE_INVALIDATION;
+ change->data.inval.ninvalidations = nmsgs;
+ change->data.inval.invalidations = (SharedInvalidationMessage *)
+ palloc(sizeof(SharedInvalidationMessage) * nmsgs);
+ memcpy(change->data.inval.invalidations, msgs,
+ sizeof(SharedInvalidationMessage) * nmsgs);
+
+ ReorderBufferQueueChange(rb, xid, lsn, change, false);
+}
+
+/*
+ * A helper function for ReorderBufferAddInvalidations() and
+ * ReorderBufferAddDistributedInvalidations() to accumulate the invalidation
+ * messages to the **invals_out.
+ */
+static void
+ReorderBufferAccumulateInvalidations(SharedInvalidationMessage **invals_out,
+ uint32 *ninvals_out,
+ SharedInvalidationMessage *msgs_new,
+ Size nmsgs_new)
+{
+ if (*ninvals_out == 0)
+ {
+ *ninvals_out = nmsgs_new;
+ *invals_out = (SharedInvalidationMessage *)
+ palloc(sizeof(SharedInvalidationMessage) * nmsgs_new);
+ memcpy(*invals_out, msgs_new, sizeof(SharedInvalidationMessage) * nmsgs_new);
+ }
+ else
+ {
+ /* Enlarge the array of inval messages */
+ *invals_out = (SharedInvalidationMessage *)
+ repalloc(*invals_out, sizeof(SharedInvalidationMessage) *
+ (*ninvals_out + nmsgs_new));
+ memcpy(*invals_out + *ninvals_out, msgs_new,
+ nmsgs_new * sizeof(SharedInvalidationMessage));
+ *ninvals_out += nmsgs_new;
+ }
+}
+
/*
* Accumulate the invalidations for executing them later.
*
@@ -3441,7 +3531,6 @@ ReorderBufferAddInvalidations(ReorderBuffer *rb, TransactionId xid,
{
ReorderBufferTXN *txn;
MemoryContext oldcontext;
- ReorderBufferChange *change;
txn = ReorderBufferTXNByXid(rb, xid, true, NULL, lsn, true);
@@ -3456,35 +3545,76 @@ ReorderBufferAddInvalidations(ReorderBuffer *rb, TransactionId xid,
Assert(nmsgs > 0);
- /* Accumulate invalidations. */
- if (txn->ninvalidations == 0)
- {
- txn->ninvalidations = nmsgs;
- txn->invalidations = (SharedInvalidationMessage *)
- palloc(sizeof(SharedInvalidationMessage) * nmsgs);
- memcpy(txn->invalidations, msgs,
- sizeof(SharedInvalidationMessage) * nmsgs);
- }
- else
+ ReorderBufferAccumulateInvalidations(&txn->invalidations,
+ &txn->ninvalidations,
+ msgs, nmsgs);
+
+ ReorderBufferQueueInvalidations(rb, xid, lsn, nmsgs, msgs);
+
+ MemoryContextSwitchTo(oldcontext);
+}
+
+/*
+ * Accumulate the invalidations distributed by other committed transactions
+ * for executing them later.
+ *
+ * This function is similar to ReorderBufferAddInvalidations() but stores
+ * the given inval messages to the txn->invalidations_distributed with the
+ * overflow check.
+ *
+ * This needs to be called by committed transactions to distribute their
+ * inval messages to in-progress transactions.
+ */
+void
+ReorderBufferAddDistributedInvalidations(ReorderBuffer *rb, TransactionId xid,
+ XLogRecPtr lsn, Size nmsgs,
+ SharedInvalidationMessage *msgs)
+{
+ ReorderBufferTXN *txn;
+ MemoryContext oldcontext;
+
+ txn = ReorderBufferTXNByXid(rb, xid, true, NULL, lsn, true);
+
+ oldcontext = MemoryContextSwitchTo(rb->context);
+
+ /*
+ * Collect all the invalidations under the top transaction, if available,
+ * so that we can execute them all together. See comments
+ * ReorderBufferAddInvalidations.
+ */
+ txn = rbtxn_get_toptxn(txn);
+
+ Assert(nmsgs > 0);
+
+ if (!rbtxn_distr_inval_overflowed(txn))
{
- txn->invalidations = (SharedInvalidationMessage *)
- repalloc(txn->invalidations, sizeof(SharedInvalidationMessage) *
- (txn->ninvalidations + nmsgs));
+ /*
+ * Check the transaction has enough space for storing distributed
+ * invalidation messages.
+ */
+ if (txn->ninvalidations_distributed + nmsgs >= MAX_DISTR_INVAL_MSG_PER_TXN)
+ {
+ /*
+ * Mark the invalidation message as overflowed and free up the
+ * messages accumulated so far.
+ */
+ txn->txn_flags |= RBTXN_DISTR_INVAL_OVERFLOWED;
- memcpy(txn->invalidations + txn->ninvalidations, msgs,
- nmsgs * sizeof(SharedInvalidationMessage));
- txn->ninvalidations += nmsgs;
+ if (txn->invalidations_distributed)
+ {
+ pfree(txn->invalidations_distributed);
+ txn->invalidations_distributed = NULL;
+ txn->ninvalidations_distributed = 0;
+ }
+ }
+ else
+ ReorderBufferAccumulateInvalidations(&txn->invalidations_distributed,
+ &txn->ninvalidations_distributed,
+ msgs, nmsgs);
}
- change = ReorderBufferAllocChange(rb);
- change->action = REORDER_BUFFER_CHANGE_INVALIDATION;
- change->data.inval.ninvalidations = nmsgs;
- change->data.inval.invalidations = (SharedInvalidationMessage *)
- palloc(sizeof(SharedInvalidationMessage) * nmsgs);
- memcpy(change->data.inval.invalidations, msgs,
- sizeof(SharedInvalidationMessage) * nmsgs);
-
- ReorderBufferQueueChange(rb, xid, lsn, change, false);
+ /* Queue the invalidation messages into the transaction */
+ ReorderBufferQueueInvalidations(rb, xid, lsn, nmsgs, msgs);
MemoryContextSwitchTo(oldcontext);
}
diff --git a/src/backend/replication/logical/snapbuild.c b/src/backend/replication/logical/snapbuild.c
index 0d7bddbe4ed..adf18c397db 100644
--- a/src/backend/replication/logical/snapbuild.c
+++ b/src/backend/replication/logical/snapbuild.c
@@ -794,6 +794,13 @@ SnapBuildDistributeSnapshotAndInval(SnapBuild *builder, XLogRecPtr lsn, Transact
* contents built by the current transaction even after its decoding,
* which should have been invalidated due to concurrent catalog
* changing transaction.
+ *
+ * Distribute only the invalidation messages generated by the current
+ * committed transaction. Invalidation messages received from other
+ * transactions would have already been propagated to the relevant
+ * in-progress transactions. This transaction would have processed
+ * those invalidations, ensuring that subsequent transactions observe
+ * a consistent cache state.
*/
if (txn->xid != xid)
{
@@ -807,8 +814,9 @@ SnapBuildDistributeSnapshotAndInval(SnapBuild *builder, XLogRecPtr lsn, Transact
{
Assert(msgs != NULL);
- ReorderBufferAddInvalidations(builder->reorder, txn->xid, lsn,
- ninvalidations, msgs);
+ ReorderBufferAddDistributedInvalidations(builder->reorder,
+ txn->xid, lsn,
+ ninvalidations, msgs);
}
}
}
diff --git a/src/include/replication/reorderbuffer.h b/src/include/replication/reorderbuffer.h
index 24e88c409ba..fa0745552f8 100644
--- a/src/include/replication/reorderbuffer.h
+++ b/src/include/replication/reorderbuffer.h
@@ -176,6 +176,7 @@ typedef struct ReorderBufferChange
#define RBTXN_SENT_PREPARE 0x0200
#define RBTXN_IS_COMMITTED 0x0400
#define RBTXN_IS_ABORTED 0x0800
+#define RBTXN_DISTR_INVAL_OVERFLOWED 0x1000
#define RBTXN_PREPARE_STATUS_MASK (RBTXN_IS_PREPARED | RBTXN_SKIPPED_PREPARE | RBTXN_SENT_PREPARE)
@@ -265,6 +266,12 @@ typedef struct ReorderBufferChange
((txn)->txn_flags & RBTXN_SKIPPED_PREPARE) != 0 \
)
+/* Is the array of distributed inval messages overflowed? */
+#define rbtxn_distr_inval_overflowed(txn) \
+( \
+ ((txn)->txn_flags & RBTXN_DISTR_INVAL_OVERFLOWED) != 0 \
+)
+
/* Is this a top-level transaction? */
#define rbtxn_is_toptxn(txn) \
( \
@@ -422,6 +429,12 @@ typedef struct ReorderBufferTXN
uint32 ninvalidations;
SharedInvalidationMessage *invalidations;
+ /*
+ * Stores cache invalidation messages distributed by other transactions.
+ */
+ uint32 ninvalidations_distributed;
+ SharedInvalidationMessage *invalidations_distributed;
+
/* ---
* Position in one of two lists:
* * list of subtransactions if we are *known* to be subxact
@@ -738,6 +751,9 @@ extern void ReorderBufferAddNewTupleCids(ReorderBuffer *rb, TransactionId xid,
CommandId cmin, CommandId cmax, CommandId combocid);
extern void ReorderBufferAddInvalidations(ReorderBuffer *rb, TransactionId xid, XLogRecPtr lsn,
Size nmsgs, SharedInvalidationMessage *msgs);
+extern void ReorderBufferAddDistributedInvalidations(ReorderBuffer *rb, TransactionId xid,
+ XLogRecPtr lsn, Size nmsgs,
+ SharedInvalidationMessage *msgs);
extern void ReorderBufferImmediateInvalidation(ReorderBuffer *rb, uint32 ninvalidations,
SharedInvalidationMessage *invalidations);
extern void ReorderBufferProcessXid(ReorderBuffer *rb, TransactionId xid, XLogRecPtr lsn);
--
2.47.1
[application/octet-stream] v12-0002-Make-MAX_DISTR_INVAL_MSG_PER_TXN-lower-in-case-o.patch (1.2K, ../../OSCPR01MB1496608B31715BB094C194B70F56EA@OSCPR01MB14966.jpnprd01.prod.outlook.com/3-v12-0002-Make-MAX_DISTR_INVAL_MSG_PER_TXN-lower-in-case-o.patch)
download | inline diff:
From 8467e16ade6e609b8e6900d936e80f21bb9688fc Mon Sep 17 00:00:00 2001
From: Hayato Kuroda <kuroda.hayato@fujitsu.com>
Date: Fri, 6 Jun 2025 14:50:23 +0900
Subject: [PATCH v12 2/2] Make MAX_DISTR_INVAL_MSG_PER_TXN lower in case of
debug build
---
src/backend/replication/logical/reorderbuffer.c | 7 +++++++
1 file changed, 7 insertions(+)
diff --git a/src/backend/replication/logical/reorderbuffer.c b/src/backend/replication/logical/reorderbuffer.c
index c4299c76fb1..7720ae1125c 100644
--- a/src/backend/replication/logical/reorderbuffer.c
+++ b/src/backend/replication/logical/reorderbuffer.c
@@ -121,9 +121,16 @@
* messages reach this threshold, the transaction is marked as
* RBTXN_DISTR_INVAL_OVERFLOWED to invalidate the complete cache as we have lost
* some inval messages and hence don't know what needs to be invalidated.
+ *
+ * In a debugging build, use a small threshold so that we get some test
+ * coverage of both code paths.
*/
+#ifdef USE_ASSERT_CHECKING
+#define MAX_DISTR_INVAL_MSG_PER_TXN 3
+#else
#define MAX_DISTR_INVAL_MSG_PER_TXN \
((8 * 1024 * 1024) / sizeof(SharedInvalidationMessage))
+#endif
/* entry for a hash table we use to map from xid to our transaction state */
typedef struct ReorderBufferTXNByIdEnt
--
2.47.1
^ permalink raw reply [nested|flat] 94+ messages in thread
* Re: Logical replication 'invalid memory alloc request size 1585837200' after upgrading to 17.5
@ 2025-06-06 17:20 Masahiko Sawada <sawada.mshk@gmail.com>
parent: Amit Kapila <amit.kapila16@gmail.com>
0 siblings, 2 replies; 94+ messages in thread
From: Masahiko Sawada @ 2025-06-06 17:20 UTC (permalink / raw)
To: Amit Kapila <amit.kapila16@gmail.com>; +Cc: Hayato Kuroda (Fujitsu) <kuroda.hayato@fujitsu.com>; Duncan Sands <duncan.sands@deepbluecap.com>; pgsql-bugs@lists.postgresql.org <pgsql-bugs@lists.postgresql.org>; vignesh C <vignesh21@gmail.com>
On Thu, Jun 5, 2025 at 8:43 PM Amit Kapila <amit.kapila16@gmail.com> wrote:
>
> On Thu, Jun 5, 2025 at 11:14 PM Masahiko Sawada <sawada.mshk@gmail.com> wrote:
> >
> > On Wed, Jun 4, 2025 at 11:36 PM Hayato Kuroda (Fujitsu)
> > <kuroda.hayato@fujitsu.com> wrote:
> > >
> > > Dear Sawada-san,
> > >
> > > > Alternative idea is to lower the constant value when using an
> > > > assertion build. That way, we don't need to rely on injection points
> > > > being enabled.
> > >
> > > Hmm, possible but I prefer current one. Two concerns:
> > >
> > > 1.
> > > USE_ASSERT_CHECKING has not been used to change the value yet. The main usage is
> > > to call debug functions in debug build.
> >
> > I think we have a similar precedent such as MT_NRELS_HASH to improve
> > the test coverages.
> >
> > > 2.
> > > If we add tests which is usable only for debug build, it must be run only when it
> > > is enabled. IIUC such test does not exist yet.
> >
> > I think we need to test cases not to check if we reach a specific code
> > point but to check if we can get the correct results even if we've
> > executed various code paths. As for this bug, it is better to check
> > that it works properly in a variety of cases. That way, we can check
> > overflow cases and non-overflow cases also in test cases added in the
> > future, improving the test coverage more.
> >
>
> This makes sense, but we should see whether some existing tests cover
> this code path after lowering the limit in the overflow code path. One
> minor point to consider is that at the time, the value MT_NRELS_HASH
> was used to cover cases in a debug build, but we didn't have the
> injection_point framework.
True.
After thinking about it more, perhaps my proposal would not be a good
idea for this case. I think that the cases where we selectively
invalidate caches is more complex and error-prone than the cases where
we invalidate a complete cache. If we invalidated all caches after
decoding each transaction, we wouldn't have had the original data-loss
issue. Having a lower MAX_DISTR_INVAL_MSG_PER_TXN value when using an
assertio build means that we're going to test the cases using a
simpler invalidation mechanism while productions systems, which has a
higher MAX_DISTR_INVAL_MSG_PER_TXN value, would end up executing
complex cases, which is not great. What do you think?
BTW, as for a new test case, it might be worth having a case I
mentioned before[1]:
1) S1: CREATE TABLE d (data text not null);
2) S1: INSERT INTO d VALUES ('d1');
3) S2: BEGIN; INSERT INTO d VALUES ('d2');
4) S3: BEGIN; INSERT INTO d VALUES ('d3');
5) S1: ALTER PUBLICATION pb ADD TABLE d;
6) S2: INSERT INTO d VALUES ('d4');
7) S2: COMMIT;
8) S3: COMMIT;
9) S2: INSERT INTO d VALUES('d5');
10) S1: INSERT INTO d VALUES ('d6');
With this case, we can test if we need to execute the distributed
invalidations as well in the non-error path in
ReorderBufferProcessTXN().
>
> BTW, I noticed that you removed the following comments in your suggestions:
> /*
> * Stores cache invalidation messages distributed by other transactions.
> - *
> - * It is acceptable to skip invalidations received from concurrent
> - * transactions during ReorderBufferForget and ReorderBufferInvalidate,
> - * because the transaction being discarded wouldn't have loaded any shared
>
> IIUC, you only mentioned having some comments like this for ease of
> understanding, and now you are suggesting to remove those.
I forgot to mention the reason. I thought we need either a
comprehensive comment in a place about in which case we need to
execute both the current transaction's inval messages and the
distributed inval messages and in which case we need to execute only
inval messages in the current transaction or to put comments where we
need. The v11 added the comprehensive comment to the declaration of
ninvalidations_distributed and invalidations_distributed in
ReoderBufferTXN, but I'm not sure that was the right place to have
such a comment as it's beyond the description of these fields. So in
my suggestion, I tried to clarify that we execute only the inval
message in the current transaction in ReorderBufferForget() and
ReorderBufferAbort() as they seems to already have a enough comment
for the reason why we need to execute the inval message there.
Regards,
[1] https://www.postgresql.org/message-id/CAD21AoCeM2nni1P7Z6KXzLM%3D6zCdShC82sOvuvu0_hBuJkm9Qw%40mail.g...
--
Masahiko Sawada
Amazon Web Services: https://aws.amazon.com
^ permalink raw reply [nested|flat] 94+ messages in thread
* Re: Logical replication 'invalid memory alloc request size 1585837200' after upgrading to 17.5
@ 2025-06-06 17:23 Masahiko Sawada <sawada.mshk@gmail.com>
parent: Amit Kapila <amit.kapila16@gmail.com>
0 siblings, 0 replies; 94+ messages in thread
From: Masahiko Sawada @ 2025-06-06 17:23 UTC (permalink / raw)
To: Amit Kapila <amit.kapila16@gmail.com>; +Cc: Hayato Kuroda (Fujitsu) <kuroda.hayato@fujitsu.com>; vignesh C <vignesh21@gmail.com>; Duncan Sands <duncan.sands@deepbluecap.com>; pgsql-bugs@lists.postgresql.org <pgsql-bugs@lists.postgresql.org>
On Thu, Jun 5, 2025 at 8:22 PM Amit Kapila <amit.kapila16@gmail.com> wrote:
>
> On Fri, Jun 6, 2025 at 12:51 AM Masahiko Sawada <sawada.mshk@gmail.com> wrote:
> >
> > On Thu, Jun 5, 2025 at 4:07 AM Hayato Kuroda (Fujitsu)
> > <kuroda.hayato@fujitsu.com> wrote:
> > >
> > > Dear Amit,
> > >
> > > > > ---
> > > > > I'd like to make it clear again which case we need to execute
> > > > > txn->invalidations as well as txn->invalidations_distributed (like in
> > > > > ReorderBufferProcessTXN()) and which case we need to execute only
> > > > > txn->invalidations (like in ReorderBufferForget() and
> > > > > ReorderBufferAbort()). I think it might be worth putting some comments
> > > > > about overall strategy somewhere.
> > > > >
> > > > > ---
> > > > > BTW for back branches, a simple fix without ABI breakage would be to
> > > > > introduce the RBTXN_INVAL_OVERFLOWED flag to limit the size of
> > > > > txn->invalidations. That is, we accumulate inval messages both coming
> > > > > from the current transaction and distributed by other transactions but
> > > > > once the size reaches the threshold we invalidate all caches. Is it
> > > > > worth considering for back branches?
> > > > >
> > > >
> > > > It should work and is worth considering. The main concern would be
> > > > that it will hit sooner than we expect in the field, seeing the recent
> > > > reports. So, such a change has the potential to degrade the
> > > > performance. I feel that the number of people impacted due to
> > > > performance would be more than the number of people impacted due to
> > > > such an ABI change (adding the new members at the end of
> > > > ReorderBufferTXN). However, if we think we want to go safe w.r.t
> > > > extensions that can rely on the sizeof ReorderBufferTXN then your
> > > > proposal makes sense.
> > >
> > > While considering the approach, I found a doubtful point. Consider the below
> > > workload:
> > >
> > > 0. S1: CREATE TABLE d(data text not null);
> > > 1. S1: BEGIN;
> > > 2. S1: INSERT INTO d VALUES ('d1')
> > > 3. S2: BEGIN;
> > > 4. S2: INSERT INTO d VALUES ('d2')
> > > 5. S1: ALTER PUBLICATION pb ADD TABLE d;
> > > 6. S1: ... lots of DDLs so overflow happens
> > > 7. S1: COMMIT;
> > > 8. S2: INSERT INTO d VALUES ('d3');
> > > 9. S2: COMMIT;
> > > 10. S2: INSERT INTO d VALUES ('d4');
> > >
> > > In this case, the inval message generated by step 5 is discarded at step 6. No
> > > invalidation messages are distributed in the SnapBuildDistributeSnapshotAndInval().
> > > While decoding S2, relcache cannot be discarded and tuples d3 and d4 won't be
> > > replicated. Do you think this can happen?
> >
> > I think that once the S1's inval messages got overflowed, we should
> > mark other transactions as overflowed instead of distributing inval
> > messages.
> >
>
> Yeah, this should work, but are you still advocating that we go with
> this approach (marking txn->invalidations also as overflowed) for
> backbranches? In the previous email, you seemed to agree with the
> performance impact due to DDLs, so it is not clear which approach you
> prefer.
No, I just wanted to make it clear that this idea is possible. But I
agree to use the idea of having both invalidations_distributed and
ninvalidations_distributed in ReorderBufferTXN in all branches.
Regards,
--
Masahiko Sawada
Amazon Web Services: https://aws.amazon.com
^ permalink raw reply [nested|flat] 94+ messages in thread
* Re: Logical replication 'invalid memory alloc request size 1585837200' after upgrading to 17.5
@ 2025-06-07 06:03 Amit Kapila <amit.kapila16@gmail.com>
parent: Masahiko Sawada <sawada.mshk@gmail.com>
1 sibling, 0 replies; 94+ messages in thread
From: Amit Kapila @ 2025-06-07 06:03 UTC (permalink / raw)
To: Masahiko Sawada <sawada.mshk@gmail.com>; +Cc: Hayato Kuroda (Fujitsu) <kuroda.hayato@fujitsu.com>; Duncan Sands <duncan.sands@deepbluecap.com>; pgsql-bugs@lists.postgresql.org <pgsql-bugs@lists.postgresql.org>; vignesh C <vignesh21@gmail.com>
On Fri, Jun 6, 2025 at 10:51 PM Masahiko Sawada <sawada.mshk@gmail.com> wrote:
>
> After thinking about it more, perhaps my proposal would not be a good
> idea for this case. I think that the cases where we selectively
> invalidate caches is more complex and error-prone than the cases where
> we invalidate a complete cache. If we invalidated all caches after
> decoding each transaction, we wouldn't have had the original data-loss
> issue. Having a lower MAX_DISTR_INVAL_MSG_PER_TXN value when using an
> assertio build means that we're going to test the cases using a
> simpler invalidation mechanism while productions systems, which has a
> higher MAX_DISTR_INVAL_MSG_PER_TXN value, would end up executing
> complex cases, which is not great. What do you think?
>
Your reasoning makes sense to me. The other thing is that it would be
better if we don't add more cases to rely on debug build for testing.
Going forward, it can become difficult to decide which cases are good
to test only in debug mode.
> BTW, as for a new test case, it might be worth having a case I
> mentioned before[1]:
>
> 1) S1: CREATE TABLE d (data text not null);
> 2) S1: INSERT INTO d VALUES ('d1');
> 3) S2: BEGIN; INSERT INTO d VALUES ('d2');
> 4) S3: BEGIN; INSERT INTO d VALUES ('d3');
> 5) S1: ALTER PUBLICATION pb ADD TABLE d;
> 6) S2: INSERT INTO d VALUES ('d4');
> 7) S2: COMMIT;
> 8) S3: COMMIT;
> 9) S2: INSERT INTO d VALUES('d5');
> 10) S1: INSERT INTO d VALUES ('d6');
>
> With this case, we can test if we need to execute the distributed
> invalidations as well in the non-error path in
> ReorderBufferProcessTXN().
>
+1.
> >
> > BTW, I noticed that you removed the following comments in your suggestions:
> > /*
> > * Stores cache invalidation messages distributed by other transactions.
> > - *
> > - * It is acceptable to skip invalidations received from concurrent
> > - * transactions during ReorderBufferForget and ReorderBufferInvalidate,
> > - * because the transaction being discarded wouldn't have loaded any shared
> >
> > IIUC, you only mentioned having some comments like this for ease of
> > understanding, and now you are suggesting to remove those.
>
> I forgot to mention the reason. I thought we need either a
> comprehensive comment in a place about in which case we need to
> execute both the current transaction's inval messages and the
> distributed inval messages and in which case we need to execute only
> inval messages in the current transaction or to put comments where we
> need. The v11 added the comprehensive comment to the declaration of
> ninvalidations_distributed and invalidations_distributed in
> ReoderBufferTXN, but I'm not sure that was the right place to have
> such a comment as it's beyond the description of these fields. So in
> my suggestion, I tried to clarify that we execute only the inval
> message in the current transaction in ReorderBufferForget() and
> ReorderBufferAbort() as they seems to already have a enough comment
> for the reason why we need to execute the inval message there.
>
Fair enough. I see one more case aka the call to
ReorderBufferExecuteInvalidations() in ReorderBufferFinishPrepared().
I think we should have executed the required invalidations at the end
of prepare only, so why do we need to execute in
ReorderBufferFinishPrepared? It might be kept as a general cleanup
call and if that is the case we might want to even adjust comments
there. What do you think?
--
With Regards,
Amit Kapila.
^ permalink raw reply [nested|flat] 94+ messages in thread
* Re: Logical replication 'invalid memory alloc request size 1585837200' after upgrading to 17.5
@ 2025-06-07 07:17 vignesh C <vignesh21@gmail.com>
parent: Masahiko Sawada <sawada.mshk@gmail.com>
1 sibling, 1 reply; 94+ messages in thread
From: vignesh C @ 2025-06-07 07:17 UTC (permalink / raw)
To: Masahiko Sawada <sawada.mshk@gmail.com>; +Cc: Amit Kapila <amit.kapila16@gmail.com>; Hayato Kuroda (Fujitsu) <kuroda.hayato@fujitsu.com>; Duncan Sands <duncan.sands@deepbluecap.com>; pgsql-bugs@lists.postgresql.org <pgsql-bugs@lists.postgresql.org>
On Fri, 6 Jun 2025 at 22:51, Masahiko Sawada <sawada.mshk@gmail.com> wrote:
>
> On Thu, Jun 5, 2025 at 8:43 PM Amit Kapila <amit.kapila16@gmail.com> wrote:
> >
> > On Thu, Jun 5, 2025 at 11:14 PM Masahiko Sawada <sawada.mshk@gmail.com> wrote:
> > >
> > > On Wed, Jun 4, 2025 at 11:36 PM Hayato Kuroda (Fujitsu)
> > > <kuroda.hayato@fujitsu.com> wrote:
> > > >
> > > > Dear Sawada-san,
> > > >
> > > > > Alternative idea is to lower the constant value when using an
> > > > > assertion build. That way, we don't need to rely on injection points
> > > > > being enabled.
> > > >
> > > > Hmm, possible but I prefer current one. Two concerns:
> > > >
> > > > 1.
> > > > USE_ASSERT_CHECKING has not been used to change the value yet. The main usage is
> > > > to call debug functions in debug build.
> > >
> > > I think we have a similar precedent such as MT_NRELS_HASH to improve
> > > the test coverages.
> > >
> > > > 2.
> > > > If we add tests which is usable only for debug build, it must be run only when it
> > > > is enabled. IIUC such test does not exist yet.
> > >
> > > I think we need to test cases not to check if we reach a specific code
> > > point but to check if we can get the correct results even if we've
> > > executed various code paths. As for this bug, it is better to check
> > > that it works properly in a variety of cases. That way, we can check
> > > overflow cases and non-overflow cases also in test cases added in the
> > > future, improving the test coverage more.
> > >
> >
> > This makes sense, but we should see whether some existing tests cover
> > this code path after lowering the limit in the overflow code path. One
> > minor point to consider is that at the time, the value MT_NRELS_HASH
> > was used to cover cases in a debug build, but we didn't have the
> > injection_point framework.
>
> True.
>
> After thinking about it more, perhaps my proposal would not be a good
> idea for this case. I think that the cases where we selectively
> invalidate caches is more complex and error-prone than the cases where
> we invalidate a complete cache. If we invalidated all caches after
> decoding each transaction, we wouldn't have had the original data-loss
> issue. Having a lower MAX_DISTR_INVAL_MSG_PER_TXN value when using an
> assertio build means that we're going to test the cases using a
> simpler invalidation mechanism while productions systems, which has a
> higher MAX_DISTR_INVAL_MSG_PER_TXN value, would end up executing
> complex cases, which is not great. What do you think?
>
> BTW, as for a new test case, it might be worth having a case I
> mentioned before[1]:
>
> 1) S1: CREATE TABLE d (data text not null);
> 2) S1: INSERT INTO d VALUES ('d1');
> 3) S2: BEGIN; INSERT INTO d VALUES ('d2');
> 4) S3: BEGIN; INSERT INTO d VALUES ('d3');
> 5) S1: ALTER PUBLICATION pb ADD TABLE d;
> 6) S2: INSERT INTO d VALUES ('d4');
> 7) S2: COMMIT;
> 8) S3: COMMIT;
> 9) S2: INSERT INTO d VALUES('d5');
> 10) S1: INSERT INTO d VALUES ('d6');
>
> With this case, we can test if we need to execute the distributed
> invalidations as well in the non-error path in
> ReorderBufferProcessTXN().
The attached v13 version patch has the change to include this test case.
Regards,
Vignesh
Attachments:
[application/octet-stream] v13-master-0001-Fix-exponential-memory-allocation-issue-i.patch (17.2K, ../../CALDaNm0DkWbr8eA3kAkEw+cZA-n=KPmFxZ_5bz_YDqhYQGk6iA@mail.gmail.com/2-v13-master-0001-Fix-exponential-memory-allocation-issue-i.patch)
download | inline diff:
From e4e905f2eb724dbe0484720f801195fefad0a4aa Mon Sep 17 00:00:00 2001
From: Vignesh <vignesh21@gmail.com>
Date: Fri, 30 May 2025 12:06:07 +0530
Subject: [PATCH v13-master] Fix exponential memory allocation issue in logical
decoding due to invalidation message distribution
Commit 4909b38af0 added logic to distribute invalidation messages from
one transaction to all concurrent in-progress transactions. However,
since the source of each invalidation message was not tracked, transactions
could redundantly re-distribute already propagated messages. This led to an
exponential increase in invalidation messages and, ultimately, memory
allocation failures.
To fix this, a new field is introduced to track distributed invalidation
messages separately. This ensures that only transactions in the pre-existing
list receive the invalidations, preventing redundant propagation.
The accumulated distributed invalidations are now consumed at the end of
transaction processing in `ReorderBufferProcessTXN()`.
---
.../expected/invalidation_distribution.out | 26 ++-
.../specs/invalidation_distribution.spec | 21 ++
.../replication/logical/reorderbuffer.c | 196 +++++++++++++++---
src/backend/replication/logical/snapbuild.c | 12 +-
src/include/replication/reorderbuffer.h | 16 ++
5 files changed, 235 insertions(+), 36 deletions(-)
diff --git a/contrib/test_decoding/expected/invalidation_distribution.out b/contrib/test_decoding/expected/invalidation_distribution.out
index ad0a944cbf3..9fe87e2cda9 100644
--- a/contrib/test_decoding/expected/invalidation_distribution.out
+++ b/contrib/test_decoding/expected/invalidation_distribution.out
@@ -1,4 +1,4 @@
-Parsed test spec with 2 sessions
+Parsed test spec with 5 sessions
starting permutation: s1_insert_tbl1 s1_begin s1_insert_tbl1 s2_alter_pub_add_tbl s1_commit s1_insert_tbl1 s2_get_binary_changes
step s1_insert_tbl1: INSERT INTO tbl1 (val1, val2) VALUES (1, 1);
@@ -18,3 +18,27 @@ count
stop
(1 row)
+
+starting permutation: s3i1 s4b s4i1 s5b s5i1 s3a s4i2 s4c s5c s4i3 s3i2 s3_get_binary_changes
+step s3i1: INSERT INTO tbl1 (val1, val2) VALUES (1, 1);
+step s4b: BEGIN;
+step s4i1: INSERT INTO tbl1 (val1, val2) VALUES (2, 2);
+step s5b: BEGIN;
+step s5i1: INSERT INTO tbl1 (val1, val2) VALUES (3, 3);
+step s3a: ALTER PUBLICATION pub ADD TABLE tbl1;
+step s4i2: INSERT INTO tbl1 (val1, val2) VALUES (4, 4);
+step s4c: COMMIT;
+step s5c: COMMIT;
+step s4i3: INSERT INTO tbl1 (val1, val2) VALUES (5, 5);
+step s3i2: INSERT INTO tbl1 (val1, val2) VALUES (6, 6);
+step s3_get_binary_changes: SELECT count(data) FROM pg_logical_slot_get_binary_changes('isolation_slot', NULL, NULL, 'proto_version', '4', 'publication_names', 'pub') WHERE get_byte(data, 0) = 73;
+count
+-----
+ 3
+(1 row)
+
+?column?
+--------
+stop
+(1 row)
+
diff --git a/contrib/test_decoding/specs/invalidation_distribution.spec b/contrib/test_decoding/specs/invalidation_distribution.spec
index decbed627e3..1a90453f1ea 100644
--- a/contrib/test_decoding/specs/invalidation_distribution.spec
+++ b/contrib/test_decoding/specs/invalidation_distribution.spec
@@ -28,5 +28,26 @@ setup { SET synchronous_commit=on; }
step "s2_alter_pub_add_tbl" { ALTER PUBLICATION pub ADD TABLE tbl1; }
step "s2_get_binary_changes" { SELECT count(data) FROM pg_logical_slot_get_binary_changes('isolation_slot', NULL, NULL, 'proto_version', '4', 'publication_names', 'pub') WHERE get_byte(data, 0) = 73; }
+session "s3"
+step "s3i1" { INSERT INTO tbl1 (val1, val2) VALUES (1, 1);}
+step "s3a" { ALTER PUBLICATION pub ADD TABLE tbl1; }
+step "s3i2" { INSERT INTO tbl1 (val1, val2) VALUES (6, 6); }
+step "s3_get_binary_changes" { SELECT count(data) FROM pg_logical_slot_get_binary_changes('isolation_slot', NULL, NULL, 'proto_version', '4', 'publication_names', 'pub') WHERE get_byte(data, 0) = 73; }
+
+session "s4"
+step "s4b" { BEGIN; }
+step "s4i1" { INSERT INTO tbl1 (val1, val2) VALUES (2, 2);}
+step "s4i2" { INSERT INTO tbl1 (val1, val2) VALUES (4, 4); }
+step "s4c" { COMMIT; }
+step "s4i3" { INSERT INTO tbl1 (val1, val2) VALUES (5, 5); }
+
+session "s5"
+step "s5b" { BEGIN; }
+step "s5i1" { INSERT INTO tbl1 (val1, val2) VALUES (3, 3); }
+step "s5c" { COMMIT; }
+
# Expect to get one insert change. LOGICAL_REP_MSG_INSERT = 'I'
permutation "s1_insert_tbl1" "s1_begin" "s1_insert_tbl1" "s2_alter_pub_add_tbl" "s1_commit" "s1_insert_tbl1" "s2_get_binary_changes"
+
+# Expect to get three insert changes with LOGICAL_REP_MSG_INSERT = 'I' that happened after the table tbl1 is added to the publication.
+permutation "s3i1" "s4b" "s4i1" "s5b" "s5i1" "s3a" "s4i2" "s4c" "s5c" "s4i3" "s3i2" "s3_get_binary_changes"
diff --git a/src/backend/replication/logical/reorderbuffer.c b/src/backend/replication/logical/reorderbuffer.c
index 67655111875..c4299c76fb1 100644
--- a/src/backend/replication/logical/reorderbuffer.c
+++ b/src/backend/replication/logical/reorderbuffer.c
@@ -109,10 +109,22 @@
#include "storage/procarray.h"
#include "storage/sinval.h"
#include "utils/builtins.h"
+#include "utils/inval.h"
#include "utils/memutils.h"
#include "utils/rel.h"
#include "utils/relfilenumbermap.h"
+/*
+ * Each transaction has an 8MB limit for invalidation messages distributed from
+ * other transactions. This limit is set considering scenarios with many
+ * concurrent logical decoding operations. When the distributed invalidation
+ * messages reach this threshold, the transaction is marked as
+ * RBTXN_DISTR_INVAL_OVERFLOWED to invalidate the complete cache as we have lost
+ * some inval messages and hence don't know what needs to be invalidated.
+ */
+#define MAX_DISTR_INVAL_MSG_PER_TXN \
+ ((8 * 1024 * 1024) / sizeof(SharedInvalidationMessage))
+
/* entry for a hash table we use to map from xid to our transaction state */
typedef struct ReorderBufferTXNByIdEnt
{
@@ -472,6 +484,12 @@ ReorderBufferFreeTXN(ReorderBuffer *rb, ReorderBufferTXN *txn)
txn->invalidations = NULL;
}
+ if (txn->invalidations_distributed)
+ {
+ pfree(txn->invalidations_distributed);
+ txn->invalidations_distributed = NULL;
+ }
+
/* Reset the toast hash */
ReorderBufferToastReset(rb, txn);
@@ -2661,7 +2679,17 @@ ReorderBufferProcessTXN(ReorderBuffer *rb, ReorderBufferTXN *txn,
AbortCurrentTransaction();
/* make sure there's no cache pollution */
- ReorderBufferExecuteInvalidations(txn->ninvalidations, txn->invalidations);
+ if (rbtxn_distr_inval_overflowed(txn))
+ {
+ Assert(txn->ninvalidations_distributed == 0);
+ InvalidateSystemCaches();
+ }
+ else
+ {
+ ReorderBufferExecuteInvalidations(txn->ninvalidations, txn->invalidations);
+ ReorderBufferExecuteInvalidations(txn->ninvalidations_distributed,
+ txn->invalidations_distributed);
+ }
if (using_subtxn)
RollbackAndReleaseCurrentSubTransaction();
@@ -2710,8 +2738,17 @@ ReorderBufferProcessTXN(ReorderBuffer *rb, ReorderBufferTXN *txn,
AbortCurrentTransaction();
/* make sure there's no cache pollution */
- ReorderBufferExecuteInvalidations(txn->ninvalidations,
- txn->invalidations);
+ if (rbtxn_distr_inval_overflowed(txn))
+ {
+ Assert(txn->ninvalidations_distributed == 0);
+ InvalidateSystemCaches();
+ }
+ else
+ {
+ ReorderBufferExecuteInvalidations(txn->ninvalidations, txn->invalidations);
+ ReorderBufferExecuteInvalidations(txn->ninvalidations_distributed,
+ txn->invalidations_distributed);
+ }
if (using_subtxn)
RollbackAndReleaseCurrentSubTransaction();
@@ -3060,7 +3097,8 @@ ReorderBufferAbort(ReorderBuffer *rb, TransactionId xid, XLogRecPtr lsn,
* We might have decoded changes for this transaction that could load
* the cache as per the current transaction's view (consider DDL's
* happened in this transaction). We don't want the decoding of future
- * transactions to use those cache entries so execute invalidations.
+ * transactions to use those cache entries so execute only the inval
+ * messages in this transaction.
*/
if (txn->ninvalidations > 0)
ReorderBufferImmediateInvalidation(rb, txn->ninvalidations,
@@ -3147,9 +3185,10 @@ ReorderBufferForget(ReorderBuffer *rb, TransactionId xid, XLogRecPtr lsn)
txn->final_lsn = lsn;
/*
- * Process cache invalidation messages if there are any. Even if we're not
- * interested in the transaction's contents, it could have manipulated the
- * catalog and we need to update the caches according to that.
+ * Process only cache invalidation messages in this transaction if there
+ * are any. Even if we're not interested in the transaction's contents, it
+ * could have manipulated the catalog and we need to update the caches
+ * according to that.
*/
if (txn->base_snapshot != NULL && txn->ninvalidations > 0)
ReorderBufferImmediateInvalidation(rb, txn->ninvalidations,
@@ -3421,6 +3460,57 @@ ReorderBufferAddNewTupleCids(ReorderBuffer *rb, TransactionId xid,
txn->ntuplecids++;
}
+/*
+ * Add new invalidation messages to the reorder buffer queue.
+ */
+static void
+ReorderBufferQueueInvalidations(ReorderBuffer *rb, TransactionId xid,
+ XLogRecPtr lsn, Size nmsgs,
+ SharedInvalidationMessage *msgs)
+{
+ ReorderBufferChange *change;
+
+ change = ReorderBufferAllocChange(rb);
+ change->action = REORDER_BUFFER_CHANGE_INVALIDATION;
+ change->data.inval.ninvalidations = nmsgs;
+ change->data.inval.invalidations = (SharedInvalidationMessage *)
+ palloc(sizeof(SharedInvalidationMessage) * nmsgs);
+ memcpy(change->data.inval.invalidations, msgs,
+ sizeof(SharedInvalidationMessage) * nmsgs);
+
+ ReorderBufferQueueChange(rb, xid, lsn, change, false);
+}
+
+/*
+ * A helper function for ReorderBufferAddInvalidations() and
+ * ReorderBufferAddDistributedInvalidations() to accumulate the invalidation
+ * messages to the **invals_out.
+ */
+static void
+ReorderBufferAccumulateInvalidations(SharedInvalidationMessage **invals_out,
+ uint32 *ninvals_out,
+ SharedInvalidationMessage *msgs_new,
+ Size nmsgs_new)
+{
+ if (*ninvals_out == 0)
+ {
+ *ninvals_out = nmsgs_new;
+ *invals_out = (SharedInvalidationMessage *)
+ palloc(sizeof(SharedInvalidationMessage) * nmsgs_new);
+ memcpy(*invals_out, msgs_new, sizeof(SharedInvalidationMessage) * nmsgs_new);
+ }
+ else
+ {
+ /* Enlarge the array of inval messages */
+ *invals_out = (SharedInvalidationMessage *)
+ repalloc(*invals_out, sizeof(SharedInvalidationMessage) *
+ (*ninvals_out + nmsgs_new));
+ memcpy(*invals_out + *ninvals_out, msgs_new,
+ nmsgs_new * sizeof(SharedInvalidationMessage));
+ *ninvals_out += nmsgs_new;
+ }
+}
+
/*
* Accumulate the invalidations for executing them later.
*
@@ -3441,7 +3531,6 @@ ReorderBufferAddInvalidations(ReorderBuffer *rb, TransactionId xid,
{
ReorderBufferTXN *txn;
MemoryContext oldcontext;
- ReorderBufferChange *change;
txn = ReorderBufferTXNByXid(rb, xid, true, NULL, lsn, true);
@@ -3456,35 +3545,76 @@ ReorderBufferAddInvalidations(ReorderBuffer *rb, TransactionId xid,
Assert(nmsgs > 0);
- /* Accumulate invalidations. */
- if (txn->ninvalidations == 0)
- {
- txn->ninvalidations = nmsgs;
- txn->invalidations = (SharedInvalidationMessage *)
- palloc(sizeof(SharedInvalidationMessage) * nmsgs);
- memcpy(txn->invalidations, msgs,
- sizeof(SharedInvalidationMessage) * nmsgs);
- }
- else
+ ReorderBufferAccumulateInvalidations(&txn->invalidations,
+ &txn->ninvalidations,
+ msgs, nmsgs);
+
+ ReorderBufferQueueInvalidations(rb, xid, lsn, nmsgs, msgs);
+
+ MemoryContextSwitchTo(oldcontext);
+}
+
+/*
+ * Accumulate the invalidations distributed by other committed transactions
+ * for executing them later.
+ *
+ * This function is similar to ReorderBufferAddInvalidations() but stores
+ * the given inval messages to the txn->invalidations_distributed with the
+ * overflow check.
+ *
+ * This needs to be called by committed transactions to distribute their
+ * inval messages to in-progress transactions.
+ */
+void
+ReorderBufferAddDistributedInvalidations(ReorderBuffer *rb, TransactionId xid,
+ XLogRecPtr lsn, Size nmsgs,
+ SharedInvalidationMessage *msgs)
+{
+ ReorderBufferTXN *txn;
+ MemoryContext oldcontext;
+
+ txn = ReorderBufferTXNByXid(rb, xid, true, NULL, lsn, true);
+
+ oldcontext = MemoryContextSwitchTo(rb->context);
+
+ /*
+ * Collect all the invalidations under the top transaction, if available,
+ * so that we can execute them all together. See comments
+ * ReorderBufferAddInvalidations.
+ */
+ txn = rbtxn_get_toptxn(txn);
+
+ Assert(nmsgs > 0);
+
+ if (!rbtxn_distr_inval_overflowed(txn))
{
- txn->invalidations = (SharedInvalidationMessage *)
- repalloc(txn->invalidations, sizeof(SharedInvalidationMessage) *
- (txn->ninvalidations + nmsgs));
+ /*
+ * Check the transaction has enough space for storing distributed
+ * invalidation messages.
+ */
+ if (txn->ninvalidations_distributed + nmsgs >= MAX_DISTR_INVAL_MSG_PER_TXN)
+ {
+ /*
+ * Mark the invalidation message as overflowed and free up the
+ * messages accumulated so far.
+ */
+ txn->txn_flags |= RBTXN_DISTR_INVAL_OVERFLOWED;
- memcpy(txn->invalidations + txn->ninvalidations, msgs,
- nmsgs * sizeof(SharedInvalidationMessage));
- txn->ninvalidations += nmsgs;
+ if (txn->invalidations_distributed)
+ {
+ pfree(txn->invalidations_distributed);
+ txn->invalidations_distributed = NULL;
+ txn->ninvalidations_distributed = 0;
+ }
+ }
+ else
+ ReorderBufferAccumulateInvalidations(&txn->invalidations_distributed,
+ &txn->ninvalidations_distributed,
+ msgs, nmsgs);
}
- change = ReorderBufferAllocChange(rb);
- change->action = REORDER_BUFFER_CHANGE_INVALIDATION;
- change->data.inval.ninvalidations = nmsgs;
- change->data.inval.invalidations = (SharedInvalidationMessage *)
- palloc(sizeof(SharedInvalidationMessage) * nmsgs);
- memcpy(change->data.inval.invalidations, msgs,
- sizeof(SharedInvalidationMessage) * nmsgs);
-
- ReorderBufferQueueChange(rb, xid, lsn, change, false);
+ /* Queue the invalidation messages into the transaction */
+ ReorderBufferQueueInvalidations(rb, xid, lsn, nmsgs, msgs);
MemoryContextSwitchTo(oldcontext);
}
diff --git a/src/backend/replication/logical/snapbuild.c b/src/backend/replication/logical/snapbuild.c
index 0d7bddbe4ed..adf18c397db 100644
--- a/src/backend/replication/logical/snapbuild.c
+++ b/src/backend/replication/logical/snapbuild.c
@@ -794,6 +794,13 @@ SnapBuildDistributeSnapshotAndInval(SnapBuild *builder, XLogRecPtr lsn, Transact
* contents built by the current transaction even after its decoding,
* which should have been invalidated due to concurrent catalog
* changing transaction.
+ *
+ * Distribute only the invalidation messages generated by the current
+ * committed transaction. Invalidation messages received from other
+ * transactions would have already been propagated to the relevant
+ * in-progress transactions. This transaction would have processed
+ * those invalidations, ensuring that subsequent transactions observe
+ * a consistent cache state.
*/
if (txn->xid != xid)
{
@@ -807,8 +814,9 @@ SnapBuildDistributeSnapshotAndInval(SnapBuild *builder, XLogRecPtr lsn, Transact
{
Assert(msgs != NULL);
- ReorderBufferAddInvalidations(builder->reorder, txn->xid, lsn,
- ninvalidations, msgs);
+ ReorderBufferAddDistributedInvalidations(builder->reorder,
+ txn->xid, lsn,
+ ninvalidations, msgs);
}
}
}
diff --git a/src/include/replication/reorderbuffer.h b/src/include/replication/reorderbuffer.h
index 24e88c409ba..fa0745552f8 100644
--- a/src/include/replication/reorderbuffer.h
+++ b/src/include/replication/reorderbuffer.h
@@ -176,6 +176,7 @@ typedef struct ReorderBufferChange
#define RBTXN_SENT_PREPARE 0x0200
#define RBTXN_IS_COMMITTED 0x0400
#define RBTXN_IS_ABORTED 0x0800
+#define RBTXN_DISTR_INVAL_OVERFLOWED 0x1000
#define RBTXN_PREPARE_STATUS_MASK (RBTXN_IS_PREPARED | RBTXN_SKIPPED_PREPARE | RBTXN_SENT_PREPARE)
@@ -265,6 +266,12 @@ typedef struct ReorderBufferChange
((txn)->txn_flags & RBTXN_SKIPPED_PREPARE) != 0 \
)
+/* Is the array of distributed inval messages overflowed? */
+#define rbtxn_distr_inval_overflowed(txn) \
+( \
+ ((txn)->txn_flags & RBTXN_DISTR_INVAL_OVERFLOWED) != 0 \
+)
+
/* Is this a top-level transaction? */
#define rbtxn_is_toptxn(txn) \
( \
@@ -422,6 +429,12 @@ typedef struct ReorderBufferTXN
uint32 ninvalidations;
SharedInvalidationMessage *invalidations;
+ /*
+ * Stores cache invalidation messages distributed by other transactions.
+ */
+ uint32 ninvalidations_distributed;
+ SharedInvalidationMessage *invalidations_distributed;
+
/* ---
* Position in one of two lists:
* * list of subtransactions if we are *known* to be subxact
@@ -738,6 +751,9 @@ extern void ReorderBufferAddNewTupleCids(ReorderBuffer *rb, TransactionId xid,
CommandId cmin, CommandId cmax, CommandId combocid);
extern void ReorderBufferAddInvalidations(ReorderBuffer *rb, TransactionId xid, XLogRecPtr lsn,
Size nmsgs, SharedInvalidationMessage *msgs);
+extern void ReorderBufferAddDistributedInvalidations(ReorderBuffer *rb, TransactionId xid,
+ XLogRecPtr lsn, Size nmsgs,
+ SharedInvalidationMessage *msgs);
extern void ReorderBufferImmediateInvalidation(ReorderBuffer *rb, uint32 ninvalidations,
SharedInvalidationMessage *invalidations);
extern void ReorderBufferProcessXid(ReorderBuffer *rb, TransactionId xid, XLogRecPtr lsn);
--
2.43.0
^ permalink raw reply [nested|flat] 94+ messages in thread
* Re: Logical replication 'invalid memory alloc request size 1585837200' after upgrading to 17.5
@ 2025-06-08 13:43 vignesh C <vignesh21@gmail.com>
parent: vignesh C <vignesh21@gmail.com>
0 siblings, 2 replies; 94+ messages in thread
From: vignesh C @ 2025-06-08 13:43 UTC (permalink / raw)
To: Masahiko Sawada <sawada.mshk@gmail.com>; +Cc: Amit Kapila <amit.kapila16@gmail.com>; Hayato Kuroda (Fujitsu) <kuroda.hayato@fujitsu.com>; Duncan Sands <duncan.sands@deepbluecap.com>; pgsql-bugs@lists.postgresql.org <pgsql-bugs@lists.postgresql.org>
On Sat, 7 Jun 2025 at 12:47, vignesh C <vignesh21@gmail.com> wrote:
>
> On Fri, 6 Jun 2025 at 22:51, Masahiko Sawada <sawada.mshk@gmail.com> wrote:
> >
> > On Thu, Jun 5, 2025 at 8:43 PM Amit Kapila <amit.kapila16@gmail.com> wrote:
> > >
> > > On Thu, Jun 5, 2025 at 11:14 PM Masahiko Sawada <sawada.mshk@gmail.com> wrote:
> > > >
> > > > On Wed, Jun 4, 2025 at 11:36 PM Hayato Kuroda (Fujitsu)
> > > > <kuroda.hayato@fujitsu.com> wrote:
> > > > >
> > > > > Dear Sawada-san,
> > > > >
> > > > > > Alternative idea is to lower the constant value when using an
> > > > > > assertion build. That way, we don't need to rely on injection points
> > > > > > being enabled.
> > > > >
> > > > > Hmm, possible but I prefer current one. Two concerns:
> > > > >
> > > > > 1.
> > > > > USE_ASSERT_CHECKING has not been used to change the value yet. The main usage is
> > > > > to call debug functions in debug build.
> > > >
> > > > I think we have a similar precedent such as MT_NRELS_HASH to improve
> > > > the test coverages.
> > > >
> > > > > 2.
> > > > > If we add tests which is usable only for debug build, it must be run only when it
> > > > > is enabled. IIUC such test does not exist yet.
> > > >
> > > > I think we need to test cases not to check if we reach a specific code
> > > > point but to check if we can get the correct results even if we've
> > > > executed various code paths. As for this bug, it is better to check
> > > > that it works properly in a variety of cases. That way, we can check
> > > > overflow cases and non-overflow cases also in test cases added in the
> > > > future, improving the test coverage more.
> > > >
> > >
> > > This makes sense, but we should see whether some existing tests cover
> > > this code path after lowering the limit in the overflow code path. One
> > > minor point to consider is that at the time, the value MT_NRELS_HASH
> > > was used to cover cases in a debug build, but we didn't have the
> > > injection_point framework.
> >
> > True.
> >
> > After thinking about it more, perhaps my proposal would not be a good
> > idea for this case. I think that the cases where we selectively
> > invalidate caches is more complex and error-prone than the cases where
> > we invalidate a complete cache. If we invalidated all caches after
> > decoding each transaction, we wouldn't have had the original data-loss
> > issue. Having a lower MAX_DISTR_INVAL_MSG_PER_TXN value when using an
> > assertio build means that we're going to test the cases using a
> > simpler invalidation mechanism while productions systems, which has a
> > higher MAX_DISTR_INVAL_MSG_PER_TXN value, would end up executing
> > complex cases, which is not great. What do you think?
> >
> > BTW, as for a new test case, it might be worth having a case I
> > mentioned before[1]:
> >
> > 1) S1: CREATE TABLE d (data text not null);
> > 2) S1: INSERT INTO d VALUES ('d1');
> > 3) S2: BEGIN; INSERT INTO d VALUES ('d2');
> > 4) S3: BEGIN; INSERT INTO d VALUES ('d3');
> > 5) S1: ALTER PUBLICATION pb ADD TABLE d;
> > 6) S2: INSERT INTO d VALUES ('d4');
> > 7) S2: COMMIT;
> > 8) S3: COMMIT;
> > 9) S2: INSERT INTO d VALUES('d5');
> > 10) S1: INSERT INTO d VALUES ('d6');
> >
> > With this case, we can test if we need to execute the distributed
> > invalidations as well in the non-error path in
> > ReorderBufferProcessTXN().
>
> The attached v13 version patch has the change to include this test case.
Attached are the patches, including those required for the back branches.
Regards,
Vignesh
Attachments:
[application/octet-stream] v13-master-0001-Fix-exponential-memory-allocation-issue-i.patch (17.2K, ../../CALDaNm0TaTPuza7Fa+DRMzL+mqK3+7RVEvFiRoDJbU2vkJESwg@mail.gmail.com/2-v13-master-0001-Fix-exponential-memory-allocation-issue-i.patch)
download | inline diff:
From e4e905f2eb724dbe0484720f801195fefad0a4aa Mon Sep 17 00:00:00 2001
From: Vignesh <vignesh21@gmail.com>
Date: Fri, 30 May 2025 12:06:07 +0530
Subject: [PATCH v13-master] Fix exponential memory allocation issue in logical
decoding due to invalidation message distribution
Commit 4909b38af0 added logic to distribute invalidation messages from
one transaction to all concurrent in-progress transactions. However,
since the source of each invalidation message was not tracked, transactions
could redundantly re-distribute already propagated messages. This led to an
exponential increase in invalidation messages and, ultimately, memory
allocation failures.
To fix this, a new field is introduced to track distributed invalidation
messages separately. This ensures that only transactions in the pre-existing
list receive the invalidations, preventing redundant propagation.
The accumulated distributed invalidations are now consumed at the end of
transaction processing in `ReorderBufferProcessTXN()`.
---
.../expected/invalidation_distribution.out | 26 ++-
.../specs/invalidation_distribution.spec | 21 ++
.../replication/logical/reorderbuffer.c | 196 +++++++++++++++---
src/backend/replication/logical/snapbuild.c | 12 +-
src/include/replication/reorderbuffer.h | 16 ++
5 files changed, 235 insertions(+), 36 deletions(-)
diff --git a/contrib/test_decoding/expected/invalidation_distribution.out b/contrib/test_decoding/expected/invalidation_distribution.out
index ad0a944cbf3..9fe87e2cda9 100644
--- a/contrib/test_decoding/expected/invalidation_distribution.out
+++ b/contrib/test_decoding/expected/invalidation_distribution.out
@@ -1,4 +1,4 @@
-Parsed test spec with 2 sessions
+Parsed test spec with 5 sessions
starting permutation: s1_insert_tbl1 s1_begin s1_insert_tbl1 s2_alter_pub_add_tbl s1_commit s1_insert_tbl1 s2_get_binary_changes
step s1_insert_tbl1: INSERT INTO tbl1 (val1, val2) VALUES (1, 1);
@@ -18,3 +18,27 @@ count
stop
(1 row)
+
+starting permutation: s3i1 s4b s4i1 s5b s5i1 s3a s4i2 s4c s5c s4i3 s3i2 s3_get_binary_changes
+step s3i1: INSERT INTO tbl1 (val1, val2) VALUES (1, 1);
+step s4b: BEGIN;
+step s4i1: INSERT INTO tbl1 (val1, val2) VALUES (2, 2);
+step s5b: BEGIN;
+step s5i1: INSERT INTO tbl1 (val1, val2) VALUES (3, 3);
+step s3a: ALTER PUBLICATION pub ADD TABLE tbl1;
+step s4i2: INSERT INTO tbl1 (val1, val2) VALUES (4, 4);
+step s4c: COMMIT;
+step s5c: COMMIT;
+step s4i3: INSERT INTO tbl1 (val1, val2) VALUES (5, 5);
+step s3i2: INSERT INTO tbl1 (val1, val2) VALUES (6, 6);
+step s3_get_binary_changes: SELECT count(data) FROM pg_logical_slot_get_binary_changes('isolation_slot', NULL, NULL, 'proto_version', '4', 'publication_names', 'pub') WHERE get_byte(data, 0) = 73;
+count
+-----
+ 3
+(1 row)
+
+?column?
+--------
+stop
+(1 row)
+
diff --git a/contrib/test_decoding/specs/invalidation_distribution.spec b/contrib/test_decoding/specs/invalidation_distribution.spec
index decbed627e3..1a90453f1ea 100644
--- a/contrib/test_decoding/specs/invalidation_distribution.spec
+++ b/contrib/test_decoding/specs/invalidation_distribution.spec
@@ -28,5 +28,26 @@ setup { SET synchronous_commit=on; }
step "s2_alter_pub_add_tbl" { ALTER PUBLICATION pub ADD TABLE tbl1; }
step "s2_get_binary_changes" { SELECT count(data) FROM pg_logical_slot_get_binary_changes('isolation_slot', NULL, NULL, 'proto_version', '4', 'publication_names', 'pub') WHERE get_byte(data, 0) = 73; }
+session "s3"
+step "s3i1" { INSERT INTO tbl1 (val1, val2) VALUES (1, 1);}
+step "s3a" { ALTER PUBLICATION pub ADD TABLE tbl1; }
+step "s3i2" { INSERT INTO tbl1 (val1, val2) VALUES (6, 6); }
+step "s3_get_binary_changes" { SELECT count(data) FROM pg_logical_slot_get_binary_changes('isolation_slot', NULL, NULL, 'proto_version', '4', 'publication_names', 'pub') WHERE get_byte(data, 0) = 73; }
+
+session "s4"
+step "s4b" { BEGIN; }
+step "s4i1" { INSERT INTO tbl1 (val1, val2) VALUES (2, 2);}
+step "s4i2" { INSERT INTO tbl1 (val1, val2) VALUES (4, 4); }
+step "s4c" { COMMIT; }
+step "s4i3" { INSERT INTO tbl1 (val1, val2) VALUES (5, 5); }
+
+session "s5"
+step "s5b" { BEGIN; }
+step "s5i1" { INSERT INTO tbl1 (val1, val2) VALUES (3, 3); }
+step "s5c" { COMMIT; }
+
# Expect to get one insert change. LOGICAL_REP_MSG_INSERT = 'I'
permutation "s1_insert_tbl1" "s1_begin" "s1_insert_tbl1" "s2_alter_pub_add_tbl" "s1_commit" "s1_insert_tbl1" "s2_get_binary_changes"
+
+# Expect to get three insert changes with LOGICAL_REP_MSG_INSERT = 'I' that happened after the table tbl1 is added to the publication.
+permutation "s3i1" "s4b" "s4i1" "s5b" "s5i1" "s3a" "s4i2" "s4c" "s5c" "s4i3" "s3i2" "s3_get_binary_changes"
diff --git a/src/backend/replication/logical/reorderbuffer.c b/src/backend/replication/logical/reorderbuffer.c
index 67655111875..c4299c76fb1 100644
--- a/src/backend/replication/logical/reorderbuffer.c
+++ b/src/backend/replication/logical/reorderbuffer.c
@@ -109,10 +109,22 @@
#include "storage/procarray.h"
#include "storage/sinval.h"
#include "utils/builtins.h"
+#include "utils/inval.h"
#include "utils/memutils.h"
#include "utils/rel.h"
#include "utils/relfilenumbermap.h"
+/*
+ * Each transaction has an 8MB limit for invalidation messages distributed from
+ * other transactions. This limit is set considering scenarios with many
+ * concurrent logical decoding operations. When the distributed invalidation
+ * messages reach this threshold, the transaction is marked as
+ * RBTXN_DISTR_INVAL_OVERFLOWED to invalidate the complete cache as we have lost
+ * some inval messages and hence don't know what needs to be invalidated.
+ */
+#define MAX_DISTR_INVAL_MSG_PER_TXN \
+ ((8 * 1024 * 1024) / sizeof(SharedInvalidationMessage))
+
/* entry for a hash table we use to map from xid to our transaction state */
typedef struct ReorderBufferTXNByIdEnt
{
@@ -472,6 +484,12 @@ ReorderBufferFreeTXN(ReorderBuffer *rb, ReorderBufferTXN *txn)
txn->invalidations = NULL;
}
+ if (txn->invalidations_distributed)
+ {
+ pfree(txn->invalidations_distributed);
+ txn->invalidations_distributed = NULL;
+ }
+
/* Reset the toast hash */
ReorderBufferToastReset(rb, txn);
@@ -2661,7 +2679,17 @@ ReorderBufferProcessTXN(ReorderBuffer *rb, ReorderBufferTXN *txn,
AbortCurrentTransaction();
/* make sure there's no cache pollution */
- ReorderBufferExecuteInvalidations(txn->ninvalidations, txn->invalidations);
+ if (rbtxn_distr_inval_overflowed(txn))
+ {
+ Assert(txn->ninvalidations_distributed == 0);
+ InvalidateSystemCaches();
+ }
+ else
+ {
+ ReorderBufferExecuteInvalidations(txn->ninvalidations, txn->invalidations);
+ ReorderBufferExecuteInvalidations(txn->ninvalidations_distributed,
+ txn->invalidations_distributed);
+ }
if (using_subtxn)
RollbackAndReleaseCurrentSubTransaction();
@@ -2710,8 +2738,17 @@ ReorderBufferProcessTXN(ReorderBuffer *rb, ReorderBufferTXN *txn,
AbortCurrentTransaction();
/* make sure there's no cache pollution */
- ReorderBufferExecuteInvalidations(txn->ninvalidations,
- txn->invalidations);
+ if (rbtxn_distr_inval_overflowed(txn))
+ {
+ Assert(txn->ninvalidations_distributed == 0);
+ InvalidateSystemCaches();
+ }
+ else
+ {
+ ReorderBufferExecuteInvalidations(txn->ninvalidations, txn->invalidations);
+ ReorderBufferExecuteInvalidations(txn->ninvalidations_distributed,
+ txn->invalidations_distributed);
+ }
if (using_subtxn)
RollbackAndReleaseCurrentSubTransaction();
@@ -3060,7 +3097,8 @@ ReorderBufferAbort(ReorderBuffer *rb, TransactionId xid, XLogRecPtr lsn,
* We might have decoded changes for this transaction that could load
* the cache as per the current transaction's view (consider DDL's
* happened in this transaction). We don't want the decoding of future
- * transactions to use those cache entries so execute invalidations.
+ * transactions to use those cache entries so execute only the inval
+ * messages in this transaction.
*/
if (txn->ninvalidations > 0)
ReorderBufferImmediateInvalidation(rb, txn->ninvalidations,
@@ -3147,9 +3185,10 @@ ReorderBufferForget(ReorderBuffer *rb, TransactionId xid, XLogRecPtr lsn)
txn->final_lsn = lsn;
/*
- * Process cache invalidation messages if there are any. Even if we're not
- * interested in the transaction's contents, it could have manipulated the
- * catalog and we need to update the caches according to that.
+ * Process only cache invalidation messages in this transaction if there
+ * are any. Even if we're not interested in the transaction's contents, it
+ * could have manipulated the catalog and we need to update the caches
+ * according to that.
*/
if (txn->base_snapshot != NULL && txn->ninvalidations > 0)
ReorderBufferImmediateInvalidation(rb, txn->ninvalidations,
@@ -3421,6 +3460,57 @@ ReorderBufferAddNewTupleCids(ReorderBuffer *rb, TransactionId xid,
txn->ntuplecids++;
}
+/*
+ * Add new invalidation messages to the reorder buffer queue.
+ */
+static void
+ReorderBufferQueueInvalidations(ReorderBuffer *rb, TransactionId xid,
+ XLogRecPtr lsn, Size nmsgs,
+ SharedInvalidationMessage *msgs)
+{
+ ReorderBufferChange *change;
+
+ change = ReorderBufferAllocChange(rb);
+ change->action = REORDER_BUFFER_CHANGE_INVALIDATION;
+ change->data.inval.ninvalidations = nmsgs;
+ change->data.inval.invalidations = (SharedInvalidationMessage *)
+ palloc(sizeof(SharedInvalidationMessage) * nmsgs);
+ memcpy(change->data.inval.invalidations, msgs,
+ sizeof(SharedInvalidationMessage) * nmsgs);
+
+ ReorderBufferQueueChange(rb, xid, lsn, change, false);
+}
+
+/*
+ * A helper function for ReorderBufferAddInvalidations() and
+ * ReorderBufferAddDistributedInvalidations() to accumulate the invalidation
+ * messages to the **invals_out.
+ */
+static void
+ReorderBufferAccumulateInvalidations(SharedInvalidationMessage **invals_out,
+ uint32 *ninvals_out,
+ SharedInvalidationMessage *msgs_new,
+ Size nmsgs_new)
+{
+ if (*ninvals_out == 0)
+ {
+ *ninvals_out = nmsgs_new;
+ *invals_out = (SharedInvalidationMessage *)
+ palloc(sizeof(SharedInvalidationMessage) * nmsgs_new);
+ memcpy(*invals_out, msgs_new, sizeof(SharedInvalidationMessage) * nmsgs_new);
+ }
+ else
+ {
+ /* Enlarge the array of inval messages */
+ *invals_out = (SharedInvalidationMessage *)
+ repalloc(*invals_out, sizeof(SharedInvalidationMessage) *
+ (*ninvals_out + nmsgs_new));
+ memcpy(*invals_out + *ninvals_out, msgs_new,
+ nmsgs_new * sizeof(SharedInvalidationMessage));
+ *ninvals_out += nmsgs_new;
+ }
+}
+
/*
* Accumulate the invalidations for executing them later.
*
@@ -3441,7 +3531,6 @@ ReorderBufferAddInvalidations(ReorderBuffer *rb, TransactionId xid,
{
ReorderBufferTXN *txn;
MemoryContext oldcontext;
- ReorderBufferChange *change;
txn = ReorderBufferTXNByXid(rb, xid, true, NULL, lsn, true);
@@ -3456,35 +3545,76 @@ ReorderBufferAddInvalidations(ReorderBuffer *rb, TransactionId xid,
Assert(nmsgs > 0);
- /* Accumulate invalidations. */
- if (txn->ninvalidations == 0)
- {
- txn->ninvalidations = nmsgs;
- txn->invalidations = (SharedInvalidationMessage *)
- palloc(sizeof(SharedInvalidationMessage) * nmsgs);
- memcpy(txn->invalidations, msgs,
- sizeof(SharedInvalidationMessage) * nmsgs);
- }
- else
+ ReorderBufferAccumulateInvalidations(&txn->invalidations,
+ &txn->ninvalidations,
+ msgs, nmsgs);
+
+ ReorderBufferQueueInvalidations(rb, xid, lsn, nmsgs, msgs);
+
+ MemoryContextSwitchTo(oldcontext);
+}
+
+/*
+ * Accumulate the invalidations distributed by other committed transactions
+ * for executing them later.
+ *
+ * This function is similar to ReorderBufferAddInvalidations() but stores
+ * the given inval messages to the txn->invalidations_distributed with the
+ * overflow check.
+ *
+ * This needs to be called by committed transactions to distribute their
+ * inval messages to in-progress transactions.
+ */
+void
+ReorderBufferAddDistributedInvalidations(ReorderBuffer *rb, TransactionId xid,
+ XLogRecPtr lsn, Size nmsgs,
+ SharedInvalidationMessage *msgs)
+{
+ ReorderBufferTXN *txn;
+ MemoryContext oldcontext;
+
+ txn = ReorderBufferTXNByXid(rb, xid, true, NULL, lsn, true);
+
+ oldcontext = MemoryContextSwitchTo(rb->context);
+
+ /*
+ * Collect all the invalidations under the top transaction, if available,
+ * so that we can execute them all together. See comments
+ * ReorderBufferAddInvalidations.
+ */
+ txn = rbtxn_get_toptxn(txn);
+
+ Assert(nmsgs > 0);
+
+ if (!rbtxn_distr_inval_overflowed(txn))
{
- txn->invalidations = (SharedInvalidationMessage *)
- repalloc(txn->invalidations, sizeof(SharedInvalidationMessage) *
- (txn->ninvalidations + nmsgs));
+ /*
+ * Check the transaction has enough space for storing distributed
+ * invalidation messages.
+ */
+ if (txn->ninvalidations_distributed + nmsgs >= MAX_DISTR_INVAL_MSG_PER_TXN)
+ {
+ /*
+ * Mark the invalidation message as overflowed and free up the
+ * messages accumulated so far.
+ */
+ txn->txn_flags |= RBTXN_DISTR_INVAL_OVERFLOWED;
- memcpy(txn->invalidations + txn->ninvalidations, msgs,
- nmsgs * sizeof(SharedInvalidationMessage));
- txn->ninvalidations += nmsgs;
+ if (txn->invalidations_distributed)
+ {
+ pfree(txn->invalidations_distributed);
+ txn->invalidations_distributed = NULL;
+ txn->ninvalidations_distributed = 0;
+ }
+ }
+ else
+ ReorderBufferAccumulateInvalidations(&txn->invalidations_distributed,
+ &txn->ninvalidations_distributed,
+ msgs, nmsgs);
}
- change = ReorderBufferAllocChange(rb);
- change->action = REORDER_BUFFER_CHANGE_INVALIDATION;
- change->data.inval.ninvalidations = nmsgs;
- change->data.inval.invalidations = (SharedInvalidationMessage *)
- palloc(sizeof(SharedInvalidationMessage) * nmsgs);
- memcpy(change->data.inval.invalidations, msgs,
- sizeof(SharedInvalidationMessage) * nmsgs);
-
- ReorderBufferQueueChange(rb, xid, lsn, change, false);
+ /* Queue the invalidation messages into the transaction */
+ ReorderBufferQueueInvalidations(rb, xid, lsn, nmsgs, msgs);
MemoryContextSwitchTo(oldcontext);
}
diff --git a/src/backend/replication/logical/snapbuild.c b/src/backend/replication/logical/snapbuild.c
index 0d7bddbe4ed..adf18c397db 100644
--- a/src/backend/replication/logical/snapbuild.c
+++ b/src/backend/replication/logical/snapbuild.c
@@ -794,6 +794,13 @@ SnapBuildDistributeSnapshotAndInval(SnapBuild *builder, XLogRecPtr lsn, Transact
* contents built by the current transaction even after its decoding,
* which should have been invalidated due to concurrent catalog
* changing transaction.
+ *
+ * Distribute only the invalidation messages generated by the current
+ * committed transaction. Invalidation messages received from other
+ * transactions would have already been propagated to the relevant
+ * in-progress transactions. This transaction would have processed
+ * those invalidations, ensuring that subsequent transactions observe
+ * a consistent cache state.
*/
if (txn->xid != xid)
{
@@ -807,8 +814,9 @@ SnapBuildDistributeSnapshotAndInval(SnapBuild *builder, XLogRecPtr lsn, Transact
{
Assert(msgs != NULL);
- ReorderBufferAddInvalidations(builder->reorder, txn->xid, lsn,
- ninvalidations, msgs);
+ ReorderBufferAddDistributedInvalidations(builder->reorder,
+ txn->xid, lsn,
+ ninvalidations, msgs);
}
}
}
diff --git a/src/include/replication/reorderbuffer.h b/src/include/replication/reorderbuffer.h
index 24e88c409ba..fa0745552f8 100644
--- a/src/include/replication/reorderbuffer.h
+++ b/src/include/replication/reorderbuffer.h
@@ -176,6 +176,7 @@ typedef struct ReorderBufferChange
#define RBTXN_SENT_PREPARE 0x0200
#define RBTXN_IS_COMMITTED 0x0400
#define RBTXN_IS_ABORTED 0x0800
+#define RBTXN_DISTR_INVAL_OVERFLOWED 0x1000
#define RBTXN_PREPARE_STATUS_MASK (RBTXN_IS_PREPARED | RBTXN_SKIPPED_PREPARE | RBTXN_SENT_PREPARE)
@@ -265,6 +266,12 @@ typedef struct ReorderBufferChange
((txn)->txn_flags & RBTXN_SKIPPED_PREPARE) != 0 \
)
+/* Is the array of distributed inval messages overflowed? */
+#define rbtxn_distr_inval_overflowed(txn) \
+( \
+ ((txn)->txn_flags & RBTXN_DISTR_INVAL_OVERFLOWED) != 0 \
+)
+
/* Is this a top-level transaction? */
#define rbtxn_is_toptxn(txn) \
( \
@@ -422,6 +429,12 @@ typedef struct ReorderBufferTXN
uint32 ninvalidations;
SharedInvalidationMessage *invalidations;
+ /*
+ * Stores cache invalidation messages distributed by other transactions.
+ */
+ uint32 ninvalidations_distributed;
+ SharedInvalidationMessage *invalidations_distributed;
+
/* ---
* Position in one of two lists:
* * list of subtransactions if we are *known* to be subxact
@@ -738,6 +751,9 @@ extern void ReorderBufferAddNewTupleCids(ReorderBuffer *rb, TransactionId xid,
CommandId cmin, CommandId cmax, CommandId combocid);
extern void ReorderBufferAddInvalidations(ReorderBuffer *rb, TransactionId xid, XLogRecPtr lsn,
Size nmsgs, SharedInvalidationMessage *msgs);
+extern void ReorderBufferAddDistributedInvalidations(ReorderBuffer *rb, TransactionId xid,
+ XLogRecPtr lsn, Size nmsgs,
+ SharedInvalidationMessage *msgs);
extern void ReorderBufferImmediateInvalidation(ReorderBuffer *rb, uint32 ninvalidations,
SharedInvalidationMessage *invalidations);
extern void ReorderBufferProcessXid(ReorderBuffer *rb, TransactionId xid, XLogRecPtr lsn);
--
2.43.0
[application/octet-stream] v13-PG13-0001-Fix-exponential-memory-allocation-issue-in-.patch (16.7K, ../../CALDaNm0TaTPuza7Fa+DRMzL+mqK3+7RVEvFiRoDJbU2vkJESwg@mail.gmail.com/3-v13-PG13-0001-Fix-exponential-memory-allocation-issue-in-.patch)
download | inline diff:
From beca6c7681a8942805f24fbf8fa6e002b9719ebc Mon Sep 17 00:00:00 2001
From: Vignesh C <vignesh21@gmail.com>
Date: Sun, 8 Jun 2025 18:39:34 +0530
Subject: [PATCH v13-PG13] Fix exponential memory allocation issue in logical
decoding due to invalidation message distribution
Commit 4909b38af0 added logic to distribute invalidation messages from
one transaction to all concurrent in-progress transactions. However,
since the source of each invalidation message was not tracked, transactions
could redundantly re-distribute already propagated messages. This led to an
exponential increase in invalidation messages and, ultimately, memory
allocation failures.
To fix this, a new field is introduced to track distributed invalidation
messages separately. This ensures that only transactions in the pre-existing
list receive the invalidations, preventing redundant propagation.
The accumulated distributed invalidations are now consumed at the end of
transaction processing in `ReorderBufferProcessTXN()`.
---
.../expected/invalidation_distribution.out | 26 ++-
.../specs/invalidation_distribution.spec | 21 +++
.../replication/logical/reorderbuffer.c | 169 +++++++++++++++---
src/backend/replication/logical/snapbuild.c | 12 +-
src/include/replication/reorderbuffer.h | 22 ++-
5 files changed, 220 insertions(+), 30 deletions(-)
diff --git a/contrib/test_decoding/expected/invalidation_distribution.out b/contrib/test_decoding/expected/invalidation_distribution.out
index eb70eda9042..96e80688e39 100644
--- a/contrib/test_decoding/expected/invalidation_distribution.out
+++ b/contrib/test_decoding/expected/invalidation_distribution.out
@@ -1,4 +1,4 @@
-Parsed test spec with 2 sessions
+Parsed test spec with 5 sessions
starting permutation: s1_insert_tbl1 s1_begin s1_insert_tbl1 s2_alter_pub_add_tbl s1_commit s1_insert_tbl1 s2_get_binary_changes
step s1_insert_tbl1: INSERT INTO tbl1 (val1, val2) VALUES (1, 1);
@@ -18,3 +18,27 @@ count
stop
(1 row)
+
+starting permutation: s3i1 s4b s4i1 s5b s5i1 s3a s4i2 s4c s5c s4i3 s3i2 s3_get_binary_changes
+step s3i1: INSERT INTO tbl1 (val1, val2) VALUES (1, 1);
+step s4b: BEGIN;
+step s4i1: INSERT INTO tbl1 (val1, val2) VALUES (2, 2);
+step s5b: BEGIN;
+step s5i1: INSERT INTO tbl1 (val1, val2) VALUES (3, 3);
+step s3a: ALTER PUBLICATION pub ADD TABLE tbl1;
+step s4i2: INSERT INTO tbl1 (val1, val2) VALUES (4, 4);
+step s4c: COMMIT;
+step s5c: COMMIT;
+step s4i3: INSERT INTO tbl1 (val1, val2) VALUES (5, 5);
+step s3i2: INSERT INTO tbl1 (val1, val2) VALUES (6, 6);
+step s3_get_binary_changes: SELECT count(data) FROM pg_logical_slot_get_binary_changes('isolation_slot', NULL, NULL, 'proto_version', '1', 'publication_names', 'pub') WHERE get_byte(data, 0) = 73;
+count
+-----
+ 2
+(1 row)
+
+?column?
+--------
+stop
+(1 row)
+
diff --git a/contrib/test_decoding/specs/invalidation_distribution.spec b/contrib/test_decoding/specs/invalidation_distribution.spec
index ca051fc1e85..51688c52491 100644
--- a/contrib/test_decoding/specs/invalidation_distribution.spec
+++ b/contrib/test_decoding/specs/invalidation_distribution.spec
@@ -28,5 +28,26 @@ setup { SET synchronous_commit=on; }
step "s2_alter_pub_add_tbl" { ALTER PUBLICATION pub ADD TABLE tbl1; }
step "s2_get_binary_changes" { SELECT count(data) FROM pg_logical_slot_get_binary_changes('isolation_slot', NULL, NULL, 'proto_version', '1', 'publication_names', 'pub') WHERE get_byte(data, 0) = 73; }
+session "s3"
+step "s3i1" { INSERT INTO tbl1 (val1, val2) VALUES (1, 1);}
+step "s3a" { ALTER PUBLICATION pub ADD TABLE tbl1; }
+step "s3i2" { INSERT INTO tbl1 (val1, val2) VALUES (6, 6); }
+step "s3_get_binary_changes" { SELECT count(data) FROM pg_logical_slot_get_binary_changes('isolation_slot', NULL, NULL, 'proto_version', '1', 'publication_names', 'pub') WHERE get_byte(data, 0) = 73; }
+
+session "s4"
+step "s4b" { BEGIN; }
+step "s4i1" { INSERT INTO tbl1 (val1, val2) VALUES (2, 2);}
+step "s4i2" { INSERT INTO tbl1 (val1, val2) VALUES (4, 4); }
+step "s4c" { COMMIT; }
+step "s4i3" { INSERT INTO tbl1 (val1, val2) VALUES (5, 5); }
+
+session "s5"
+step "s5b" { BEGIN; }
+step "s5i1" { INSERT INTO tbl1 (val1, val2) VALUES (3, 3); }
+step "s5c" { COMMIT; }
+
# Expect to get one insert change. LOGICAL_REP_MSG_INSERT = 'I'
permutation "s1_insert_tbl1" "s1_begin" "s1_insert_tbl1" "s2_alter_pub_add_tbl" "s1_commit" "s1_insert_tbl1" "s2_get_binary_changes"
+
+# Expect to get three insert changes with LOGICAL_REP_MSG_INSERT = 'I' that happened after the table tbl1 is added to the publication.
+permutation "s3i1" "s4b" "s4i1" "s5b" "s5i1" "s3a" "s4i2" "s4c" "s5c" "s4i3" "s3i2" "s3_get_binary_changes"
diff --git a/src/backend/replication/logical/reorderbuffer.c b/src/backend/replication/logical/reorderbuffer.c
index fa9413fa2a0..e5fa1d62151 100644
--- a/src/backend/replication/logical/reorderbuffer.c
+++ b/src/backend/replication/logical/reorderbuffer.c
@@ -103,12 +103,24 @@
#include "storage/sinval.h"
#include "utils/builtins.h"
#include "utils/combocid.h"
+#include "utils/inval.h"
#include "utils/memdebug.h"
#include "utils/memutils.h"
#include "utils/rel.h"
#include "utils/relfilenodemap.h"
+/*
+ * Each transaction has an 8MB limit for invalidation messages distributed from
+ * other transactions. This limit is set considering scenarios with many
+ * concurrent logical decoding operations. When the distributed invalidation
+ * messages reach this threshold, the transaction is marked as
+ * RBTXN_DISTR_INVAL_OVERFLOWED to invalidate the complete cache as we have lost
+ * some inval messages and hence don't know what needs to be invalidated.
+ */
+#define MAX_DISTR_INVAL_MSG_PER_TXN \
+ ((8 * 1024 * 1024) / sizeof(SharedInvalidationMessage))
+
/* entry for a hash table we use to map from xid to our transaction state */
typedef struct ReorderBufferTXNByIdEnt
{
@@ -220,7 +232,8 @@ static void ReorderBufferIterTXNInit(ReorderBuffer *rb, ReorderBufferTXN *txn,
static ReorderBufferChange *ReorderBufferIterTXNNext(ReorderBuffer *rb, ReorderBufferIterTXNState *state);
static void ReorderBufferIterTXNFinish(ReorderBuffer *rb,
ReorderBufferIterTXNState *state);
-static void ReorderBufferExecuteInvalidations(ReorderBuffer *rb, ReorderBufferTXN *txn);
+static void ReorderBufferExecuteInvalidations(uint32 nmsgs,
+ SharedInvalidationMessage *msgs);
/*
* ---------------------------------------
@@ -406,6 +419,12 @@ ReorderBufferReturnTXN(ReorderBuffer *rb, ReorderBufferTXN *txn)
txn->invalidations = NULL;
}
+ if (txn->invalidations_distributed)
+ {
+ pfree(txn->invalidations_distributed);
+ txn->invalidations_distributed = NULL;
+ }
+
/* Reset the toast hash */
ReorderBufferToastReset(rb, txn);
@@ -1883,7 +1902,8 @@ ReorderBufferCommit(ReorderBuffer *rb, TransactionId xid,
* see new catalog contents, so execute all
* invalidations.
*/
- ReorderBufferExecuteInvalidations(rb, txn);
+ ReorderBufferExecuteInvalidations(txn->ninvalidations,
+ txn->invalidations);
}
break;
@@ -1921,7 +1941,17 @@ ReorderBufferCommit(ReorderBuffer *rb, TransactionId xid,
AbortCurrentTransaction();
/* make sure there's no cache pollution */
- ReorderBufferExecuteInvalidations(rb, txn);
+ if (rbtxn_distr_inval_overflowed(txn))
+ {
+ Assert(txn->ninvalidations_distributed == 0);
+ InvalidateSystemCaches();
+ }
+ else
+ {
+ ReorderBufferExecuteInvalidations(txn->ninvalidations, txn->invalidations);
+ ReorderBufferExecuteInvalidations(txn->ninvalidations_distributed,
+ txn->invalidations_distributed);
+ }
if (using_subtxn)
RollbackAndReleaseCurrentSubTransaction();
@@ -1947,7 +1977,17 @@ ReorderBufferCommit(ReorderBuffer *rb, TransactionId xid,
AbortCurrentTransaction();
/* make sure there's no cache pollution */
- ReorderBufferExecuteInvalidations(rb, txn);
+ if (rbtxn_distr_inval_overflowed(txn))
+ {
+ Assert(txn->ninvalidations_distributed == 0);
+ InvalidateSystemCaches();
+ }
+ else
+ {
+ ReorderBufferExecuteInvalidations(txn->ninvalidations, txn->invalidations);
+ ReorderBufferExecuteInvalidations(txn->ninvalidations_distributed,
+ txn->invalidations_distributed);
+ }
if (using_subtxn)
RollbackAndReleaseCurrentSubTransaction();
@@ -2060,9 +2100,10 @@ ReorderBufferForget(ReorderBuffer *rb, TransactionId xid, XLogRecPtr lsn)
txn->final_lsn = lsn;
/*
- * Process cache invalidation messages if there are any. Even if we're not
- * interested in the transaction's contents, it could have manipulated the
- * catalog and we need to update the caches according to that.
+ * Process only cache invalidation messages in this transaction if there
+ * are any. Even if we're not interested in the transaction's contents, it
+ * could have manipulated the catalog and we need to update the caches
+ * according to that.
*/
if (txn->base_snapshot != NULL && txn->ninvalidations > 0)
ReorderBufferImmediateInvalidation(rb, txn->ninvalidations,
@@ -2253,6 +2294,36 @@ ReorderBufferAddNewTupleCids(ReorderBuffer *rb, TransactionId xid,
txn->ntuplecids++;
}
+/*
+ * A helper function for ReorderBufferAddInvalidations() and
+ * ReorderBufferAddDistributedInvalidations() to accumulate the invalidation
+ * messages to the **invals_out.
+ */
+static void
+ReorderBufferAccumulateInvalidations(SharedInvalidationMessage **invals_out,
+ uint32 *ninvals_out,
+ SharedInvalidationMessage *msgs_new,
+ Size nmsgs_new)
+{
+ if (*ninvals_out == 0)
+ {
+ *ninvals_out = nmsgs_new;
+ *invals_out = (SharedInvalidationMessage *)
+ palloc(sizeof(SharedInvalidationMessage) * nmsgs_new);
+ memcpy(*invals_out, msgs_new, sizeof(SharedInvalidationMessage) * nmsgs_new);
+ }
+ else
+ {
+ /* Enlarge the array of inval messages */
+ *invals_out = (SharedInvalidationMessage *)
+ repalloc(*invals_out, sizeof(SharedInvalidationMessage) *
+ (*ninvals_out + nmsgs_new));
+ memcpy(*invals_out + *ninvals_out, msgs_new,
+ nmsgs_new * sizeof(SharedInvalidationMessage));
+ *ninvals_out += nmsgs_new;
+ }
+}
+
/*
* Setup the invalidation of the toplevel transaction.
*
@@ -2282,24 +2353,74 @@ ReorderBufferAddInvalidations(ReorderBuffer *rb, TransactionId xid,
Assert(nmsgs > 0);
- /* Accumulate invalidations. */
- if (txn->ninvalidations == 0)
+ ReorderBufferAccumulateInvalidations(&txn->invalidations,
+ &txn->ninvalidations,
+ msgs, nmsgs);
+
+ MemoryContextSwitchTo(oldcontext);
+}
+
+
+/*
+ * Accumulate the invalidations distributed by other committed transactions
+ * for executing them later.
+ *
+ * This function is similar to ReorderBufferAddInvalidations() but stores
+ * the given inval messages to the txn->invalidations_distributed with the
+ * overflow check.
+ *
+ * This needs to be called by committed transactions to distribute their
+ * inval messages to in-progress transactions.
+ */
+void
+ReorderBufferAddDistributedInvalidations(ReorderBuffer *rb, TransactionId xid,
+ XLogRecPtr lsn, Size nmsgs,
+ SharedInvalidationMessage *msgs)
+{
+ ReorderBufferTXN *txn;
+ MemoryContext oldcontext;
+
+ txn = ReorderBufferTXNByXid(rb, xid, true, NULL, lsn, true);
+
+ oldcontext = MemoryContextSwitchTo(rb->context);
+
+ /*
+ * Collect all the invalidations under the top transaction, if available,
+ * so that we can execute them all together.
+ */
+ if (txn->toplevel_xid)
{
- txn->ninvalidations = nmsgs;
- txn->invalidations = (SharedInvalidationMessage *)
- palloc(sizeof(SharedInvalidationMessage) * nmsgs);
- memcpy(txn->invalidations, msgs,
- sizeof(SharedInvalidationMessage) * nmsgs);
+ txn = ReorderBufferTXNByXid(rb, txn->toplevel_xid, true, NULL, lsn,
+ true);
}
- else
+
+ Assert(nmsgs > 0);
+
+ if (!rbtxn_distr_inval_overflowed(txn))
{
- txn->invalidations = (SharedInvalidationMessage *)
- repalloc(txn->invalidations, sizeof(SharedInvalidationMessage) *
- (txn->ninvalidations + nmsgs));
+ /*
+ * Check the transaction has enough space for storing distributed
+ * invalidation messages.
+ */
+ if (txn->ninvalidations_distributed + nmsgs >= MAX_DISTR_INVAL_MSG_PER_TXN)
+ {
+ /*
+ * Mark the invalidation message as overflowed and free up the
+ * messages accumulated so far.
+ */
+ txn->txn_flags |= RBTXN_DISTR_INVAL_OVERFLOWED;
- memcpy(txn->invalidations + txn->ninvalidations, msgs,
- nmsgs * sizeof(SharedInvalidationMessage));
- txn->ninvalidations += nmsgs;
+ if (txn->invalidations_distributed)
+ {
+ pfree(txn->invalidations_distributed);
+ txn->invalidations_distributed = NULL;
+ txn->ninvalidations_distributed = 0;
+ }
+ }
+ else
+ ReorderBufferAccumulateInvalidations(&txn->invalidations_distributed,
+ &txn->ninvalidations_distributed,
+ msgs, nmsgs);
}
MemoryContextSwitchTo(oldcontext);
@@ -2310,12 +2431,12 @@ ReorderBufferAddInvalidations(ReorderBuffer *rb, TransactionId xid,
* in the changestream but we don't know which those are.
*/
static void
-ReorderBufferExecuteInvalidations(ReorderBuffer *rb, ReorderBufferTXN *txn)
+ReorderBufferExecuteInvalidations(uint32 nmsgs, SharedInvalidationMessage *msgs)
{
int i;
- for (i = 0; i < txn->ninvalidations; i++)
- LocalExecuteInvalidationMessage(&txn->invalidations[i]);
+ for (i = 0; i < nmsgs; i++)
+ LocalExecuteInvalidationMessage(&msgs[i]);
}
/*
diff --git a/src/backend/replication/logical/snapbuild.c b/src/backend/replication/logical/snapbuild.c
index 3bda41c5251..69b083a36fe 100644
--- a/src/backend/replication/logical/snapbuild.c
+++ b/src/backend/replication/logical/snapbuild.c
@@ -927,6 +927,13 @@ SnapBuildDistributeSnapshotAndInval(SnapBuild *builder, XLogRecPtr lsn, Transact
* contents built by the current transaction even after its decoding,
* which should have been invalidated due to concurrent catalog
* changing transaction.
+ *
+ * Distribute only the invalidation messages generated by the current
+ * committed transaction. Invalidation messages received from other
+ * transactions would have already been propagated to the relevant
+ * in-progress transactions. This transaction would have processed
+ * those invalidations, ensuring that subsequent transactions observe
+ * a consistent cache state.
*/
if (txn->xid != xid)
{
@@ -940,8 +947,9 @@ SnapBuildDistributeSnapshotAndInval(SnapBuild *builder, XLogRecPtr lsn, Transact
{
Assert(msgs != NULL);
- ReorderBufferAddInvalidations(builder->reorder, txn->xid, lsn,
- ninvalidations, msgs);
+ ReorderBufferAddDistributedInvalidations(builder->reorder,
+ txn->xid, lsn,
+ ninvalidations, msgs);
}
}
}
diff --git a/src/include/replication/reorderbuffer.h b/src/include/replication/reorderbuffer.h
index 545cee891ed..2321bb0b406 100644
--- a/src/include/replication/reorderbuffer.h
+++ b/src/include/replication/reorderbuffer.h
@@ -160,9 +160,10 @@ typedef struct ReorderBufferChange
} ReorderBufferChange;
/* ReorderBufferTXN txn_flags */
-#define RBTXN_HAS_CATALOG_CHANGES 0x0001
-#define RBTXN_IS_SUBXACT 0x0002
-#define RBTXN_IS_SERIALIZED 0x0004
+#define RBTXN_HAS_CATALOG_CHANGES 0x0001
+#define RBTXN_IS_SUBXACT 0x0002
+#define RBTXN_IS_SERIALIZED 0x0004
+#define RBTXN_DISTR_INVAL_OVERFLOWED 0x0008
/* Does the transaction have catalog changes? */
#define rbtxn_has_catalog_changes(txn) \
@@ -182,6 +183,12 @@ typedef struct ReorderBufferChange
((txn)->txn_flags & RBTXN_IS_SERIALIZED) != 0 \
)
+/* Is the array of distributed inval messages overflowed? */
+#define rbtxn_distr_inval_overflowed(txn) \
+( \
+ ((txn)->txn_flags & RBTXN_DISTR_INVAL_OVERFLOWED) != 0 \
+)
+
typedef struct ReorderBufferTXN
{
/* See above */
@@ -311,6 +318,12 @@ typedef struct ReorderBufferTXN
* Size of this transaction (changes currently in memory, in bytes).
*/
Size size;
+
+ /*
+ * Stores cache invalidation messages distributed by other transactions.
+ */
+ uint32 ninvalidations_distributed;
+ SharedInvalidationMessage *invalidations_distributed;
} ReorderBufferTXN;
/* so we can define the callbacks used inside struct ReorderBuffer itself */
@@ -451,6 +464,9 @@ void ReorderBufferAddNewTupleCids(ReorderBuffer *, TransactionId, XLogRecPtr ls
CommandId cmin, CommandId cmax, CommandId combocid);
void ReorderBufferAddInvalidations(ReorderBuffer *, TransactionId, XLogRecPtr lsn,
Size nmsgs, SharedInvalidationMessage *msgs);
+void ReorderBufferAddDistributedInvalidations(ReorderBuffer *rb, TransactionId xid,
+ XLogRecPtr lsn, Size nmsgs,
+ SharedInvalidationMessage *msgs);
void ReorderBufferImmediateInvalidation(ReorderBuffer *, uint32 ninvalidations,
SharedInvalidationMessage *invalidations);
void ReorderBufferProcessXid(ReorderBuffer *, TransactionId xid, XLogRecPtr lsn);
--
2.43.0
[application/octet-stream] v13-PG15-0001-Fix-exponential-memory-allocation-issue-in-.patch (17.9K, ../../CALDaNm0TaTPuza7Fa+DRMzL+mqK3+7RVEvFiRoDJbU2vkJESwg@mail.gmail.com/4-v13-PG15-0001-Fix-exponential-memory-allocation-issue-in-.patch)
download | inline diff:
From 2b1edf7ddc2654f087c04781f1b7581fd94fddec Mon Sep 17 00:00:00 2001
From: Vignesh <vignesh21@gmail.com>
Date: Sun, 8 Jun 2025 09:51:51 +0530
Subject: [PATCH v13-PG15] Fix exponential memory allocation issue in logical
decoding due to invalidation message distribution
Commit 4909b38af0 added logic to distribute invalidation messages from
one transaction to all concurrent in-progress transactions. However,
since the source of each invalidation message was not tracked, transactions
could redundantly re-distribute already propagated messages. This led to an
exponential increase in invalidation messages and, ultimately, memory
allocation failures.
To fix this, a new field is introduced to track distributed invalidation
messages separately. This ensures that only transactions in the pre-existing
list receive the invalidations, preventing redundant propagation.
The accumulated distributed invalidations are now consumed at the end of
transaction processing in `ReorderBufferProcessTXN()`.
---
.../expected/invalidation_distribution.out | 26 ++-
.../specs/invalidation_distribution.spec | 21 ++
.../replication/logical/reorderbuffer.c | 197 +++++++++++++++---
src/backend/replication/logical/snapbuild.c | 12 +-
src/include/replication/reorderbuffer.h | 32 ++-
5 files changed, 244 insertions(+), 44 deletions(-)
diff --git a/contrib/test_decoding/expected/invalidation_distribution.out b/contrib/test_decoding/expected/invalidation_distribution.out
index 24190ebe570..3a604306a4c 100644
--- a/contrib/test_decoding/expected/invalidation_distribution.out
+++ b/contrib/test_decoding/expected/invalidation_distribution.out
@@ -1,4 +1,4 @@
-Parsed test spec with 2 sessions
+Parsed test spec with 5 sessions
starting permutation: s1_insert_tbl1 s1_begin s1_insert_tbl1 s2_alter_pub_add_tbl s1_commit s1_insert_tbl1 s2_get_binary_changes
step s1_insert_tbl1: INSERT INTO tbl1 (val1, val2) VALUES (1, 1);
@@ -18,3 +18,27 @@ count
stop
(1 row)
+
+starting permutation: s3i1 s4b s4i1 s5b s5i1 s3a s4i2 s4c s5c s4i3 s3i2 s3_get_binary_changes
+step s3i1: INSERT INTO tbl1 (val1, val2) VALUES (1, 1);
+step s4b: BEGIN;
+step s4i1: INSERT INTO tbl1 (val1, val2) VALUES (2, 2);
+step s5b: BEGIN;
+step s5i1: INSERT INTO tbl1 (val1, val2) VALUES (3, 3);
+step s3a: ALTER PUBLICATION pub ADD TABLE tbl1;
+step s4i2: INSERT INTO tbl1 (val1, val2) VALUES (4, 4);
+step s4c: COMMIT;
+step s5c: COMMIT;
+step s4i3: INSERT INTO tbl1 (val1, val2) VALUES (5, 5);
+step s3i2: INSERT INTO tbl1 (val1, val2) VALUES (6, 6);
+step s3_get_binary_changes: SELECT count(data) FROM pg_logical_slot_get_binary_changes('isolation_slot', NULL, NULL, 'proto_version', '3', 'publication_names', 'pub') WHERE get_byte(data, 0) = 73;
+count
+-----
+ 3
+(1 row)
+
+?column?
+--------
+stop
+(1 row)
+
diff --git a/contrib/test_decoding/specs/invalidation_distribution.spec b/contrib/test_decoding/specs/invalidation_distribution.spec
index f63aba3ce96..706219c1951 100644
--- a/contrib/test_decoding/specs/invalidation_distribution.spec
+++ b/contrib/test_decoding/specs/invalidation_distribution.spec
@@ -28,5 +28,26 @@ setup { SET synchronous_commit=on; }
step "s2_alter_pub_add_tbl" { ALTER PUBLICATION pub ADD TABLE tbl1; }
step "s2_get_binary_changes" { SELECT count(data) FROM pg_logical_slot_get_binary_changes('isolation_slot', NULL, NULL, 'proto_version', '3', 'publication_names', 'pub') WHERE get_byte(data, 0) = 73; }
+session "s3"
+step "s3i1" { INSERT INTO tbl1 (val1, val2) VALUES (1, 1);}
+step "s3a" { ALTER PUBLICATION pub ADD TABLE tbl1; }
+step "s3i2" { INSERT INTO tbl1 (val1, val2) VALUES (6, 6); }
+step "s3_get_binary_changes" { SELECT count(data) FROM pg_logical_slot_get_binary_changes('isolation_slot', NULL, NULL, 'proto_version', '3', 'publication_names', 'pub') WHERE get_byte(data, 0) = 73; }
+
+session "s4"
+step "s4b" { BEGIN; }
+step "s4i1" { INSERT INTO tbl1 (val1, val2) VALUES (2, 2);}
+step "s4i2" { INSERT INTO tbl1 (val1, val2) VALUES (4, 4); }
+step "s4c" { COMMIT; }
+step "s4i3" { INSERT INTO tbl1 (val1, val2) VALUES (5, 5); }
+
+session "s5"
+step "s5b" { BEGIN; }
+step "s5i1" { INSERT INTO tbl1 (val1, val2) VALUES (3, 3); }
+step "s5c" { COMMIT; }
+
# Expect to get one insert change. LOGICAL_REP_MSG_INSERT = 'I'
permutation "s1_insert_tbl1" "s1_begin" "s1_insert_tbl1" "s2_alter_pub_add_tbl" "s1_commit" "s1_insert_tbl1" "s2_get_binary_changes"
+
+# Expect to get three insert changes with LOGICAL_REP_MSG_INSERT = 'I' that happened after the table tbl1 is added to the publication.
+permutation "s3i1" "s4b" "s4i1" "s5b" "s5i1" "s3a" "s4i2" "s4c" "s5c" "s4i3" "s3i2" "s3_get_binary_changes"
diff --git a/src/backend/replication/logical/reorderbuffer.c b/src/backend/replication/logical/reorderbuffer.c
index 835180e12f6..59efa73930f 100644
--- a/src/backend/replication/logical/reorderbuffer.c
+++ b/src/backend/replication/logical/reorderbuffer.c
@@ -103,12 +103,24 @@
#include "storage/sinval.h"
#include "utils/builtins.h"
#include "utils/combocid.h"
+#include "utils/inval.h"
#include "utils/memdebug.h"
#include "utils/memutils.h"
#include "utils/rel.h"
#include "utils/relfilenodemap.h"
+/*
+ * Each transaction has an 8MB limit for invalidation messages distributed from
+ * other transactions. This limit is set considering scenarios with many
+ * concurrent logical decoding operations. When the distributed invalidation
+ * messages reach this threshold, the transaction is marked as
+ * RBTXN_DISTR_INVAL_OVERFLOWED to invalidate the complete cache as we have lost
+ * some inval messages and hence don't know what needs to be invalidated.
+ */
+#define MAX_DISTR_INVAL_MSG_PER_TXN \
+ ((8 * 1024 * 1024) / sizeof(SharedInvalidationMessage))
+
/* entry for a hash table we use to map from xid to our transaction state */
typedef struct ReorderBufferTXNByIdEnt
{
@@ -457,6 +469,12 @@ ReorderBufferReturnTXN(ReorderBuffer *rb, ReorderBufferTXN *txn)
txn->invalidations = NULL;
}
+ if (txn->invalidations_distributed)
+ {
+ pfree(txn->invalidations_distributed);
+ txn->invalidations_distributed = NULL;
+ }
+
/* Reset the toast hash */
ReorderBufferToastReset(rb, txn);
@@ -2503,7 +2521,17 @@ ReorderBufferProcessTXN(ReorderBuffer *rb, ReorderBufferTXN *txn,
AbortCurrentTransaction();
/* make sure there's no cache pollution */
- ReorderBufferExecuteInvalidations(txn->ninvalidations, txn->invalidations);
+ if (rbtxn_distr_inval_overflowed(txn))
+ {
+ Assert(txn->ninvalidations_distributed == 0);
+ InvalidateSystemCaches();
+ }
+ else
+ {
+ ReorderBufferExecuteInvalidations(txn->ninvalidations, txn->invalidations);
+ ReorderBufferExecuteInvalidations(txn->ninvalidations_distributed,
+ txn->invalidations_distributed);
+ }
if (using_subtxn)
RollbackAndReleaseCurrentSubTransaction();
@@ -2549,8 +2577,17 @@ ReorderBufferProcessTXN(ReorderBuffer *rb, ReorderBufferTXN *txn,
AbortCurrentTransaction();
/* make sure there's no cache pollution */
- ReorderBufferExecuteInvalidations(txn->ninvalidations,
- txn->invalidations);
+ if (rbtxn_distr_inval_overflowed(txn))
+ {
+ Assert(txn->ninvalidations_distributed == 0);
+ InvalidateSystemCaches();
+ }
+ else
+ {
+ ReorderBufferExecuteInvalidations(txn->ninvalidations, txn->invalidations);
+ ReorderBufferExecuteInvalidations(txn->ninvalidations_distributed,
+ txn->invalidations_distributed);
+ }
if (using_subtxn)
RollbackAndReleaseCurrentSubTransaction();
@@ -2877,7 +2914,8 @@ ReorderBufferAbort(ReorderBuffer *rb, TransactionId xid, XLogRecPtr lsn)
* We might have decoded changes for this transaction that could load
* the cache as per the current transaction's view (consider DDL's
* happened in this transaction). We don't want the decoding of future
- * transactions to use those cache entries so execute invalidations.
+ * transactions to use those cache entries so execute only the inval
+ * messages in this transaction.
*/
if (txn->ninvalidations > 0)
ReorderBufferImmediateInvalidation(rb, txn->ninvalidations,
@@ -2965,9 +3003,10 @@ ReorderBufferForget(ReorderBuffer *rb, TransactionId xid, XLogRecPtr lsn)
txn->final_lsn = lsn;
/*
- * Process cache invalidation messages if there are any. Even if we're not
- * interested in the transaction's contents, it could have manipulated the
- * catalog and we need to update the caches according to that.
+ * Process only cache invalidation messages in this transaction if there
+ * are any. Even if we're not interested in the transaction's contents, it
+ * could have manipulated the catalog and we need to update the caches
+ * according to that.
*/
if (txn->base_snapshot != NULL && txn->ninvalidations > 0)
ReorderBufferImmediateInvalidation(rb, txn->ninvalidations,
@@ -3222,6 +3261,57 @@ ReorderBufferAddNewTupleCids(ReorderBuffer *rb, TransactionId xid,
txn->ntuplecids++;
}
+/*
+ * Add new invalidation messages to the reorder buffer queue.
+ */
+static void
+ReorderBufferQueueInvalidations(ReorderBuffer *rb, TransactionId xid,
+ XLogRecPtr lsn, Size nmsgs,
+ SharedInvalidationMessage *msgs)
+{
+ ReorderBufferChange *change;
+
+ change = ReorderBufferGetChange(rb);
+ change->action = REORDER_BUFFER_CHANGE_INVALIDATION;
+ change->data.inval.ninvalidations = nmsgs;
+ change->data.inval.invalidations = (SharedInvalidationMessage *)
+ palloc(sizeof(SharedInvalidationMessage) * nmsgs);
+ memcpy(change->data.inval.invalidations, msgs,
+ sizeof(SharedInvalidationMessage) * nmsgs);
+
+ ReorderBufferQueueChange(rb, xid, lsn, change, false);
+}
+
+/*
+ * A helper function for ReorderBufferAddInvalidations() and
+ * ReorderBufferAddDistributedInvalidations() to accumulate the invalidation
+ * messages to the **invals_out.
+ */
+static void
+ReorderBufferAccumulateInvalidations(SharedInvalidationMessage **invals_out,
+ uint32 *ninvals_out,
+ SharedInvalidationMessage *msgs_new,
+ Size nmsgs_new)
+{
+ if (*ninvals_out == 0)
+ {
+ *ninvals_out = nmsgs_new;
+ *invals_out = (SharedInvalidationMessage *)
+ palloc(sizeof(SharedInvalidationMessage) * nmsgs_new);
+ memcpy(*invals_out, msgs_new, sizeof(SharedInvalidationMessage) * nmsgs_new);
+ }
+ else
+ {
+ /* Enlarge the array of inval messages */
+ *invals_out = (SharedInvalidationMessage *)
+ repalloc(*invals_out, sizeof(SharedInvalidationMessage) *
+ (*ninvals_out + nmsgs_new));
+ memcpy(*invals_out + *ninvals_out, msgs_new,
+ nmsgs_new * sizeof(SharedInvalidationMessage));
+ *ninvals_out += nmsgs_new;
+ }
+}
+
/*
* Accumulate the invalidations for executing them later.
*
@@ -3242,7 +3332,6 @@ ReorderBufferAddInvalidations(ReorderBuffer *rb, TransactionId xid,
{
ReorderBufferTXN *txn;
MemoryContext oldcontext;
- ReorderBufferChange *change;
txn = ReorderBufferTXNByXid(rb, xid, true, NULL, lsn, true);
@@ -3258,35 +3347,77 @@ ReorderBufferAddInvalidations(ReorderBuffer *rb, TransactionId xid,
Assert(nmsgs > 0);
- /* Accumulate invalidations. */
- if (txn->ninvalidations == 0)
- {
- txn->ninvalidations = nmsgs;
- txn->invalidations = (SharedInvalidationMessage *)
- palloc(sizeof(SharedInvalidationMessage) * nmsgs);
- memcpy(txn->invalidations, msgs,
- sizeof(SharedInvalidationMessage) * nmsgs);
- }
- else
+ ReorderBufferAccumulateInvalidations(&txn->invalidations,
+ &txn->ninvalidations,
+ msgs, nmsgs);
+
+ ReorderBufferQueueInvalidations(rb, xid, lsn, nmsgs, msgs);
+
+ MemoryContextSwitchTo(oldcontext);
+}
+
+/*
+ * Accumulate the invalidations distributed by other committed transactions
+ * for executing them later.
+ *
+ * This function is similar to ReorderBufferAddInvalidations() but stores
+ * the given inval messages to the txn->invalidations_distributed with the
+ * overflow check.
+ *
+ * This needs to be called by committed transactions to distribute their
+ * inval messages to in-progress transactions.
+ */
+void
+ReorderBufferAddDistributedInvalidations(ReorderBuffer *rb, TransactionId xid,
+ XLogRecPtr lsn, Size nmsgs,
+ SharedInvalidationMessage *msgs)
+{
+ ReorderBufferTXN *txn;
+ MemoryContext oldcontext;
+
+ txn = ReorderBufferTXNByXid(rb, xid, true, NULL, lsn, true);
+
+ oldcontext = MemoryContextSwitchTo(rb->context);
+
+ /*
+ * Collect all the invalidations under the top transaction, if available,
+ * so that we can execute them all together. See comments
+ * ReorderBufferAddInvalidations.
+ */
+ if (txn->toptxn)
+ txn = txn->toptxn;
+
+ Assert(nmsgs > 0);
+
+ if (!rbtxn_distr_inval_overflowed(txn))
{
- txn->invalidations = (SharedInvalidationMessage *)
- repalloc(txn->invalidations, sizeof(SharedInvalidationMessage) *
- (txn->ninvalidations + nmsgs));
+ /*
+ * Check the transaction has enough space for storing distributed
+ * invalidation messages.
+ */
+ if (txn->ninvalidations_distributed + nmsgs >= MAX_DISTR_INVAL_MSG_PER_TXN)
+ {
+ /*
+ * Mark the invalidation message as overflowed and free up the
+ * messages accumulated so far.
+ */
+ txn->txn_flags |= RBTXN_DISTR_INVAL_OVERFLOWED;
- memcpy(txn->invalidations + txn->ninvalidations, msgs,
- nmsgs * sizeof(SharedInvalidationMessage));
- txn->ninvalidations += nmsgs;
+ if (txn->invalidations_distributed)
+ {
+ pfree(txn->invalidations_distributed);
+ txn->invalidations_distributed = NULL;
+ txn->ninvalidations_distributed = 0;
+ }
+ }
+ else
+ ReorderBufferAccumulateInvalidations(&txn->invalidations_distributed,
+ &txn->ninvalidations_distributed,
+ msgs, nmsgs);
}
- change = ReorderBufferGetChange(rb);
- change->action = REORDER_BUFFER_CHANGE_INVALIDATION;
- change->data.inval.ninvalidations = nmsgs;
- change->data.inval.invalidations = (SharedInvalidationMessage *)
- palloc(sizeof(SharedInvalidationMessage) * nmsgs);
- memcpy(change->data.inval.invalidations, msgs,
- sizeof(SharedInvalidationMessage) * nmsgs);
-
- ReorderBufferQueueChange(rb, xid, lsn, change, false);
+ /* Queue the invalidation messages into the transaction */
+ ReorderBufferQueueInvalidations(rb, xid, lsn, nmsgs, msgs);
MemoryContextSwitchTo(oldcontext);
}
diff --git a/src/backend/replication/logical/snapbuild.c b/src/backend/replication/logical/snapbuild.c
index 2ca8e6e9aff..2dd4782f086 100644
--- a/src/backend/replication/logical/snapbuild.c
+++ b/src/backend/replication/logical/snapbuild.c
@@ -925,6 +925,13 @@ SnapBuildDistributeSnapshotAndInval(SnapBuild *builder, XLogRecPtr lsn, Transact
* contents built by the current transaction even after its decoding,
* which should have been invalidated due to concurrent catalog
* changing transaction.
+ *
+ * Distribute only the invalidation messages generated by the current
+ * committed transaction. Invalidation messages received from other
+ * transactions would have already been propagated to the relevant
+ * in-progress transactions. This transaction would have processed
+ * those invalidations, ensuring that subsequent transactions observe
+ * a consistent cache state.
*/
if (txn->xid != xid)
{
@@ -938,8 +945,9 @@ SnapBuildDistributeSnapshotAndInval(SnapBuild *builder, XLogRecPtr lsn, Transact
{
Assert(msgs != NULL);
- ReorderBufferAddInvalidations(builder->reorder, txn->xid, lsn,
- ninvalidations, msgs);
+ ReorderBufferAddDistributedInvalidations(builder->reorder,
+ txn->xid, lsn,
+ ninvalidations, msgs);
}
}
}
diff --git a/src/include/replication/reorderbuffer.h b/src/include/replication/reorderbuffer.h
index 402bb7a2728..af154587618 100644
--- a/src/include/replication/reorderbuffer.h
+++ b/src/include/replication/reorderbuffer.h
@@ -168,14 +168,15 @@ typedef struct ReorderBufferChange
} ReorderBufferChange;
/* ReorderBufferTXN txn_flags */
-#define RBTXN_HAS_CATALOG_CHANGES 0x0001
-#define RBTXN_IS_SUBXACT 0x0002
-#define RBTXN_IS_SERIALIZED 0x0004
-#define RBTXN_IS_SERIALIZED_CLEAR 0x0008
-#define RBTXN_IS_STREAMED 0x0010
-#define RBTXN_HAS_PARTIAL_CHANGE 0x0020
-#define RBTXN_PREPARE 0x0040
-#define RBTXN_SKIPPED_PREPARE 0x0080
+#define RBTXN_HAS_CATALOG_CHANGES 0x0001
+#define RBTXN_IS_SUBXACT 0x0002
+#define RBTXN_IS_SERIALIZED 0x0004
+#define RBTXN_IS_SERIALIZED_CLEAR 0x0008
+#define RBTXN_IS_STREAMED 0x0010
+#define RBTXN_HAS_PARTIAL_CHANGE 0x0020
+#define RBTXN_PREPARE 0x0040
+#define RBTXN_SKIPPED_PREPARE 0x0080
+#define RBTXN_DISTR_INVAL_OVERFLOWED 0x0100
/* Does the transaction have catalog changes? */
#define rbtxn_has_catalog_changes(txn) \
@@ -233,6 +234,12 @@ typedef struct ReorderBufferChange
((txn)->txn_flags & RBTXN_SKIPPED_PREPARE) != 0 \
)
+/* Is the array of distributed inval messages overflowed? */
+#define rbtxn_distr_inval_overflowed(txn) \
+( \
+ ((txn)->txn_flags & RBTXN_DISTR_INVAL_OVERFLOWED) != 0 \
+)
+
typedef struct ReorderBufferTXN
{
/* See above */
@@ -395,6 +402,12 @@ typedef struct ReorderBufferTXN
* Private data pointer of the output plugin.
*/
void *output_plugin_private;
+
+ /*
+ * Stores cache invalidation messages distributed by other transactions.
+ */
+ uint32 ninvalidations_distributed;
+ SharedInvalidationMessage *invalidations_distributed;
} ReorderBufferTXN;
/* so we can define the callbacks used inside struct ReorderBuffer itself */
@@ -661,6 +674,9 @@ extern void ReorderBufferAddNewTupleCids(ReorderBuffer *, TransactionId, XLogRec
CommandId cmin, CommandId cmax, CommandId combocid);
extern void ReorderBufferAddInvalidations(ReorderBuffer *, TransactionId, XLogRecPtr lsn,
Size nmsgs, SharedInvalidationMessage *msgs);
+extern void ReorderBufferAddDistributedInvalidations(ReorderBuffer *rb, TransactionId xid,
+ XLogRecPtr lsn, Size nmsgs,
+ SharedInvalidationMessage *msgs);
extern void ReorderBufferImmediateInvalidation(ReorderBuffer *, uint32 ninvalidations,
SharedInvalidationMessage *invalidations);
extern void ReorderBufferProcessXid(ReorderBuffer *, TransactionId xid, XLogRecPtr lsn);
--
2.43.0
[application/octet-stream] v13-PG16-0001-Fix-exponential-memory-allocation-issue-in-.patch (18.0K, ../../CALDaNm0TaTPuza7Fa+DRMzL+mqK3+7RVEvFiRoDJbU2vkJESwg@mail.gmail.com/5-v13-PG16-0001-Fix-exponential-memory-allocation-issue-in-.patch)
download | inline diff:
From 9d070460e1b59a707fd34a96b008e50e9bee6960 Mon Sep 17 00:00:00 2001
From: Vignesh <vignesh21@gmail.com>
Date: Sat, 7 Jun 2025 21:01:10 +0530
Subject: [PATCH v13-PG16] Fix exponential memory allocation issue in logical
decoding due to invalidation message distribution
Commit 4909b38af0 added logic to distribute invalidation messages from
one transaction to all concurrent in-progress transactions. However,
since the source of each invalidation message was not tracked, transactions
could redundantly re-distribute already propagated messages. This led to an
exponential increase in invalidation messages and, ultimately, memory
allocation failures.
To fix this, a new field is introduced to track distributed invalidation
messages separately. This ensures that only transactions in the pre-existing
list receive the invalidations, preventing redundant propagation.
The accumulated distributed invalidations are now consumed at the end of
transaction processing in `ReorderBufferProcessTXN()`.
---
.../expected/invalidation_distribution.out | 26 ++-
.../specs/invalidation_distribution.spec | 21 ++
.../replication/logical/reorderbuffer.c | 195 +++++++++++++++---
src/backend/replication/logical/snapbuild.c | 12 +-
src/include/replication/reorderbuffer.h | 34 ++-
5 files changed, 243 insertions(+), 45 deletions(-)
diff --git a/contrib/test_decoding/expected/invalidation_distribution.out b/contrib/test_decoding/expected/invalidation_distribution.out
index ad0a944cbf3..9fe87e2cda9 100644
--- a/contrib/test_decoding/expected/invalidation_distribution.out
+++ b/contrib/test_decoding/expected/invalidation_distribution.out
@@ -1,4 +1,4 @@
-Parsed test spec with 2 sessions
+Parsed test spec with 5 sessions
starting permutation: s1_insert_tbl1 s1_begin s1_insert_tbl1 s2_alter_pub_add_tbl s1_commit s1_insert_tbl1 s2_get_binary_changes
step s1_insert_tbl1: INSERT INTO tbl1 (val1, val2) VALUES (1, 1);
@@ -18,3 +18,27 @@ count
stop
(1 row)
+
+starting permutation: s3i1 s4b s4i1 s5b s5i1 s3a s4i2 s4c s5c s4i3 s3i2 s3_get_binary_changes
+step s3i1: INSERT INTO tbl1 (val1, val2) VALUES (1, 1);
+step s4b: BEGIN;
+step s4i1: INSERT INTO tbl1 (val1, val2) VALUES (2, 2);
+step s5b: BEGIN;
+step s5i1: INSERT INTO tbl1 (val1, val2) VALUES (3, 3);
+step s3a: ALTER PUBLICATION pub ADD TABLE tbl1;
+step s4i2: INSERT INTO tbl1 (val1, val2) VALUES (4, 4);
+step s4c: COMMIT;
+step s5c: COMMIT;
+step s4i3: INSERT INTO tbl1 (val1, val2) VALUES (5, 5);
+step s3i2: INSERT INTO tbl1 (val1, val2) VALUES (6, 6);
+step s3_get_binary_changes: SELECT count(data) FROM pg_logical_slot_get_binary_changes('isolation_slot', NULL, NULL, 'proto_version', '4', 'publication_names', 'pub') WHERE get_byte(data, 0) = 73;
+count
+-----
+ 3
+(1 row)
+
+?column?
+--------
+stop
+(1 row)
+
diff --git a/contrib/test_decoding/specs/invalidation_distribution.spec b/contrib/test_decoding/specs/invalidation_distribution.spec
index decbed627e3..1a90453f1ea 100644
--- a/contrib/test_decoding/specs/invalidation_distribution.spec
+++ b/contrib/test_decoding/specs/invalidation_distribution.spec
@@ -28,5 +28,26 @@ setup { SET synchronous_commit=on; }
step "s2_alter_pub_add_tbl" { ALTER PUBLICATION pub ADD TABLE tbl1; }
step "s2_get_binary_changes" { SELECT count(data) FROM pg_logical_slot_get_binary_changes('isolation_slot', NULL, NULL, 'proto_version', '4', 'publication_names', 'pub') WHERE get_byte(data, 0) = 73; }
+session "s3"
+step "s3i1" { INSERT INTO tbl1 (val1, val2) VALUES (1, 1);}
+step "s3a" { ALTER PUBLICATION pub ADD TABLE tbl1; }
+step "s3i2" { INSERT INTO tbl1 (val1, val2) VALUES (6, 6); }
+step "s3_get_binary_changes" { SELECT count(data) FROM pg_logical_slot_get_binary_changes('isolation_slot', NULL, NULL, 'proto_version', '4', 'publication_names', 'pub') WHERE get_byte(data, 0) = 73; }
+
+session "s4"
+step "s4b" { BEGIN; }
+step "s4i1" { INSERT INTO tbl1 (val1, val2) VALUES (2, 2);}
+step "s4i2" { INSERT INTO tbl1 (val1, val2) VALUES (4, 4); }
+step "s4c" { COMMIT; }
+step "s4i3" { INSERT INTO tbl1 (val1, val2) VALUES (5, 5); }
+
+session "s5"
+step "s5b" { BEGIN; }
+step "s5i1" { INSERT INTO tbl1 (val1, val2) VALUES (3, 3); }
+step "s5c" { COMMIT; }
+
# Expect to get one insert change. LOGICAL_REP_MSG_INSERT = 'I'
permutation "s1_insert_tbl1" "s1_begin" "s1_insert_tbl1" "s2_alter_pub_add_tbl" "s1_commit" "s1_insert_tbl1" "s2_get_binary_changes"
+
+# Expect to get three insert changes with LOGICAL_REP_MSG_INSERT = 'I' that happened after the table tbl1 is added to the publication.
+permutation "s3i1" "s4b" "s4i1" "s5b" "s5i1" "s3a" "s4i2" "s4c" "s5c" "s4i3" "s3i2" "s3_get_binary_changes"
diff --git a/src/backend/replication/logical/reorderbuffer.c b/src/backend/replication/logical/reorderbuffer.c
index fa04e829cc9..6464a926c20 100644
--- a/src/backend/replication/logical/reorderbuffer.c
+++ b/src/backend/replication/logical/reorderbuffer.c
@@ -103,11 +103,22 @@
#include "storage/sinval.h"
#include "utils/builtins.h"
#include "utils/combocid.h"
+#include "utils/inval.h"
#include "utils/memdebug.h"
#include "utils/memutils.h"
#include "utils/rel.h"
#include "utils/relfilenumbermap.h"
+/*
+ * Each transaction has an 8MB limit for invalidation messages distributed from
+ * other transactions. This limit is set considering scenarios with many
+ * concurrent logical decoding operations. When the distributed invalidation
+ * messages reach this threshold, the transaction is marked as
+ * RBTXN_DISTR_INVAL_OVERFLOWED to invalidate the complete cache as we have lost
+ * some inval messages and hence don't know what needs to be invalidated.
+ */
+#define MAX_DISTR_INVAL_MSG_PER_TXN \
+ ((8 * 1024 * 1024) / sizeof(SharedInvalidationMessage))
/* entry for a hash table we use to map from xid to our transaction state */
typedef struct ReorderBufferTXNByIdEnt
@@ -461,6 +472,12 @@ ReorderBufferReturnTXN(ReorderBuffer *rb, ReorderBufferTXN *txn)
txn->invalidations = NULL;
}
+ if (txn->invalidations_distributed)
+ {
+ pfree(txn->invalidations_distributed);
+ txn->invalidations_distributed = NULL;
+ }
+
/* Reset the toast hash */
ReorderBufferToastReset(rb, txn);
@@ -2539,7 +2556,17 @@ ReorderBufferProcessTXN(ReorderBuffer *rb, ReorderBufferTXN *txn,
AbortCurrentTransaction();
/* make sure there's no cache pollution */
- ReorderBufferExecuteInvalidations(txn->ninvalidations, txn->invalidations);
+ if (rbtxn_distr_inval_overflowed(txn))
+ {
+ Assert(txn->ninvalidations_distributed == 0);
+ InvalidateSystemCaches();
+ }
+ else
+ {
+ ReorderBufferExecuteInvalidations(txn->ninvalidations, txn->invalidations);
+ ReorderBufferExecuteInvalidations(txn->ninvalidations_distributed,
+ txn->invalidations_distributed);
+ }
if (using_subtxn)
RollbackAndReleaseCurrentSubTransaction();
@@ -2585,8 +2612,17 @@ ReorderBufferProcessTXN(ReorderBuffer *rb, ReorderBufferTXN *txn,
AbortCurrentTransaction();
/* make sure there's no cache pollution */
- ReorderBufferExecuteInvalidations(txn->ninvalidations,
- txn->invalidations);
+ if (rbtxn_distr_inval_overflowed(txn))
+ {
+ Assert(txn->ninvalidations_distributed == 0);
+ InvalidateSystemCaches();
+ }
+ else
+ {
+ ReorderBufferExecuteInvalidations(txn->ninvalidations, txn->invalidations);
+ ReorderBufferExecuteInvalidations(txn->ninvalidations_distributed,
+ txn->invalidations_distributed);
+ }
if (using_subtxn)
RollbackAndReleaseCurrentSubTransaction();
@@ -2916,7 +2952,8 @@ ReorderBufferAbort(ReorderBuffer *rb, TransactionId xid, XLogRecPtr lsn,
* We might have decoded changes for this transaction that could load
* the cache as per the current transaction's view (consider DDL's
* happened in this transaction). We don't want the decoding of future
- * transactions to use those cache entries so execute invalidations.
+ * transactions to use those cache entries so execute only the inval
+ * messages in this transaction.
*/
if (txn->ninvalidations > 0)
ReorderBufferImmediateInvalidation(rb, txn->ninvalidations,
@@ -3003,9 +3040,10 @@ ReorderBufferForget(ReorderBuffer *rb, TransactionId xid, XLogRecPtr lsn)
txn->final_lsn = lsn;
/*
- * Process cache invalidation messages if there are any. Even if we're not
- * interested in the transaction's contents, it could have manipulated the
- * catalog and we need to update the caches according to that.
+ * Process only cache invalidation messages in this transaction if there
+ * are any. Even if we're not interested in the transaction's contents, it
+ * could have manipulated the catalog and we need to update the caches
+ * according to that.
*/
if (txn->base_snapshot != NULL && txn->ninvalidations > 0)
ReorderBufferImmediateInvalidation(rb, txn->ninvalidations,
@@ -3257,6 +3295,57 @@ ReorderBufferAddNewTupleCids(ReorderBuffer *rb, TransactionId xid,
txn->ntuplecids++;
}
+/*
+ * Add new invalidation messages to the reorder buffer queue.
+ */
+static void
+ReorderBufferQueueInvalidations(ReorderBuffer *rb, TransactionId xid,
+ XLogRecPtr lsn, Size nmsgs,
+ SharedInvalidationMessage *msgs)
+{
+ ReorderBufferChange *change;
+
+ change = ReorderBufferGetChange(rb);
+ change->action = REORDER_BUFFER_CHANGE_INVALIDATION;
+ change->data.inval.ninvalidations = nmsgs;
+ change->data.inval.invalidations = (SharedInvalidationMessage *)
+ palloc(sizeof(SharedInvalidationMessage) * nmsgs);
+ memcpy(change->data.inval.invalidations, msgs,
+ sizeof(SharedInvalidationMessage) * nmsgs);
+
+ ReorderBufferQueueChange(rb, xid, lsn, change, false);
+}
+
+/*
+ * A helper function for ReorderBufferAddInvalidations() and
+ * ReorderBufferAddDistributedInvalidations() to accumulate the invalidation
+ * messages to the **invals_out.
+ */
+static void
+ReorderBufferAccumulateInvalidations(SharedInvalidationMessage **invals_out,
+ uint32 *ninvals_out,
+ SharedInvalidationMessage *msgs_new,
+ Size nmsgs_new)
+{
+ if (*ninvals_out == 0)
+ {
+ *ninvals_out = nmsgs_new;
+ *invals_out = (SharedInvalidationMessage *)
+ palloc(sizeof(SharedInvalidationMessage) * nmsgs_new);
+ memcpy(*invals_out, msgs_new, sizeof(SharedInvalidationMessage) * nmsgs_new);
+ }
+ else
+ {
+ /* Enlarge the array of inval messages */
+ *invals_out = (SharedInvalidationMessage *)
+ repalloc(*invals_out, sizeof(SharedInvalidationMessage) *
+ (*ninvals_out + nmsgs_new));
+ memcpy(*invals_out + *ninvals_out, msgs_new,
+ nmsgs_new * sizeof(SharedInvalidationMessage));
+ *ninvals_out += nmsgs_new;
+ }
+}
+
/*
* Accumulate the invalidations for executing them later.
*
@@ -3277,7 +3366,6 @@ ReorderBufferAddInvalidations(ReorderBuffer *rb, TransactionId xid,
{
ReorderBufferTXN *txn;
MemoryContext oldcontext;
- ReorderBufferChange *change;
txn = ReorderBufferTXNByXid(rb, xid, true, NULL, lsn, true);
@@ -3292,35 +3380,76 @@ ReorderBufferAddInvalidations(ReorderBuffer *rb, TransactionId xid,
Assert(nmsgs > 0);
- /* Accumulate invalidations. */
- if (txn->ninvalidations == 0)
- {
- txn->ninvalidations = nmsgs;
- txn->invalidations = (SharedInvalidationMessage *)
- palloc(sizeof(SharedInvalidationMessage) * nmsgs);
- memcpy(txn->invalidations, msgs,
- sizeof(SharedInvalidationMessage) * nmsgs);
- }
- else
+ ReorderBufferAccumulateInvalidations(&txn->invalidations,
+ &txn->ninvalidations,
+ msgs, nmsgs);
+
+ ReorderBufferQueueInvalidations(rb, xid, lsn, nmsgs, msgs);
+
+ MemoryContextSwitchTo(oldcontext);
+}
+
+/*
+ * Accumulate the invalidations distributed by other committed transactions
+ * for executing them later.
+ *
+ * This function is similar to ReorderBufferAddInvalidations() but stores
+ * the given inval messages to the txn->invalidations_distributed with the
+ * overflow check.
+ *
+ * This needs to be called by committed transactions to distribute their
+ * inval messages to in-progress transactions.
+ */
+void
+ReorderBufferAddDistributedInvalidations(ReorderBuffer *rb, TransactionId xid,
+ XLogRecPtr lsn, Size nmsgs,
+ SharedInvalidationMessage *msgs)
+{
+ ReorderBufferTXN *txn;
+ MemoryContext oldcontext;
+
+ txn = ReorderBufferTXNByXid(rb, xid, true, NULL, lsn, true);
+
+ oldcontext = MemoryContextSwitchTo(rb->context);
+
+ /*
+ * Collect all the invalidations under the top transaction, if available,
+ * so that we can execute them all together. See comments
+ * ReorderBufferAddInvalidations.
+ */
+ txn = rbtxn_get_toptxn(txn);
+
+ Assert(nmsgs > 0);
+
+ if (!rbtxn_distr_inval_overflowed(txn))
{
- txn->invalidations = (SharedInvalidationMessage *)
- repalloc(txn->invalidations, sizeof(SharedInvalidationMessage) *
- (txn->ninvalidations + nmsgs));
+ /*
+ * Check the transaction has enough space for storing distributed
+ * invalidation messages.
+ */
+ if (txn->ninvalidations_distributed + nmsgs >= MAX_DISTR_INVAL_MSG_PER_TXN)
+ {
+ /*
+ * Mark the invalidation message as overflowed and free up the
+ * messages accumulated so far.
+ */
+ txn->txn_flags |= RBTXN_DISTR_INVAL_OVERFLOWED;
- memcpy(txn->invalidations + txn->ninvalidations, msgs,
- nmsgs * sizeof(SharedInvalidationMessage));
- txn->ninvalidations += nmsgs;
+ if (txn->invalidations_distributed)
+ {
+ pfree(txn->invalidations_distributed);
+ txn->invalidations_distributed = NULL;
+ txn->ninvalidations_distributed = 0;
+ }
+ }
+ else
+ ReorderBufferAccumulateInvalidations(&txn->invalidations_distributed,
+ &txn->ninvalidations_distributed,
+ msgs, nmsgs);
}
- change = ReorderBufferGetChange(rb);
- change->action = REORDER_BUFFER_CHANGE_INVALIDATION;
- change->data.inval.ninvalidations = nmsgs;
- change->data.inval.invalidations = (SharedInvalidationMessage *)
- palloc(sizeof(SharedInvalidationMessage) * nmsgs);
- memcpy(change->data.inval.invalidations, msgs,
- sizeof(SharedInvalidationMessage) * nmsgs);
-
- ReorderBufferQueueChange(rb, xid, lsn, change, false);
+ /* Queue the invalidation messages into the transaction */
+ ReorderBufferQueueInvalidations(rb, xid, lsn, nmsgs, msgs);
MemoryContextSwitchTo(oldcontext);
}
diff --git a/src/backend/replication/logical/snapbuild.c b/src/backend/replication/logical/snapbuild.c
index d48ebb8337b..6c49dda75b6 100644
--- a/src/backend/replication/logical/snapbuild.c
+++ b/src/backend/replication/logical/snapbuild.c
@@ -918,6 +918,13 @@ SnapBuildDistributeSnapshotAndInval(SnapBuild *builder, XLogRecPtr lsn, Transact
* contents built by the current transaction even after its decoding,
* which should have been invalidated due to concurrent catalog
* changing transaction.
+ *
+ * Distribute only the invalidation messages generated by the current
+ * committed transaction. Invalidation messages received from other
+ * transactions would have already been propagated to the relevant
+ * in-progress transactions. This transaction would have processed
+ * those invalidations, ensuring that subsequent transactions observe
+ * a consistent cache state.
*/
if (txn->xid != xid)
{
@@ -931,8 +938,9 @@ SnapBuildDistributeSnapshotAndInval(SnapBuild *builder, XLogRecPtr lsn, Transact
{
Assert(msgs != NULL);
- ReorderBufferAddInvalidations(builder->reorder, txn->xid, lsn,
- ninvalidations, msgs);
+ ReorderBufferAddDistributedInvalidations(builder->reorder,
+ txn->xid, lsn,
+ ninvalidations, msgs);
}
}
}
diff --git a/src/include/replication/reorderbuffer.h b/src/include/replication/reorderbuffer.h
index 8a32bbe28df..1606e0ea007 100644
--- a/src/include/replication/reorderbuffer.h
+++ b/src/include/replication/reorderbuffer.h
@@ -177,15 +177,16 @@ typedef struct ReorderBufferChange
} ReorderBufferChange;
/* ReorderBufferTXN txn_flags */
-#define RBTXN_HAS_CATALOG_CHANGES 0x0001
-#define RBTXN_IS_SUBXACT 0x0002
-#define RBTXN_IS_SERIALIZED 0x0004
-#define RBTXN_IS_SERIALIZED_CLEAR 0x0008
-#define RBTXN_IS_STREAMED 0x0010
-#define RBTXN_HAS_PARTIAL_CHANGE 0x0020
-#define RBTXN_PREPARE 0x0040
-#define RBTXN_SKIPPED_PREPARE 0x0080
-#define RBTXN_HAS_STREAMABLE_CHANGE 0x0100
+#define RBTXN_HAS_CATALOG_CHANGES 0x0001
+#define RBTXN_IS_SUBXACT 0x0002
+#define RBTXN_IS_SERIALIZED 0x0004
+#define RBTXN_IS_SERIALIZED_CLEAR 0x0008
+#define RBTXN_IS_STREAMED 0x0010
+#define RBTXN_HAS_PARTIAL_CHANGE 0x0020
+#define RBTXN_PREPARE 0x0040
+#define RBTXN_SKIPPED_PREPARE 0x0080
+#define RBTXN_HAS_STREAMABLE_CHANGE 0x0100
+#define RBTXN_DISTR_INVAL_OVERFLOWED 0x0200
/* Does the transaction have catalog changes? */
#define rbtxn_has_catalog_changes(txn) \
@@ -249,6 +250,12 @@ typedef struct ReorderBufferChange
((txn)->txn_flags & RBTXN_SKIPPED_PREPARE) != 0 \
)
+/* Is the array of distributed inval messages overflowed? */
+#define rbtxn_distr_inval_overflowed(txn) \
+( \
+ ((txn)->txn_flags & RBTXN_DISTR_INVAL_OVERFLOWED) != 0 \
+)
+
/* Is this a top-level transaction? */
#define rbtxn_is_toptxn(txn) \
( \
@@ -435,6 +442,12 @@ typedef struct ReorderBufferTXN
* Private data pointer of the output plugin.
*/
void *output_plugin_private;
+
+ /*
+ * Stores cache invalidation messages distributed by other transactions.
+ */
+ uint32 ninvalidations_distributed;
+ SharedInvalidationMessage *invalidations_distributed;
} ReorderBufferTXN;
/* so we can define the callbacks used inside struct ReorderBuffer itself */
@@ -728,6 +741,9 @@ extern void ReorderBufferAddNewTupleCids(ReorderBuffer *rb, TransactionId xid,
CommandId cmin, CommandId cmax, CommandId combocid);
extern void ReorderBufferAddInvalidations(ReorderBuffer *rb, TransactionId xid, XLogRecPtr lsn,
Size nmsgs, SharedInvalidationMessage *msgs);
+extern void ReorderBufferAddDistributedInvalidations(ReorderBuffer *rb, TransactionId xid,
+ XLogRecPtr lsn, Size nmsgs,
+ SharedInvalidationMessage *msgs);
extern void ReorderBufferImmediateInvalidation(ReorderBuffer *rb, uint32 ninvalidations,
SharedInvalidationMessage *invalidations);
extern void ReorderBufferProcessXid(ReorderBuffer *rb, TransactionId xid, XLogRecPtr lsn);
--
2.43.0
[application/octet-stream] v13-PG14-0001-Fix-exponential-memory-allocation-issue-in-.patch (17.8K, ../../CALDaNm0TaTPuza7Fa+DRMzL+mqK3+7RVEvFiRoDJbU2vkJESwg@mail.gmail.com/6-v13-PG14-0001-Fix-exponential-memory-allocation-issue-in-.patch)
download | inline diff:
From 6128faafa9f21e3f57dc3f696e27c2aba3dfbbe3 Mon Sep 17 00:00:00 2001
From: Vignesh <vignesh21@gmail.com>
Date: Sun, 8 Jun 2025 10:23:07 +0530
Subject: [PATCH v13-PG14] Fix exponential memory allocation issue in logical
decoding due to invalidation message distribution
Commit 4909b38af0 added logic to distribute invalidation messages from
one transaction to all concurrent in-progress transactions. However,
since the source of each invalidation message was not tracked, transactions
could redundantly re-distribute already propagated messages. This led to an
exponential increase in invalidation messages and, ultimately, memory
allocation failures.
To fix this, a new field is introduced to track distributed invalidation
messages separately. This ensures that only transactions in the pre-existing
list receive the invalidations, preventing redundant propagation.
The accumulated distributed invalidations are now consumed at the end of
transaction processing in `ReorderBufferProcessTXN()`.
---
.../expected/invalidation_distribution.out | 26 ++-
.../specs/invalidation_distribution.spec | 21 ++
.../replication/logical/reorderbuffer.c | 197 +++++++++++++++---
src/backend/replication/logical/snapbuild.c | 12 +-
src/include/replication/reorderbuffer.h | 32 ++-
5 files changed, 244 insertions(+), 44 deletions(-)
diff --git a/contrib/test_decoding/expected/invalidation_distribution.out b/contrib/test_decoding/expected/invalidation_distribution.out
index c701e290bb9..9ea1dd7a6d1 100644
--- a/contrib/test_decoding/expected/invalidation_distribution.out
+++ b/contrib/test_decoding/expected/invalidation_distribution.out
@@ -1,4 +1,4 @@
-Parsed test spec with 2 sessions
+Parsed test spec with 5 sessions
starting permutation: s1_insert_tbl1 s1_begin s1_insert_tbl1 s2_alter_pub_add_tbl s1_commit s1_insert_tbl1 s2_get_binary_changes
step s1_insert_tbl1: INSERT INTO tbl1 (val1, val2) VALUES (1, 1);
@@ -18,3 +18,27 @@ count
stop
(1 row)
+
+starting permutation: s3i1 s4b s4i1 s5b s5i1 s3a s4i2 s4c s5c s4i3 s3i2 s3_get_binary_changes
+step s3i1: INSERT INTO tbl1 (val1, val2) VALUES (1, 1);
+step s4b: BEGIN;
+step s4i1: INSERT INTO tbl1 (val1, val2) VALUES (2, 2);
+step s5b: BEGIN;
+step s5i1: INSERT INTO tbl1 (val1, val2) VALUES (3, 3);
+step s3a: ALTER PUBLICATION pub ADD TABLE tbl1;
+step s4i2: INSERT INTO tbl1 (val1, val2) VALUES (4, 4);
+step s4c: COMMIT;
+step s5c: COMMIT;
+step s4i3: INSERT INTO tbl1 (val1, val2) VALUES (5, 5);
+step s3i2: INSERT INTO tbl1 (val1, val2) VALUES (6, 6);
+step s3_get_binary_changes: SELECT count(data) FROM pg_logical_slot_get_binary_changes('isolation_slot', NULL, NULL, 'proto_version', '2', 'publication_names', 'pub') WHERE get_byte(data, 0) = 73;
+count
+-----
+ 3
+(1 row)
+
+?column?
+--------
+stop
+(1 row)
+
diff --git a/contrib/test_decoding/specs/invalidation_distribution.spec b/contrib/test_decoding/specs/invalidation_distribution.spec
index b8b14e333a1..41038125348 100644
--- a/contrib/test_decoding/specs/invalidation_distribution.spec
+++ b/contrib/test_decoding/specs/invalidation_distribution.spec
@@ -28,5 +28,26 @@ setup { SET synchronous_commit=on; }
step "s2_alter_pub_add_tbl" { ALTER PUBLICATION pub ADD TABLE tbl1; }
step "s2_get_binary_changes" { SELECT count(data) FROM pg_logical_slot_get_binary_changes('isolation_slot', NULL, NULL, 'proto_version', '2', 'publication_names', 'pub') WHERE get_byte(data, 0) = 73; }
+session "s3"
+step "s3i1" { INSERT INTO tbl1 (val1, val2) VALUES (1, 1);}
+step "s3a" { ALTER PUBLICATION pub ADD TABLE tbl1; }
+step "s3i2" { INSERT INTO tbl1 (val1, val2) VALUES (6, 6); }
+step "s3_get_binary_changes" { SELECT count(data) FROM pg_logical_slot_get_binary_changes('isolation_slot', NULL, NULL, 'proto_version', '2', 'publication_names', 'pub') WHERE get_byte(data, 0) = 73; }
+
+session "s4"
+step "s4b" { BEGIN; }
+step "s4i1" { INSERT INTO tbl1 (val1, val2) VALUES (2, 2);}
+step "s4i2" { INSERT INTO tbl1 (val1, val2) VALUES (4, 4); }
+step "s4c" { COMMIT; }
+step "s4i3" { INSERT INTO tbl1 (val1, val2) VALUES (5, 5); }
+
+session "s5"
+step "s5b" { BEGIN; }
+step "s5i1" { INSERT INTO tbl1 (val1, val2) VALUES (3, 3); }
+step "s5c" { COMMIT; }
+
# Expect to get one insert change. LOGICAL_REP_MSG_INSERT = 'I'
permutation "s1_insert_tbl1" "s1_begin" "s1_insert_tbl1" "s2_alter_pub_add_tbl" "s1_commit" "s1_insert_tbl1" "s2_get_binary_changes"
+
+# Expect to get three insert changes with LOGICAL_REP_MSG_INSERT = 'I' that happened after the table tbl1 is added to the publication.
+permutation "s3i1" "s4b" "s4i1" "s5b" "s5i1" "s3a" "s4i2" "s4c" "s5c" "s4i3" "s3i2" "s3_get_binary_changes"
diff --git a/src/backend/replication/logical/reorderbuffer.c b/src/backend/replication/logical/reorderbuffer.c
index a52b51a5e7d..b708877d965 100644
--- a/src/backend/replication/logical/reorderbuffer.c
+++ b/src/backend/replication/logical/reorderbuffer.c
@@ -103,12 +103,24 @@
#include "storage/sinval.h"
#include "utils/builtins.h"
#include "utils/combocid.h"
+#include "utils/inval.h"
#include "utils/memdebug.h"
#include "utils/memutils.h"
#include "utils/rel.h"
#include "utils/relfilenodemap.h"
+/*
+ * Each transaction has an 8MB limit for invalidation messages distributed from
+ * other transactions. This limit is set considering scenarios with many
+ * concurrent logical decoding operations. When the distributed invalidation
+ * messages reach this threshold, the transaction is marked as
+ * RBTXN_DISTR_INVAL_OVERFLOWED to invalidate the complete cache as we have lost
+ * some inval messages and hence don't know what needs to be invalidated.
+ */
+#define MAX_DISTR_INVAL_MSG_PER_TXN \
+ ((8 * 1024 * 1024) / sizeof(SharedInvalidationMessage))
+
/* entry for a hash table we use to map from xid to our transaction state */
typedef struct ReorderBufferTXNByIdEnt
{
@@ -453,6 +465,12 @@ ReorderBufferReturnTXN(ReorderBuffer *rb, ReorderBufferTXN *txn)
txn->invalidations = NULL;
}
+ if (txn->invalidations_distributed)
+ {
+ pfree(txn->invalidations_distributed);
+ txn->invalidations_distributed = NULL;
+ }
+
/* Reset the toast hash */
ReorderBufferToastReset(rb, txn);
@@ -2500,7 +2518,17 @@ ReorderBufferProcessTXN(ReorderBuffer *rb, ReorderBufferTXN *txn,
AbortCurrentTransaction();
/* make sure there's no cache pollution */
- ReorderBufferExecuteInvalidations(txn->ninvalidations, txn->invalidations);
+ if (rbtxn_distr_inval_overflowed(txn))
+ {
+ Assert(txn->ninvalidations_distributed == 0);
+ InvalidateSystemCaches();
+ }
+ else
+ {
+ ReorderBufferExecuteInvalidations(txn->ninvalidations, txn->invalidations);
+ ReorderBufferExecuteInvalidations(txn->ninvalidations_distributed,
+ txn->invalidations_distributed);
+ }
if (using_subtxn)
RollbackAndReleaseCurrentSubTransaction();
@@ -2546,8 +2574,17 @@ ReorderBufferProcessTXN(ReorderBuffer *rb, ReorderBufferTXN *txn,
AbortCurrentTransaction();
/* make sure there's no cache pollution */
- ReorderBufferExecuteInvalidations(txn->ninvalidations,
- txn->invalidations);
+ if (rbtxn_distr_inval_overflowed(txn))
+ {
+ Assert(txn->ninvalidations_distributed == 0);
+ InvalidateSystemCaches();
+ }
+ else
+ {
+ ReorderBufferExecuteInvalidations(txn->ninvalidations, txn->invalidations);
+ ReorderBufferExecuteInvalidations(txn->ninvalidations_distributed,
+ txn->invalidations_distributed);
+ }
if (using_subtxn)
RollbackAndReleaseCurrentSubTransaction();
@@ -2873,7 +2910,8 @@ ReorderBufferAbort(ReorderBuffer *rb, TransactionId xid, XLogRecPtr lsn)
* We might have decoded changes for this transaction that could load
* the cache as per the current transaction's view (consider DDL's
* happened in this transaction). We don't want the decoding of future
- * transactions to use those cache entries so execute invalidations.
+ * transactions to use those cache entries so execute only the inval
+ * messages in this transaction.
*/
if (txn->ninvalidations > 0)
ReorderBufferImmediateInvalidation(rb, txn->ninvalidations,
@@ -2961,9 +2999,10 @@ ReorderBufferForget(ReorderBuffer *rb, TransactionId xid, XLogRecPtr lsn)
txn->final_lsn = lsn;
/*
- * Process cache invalidation messages if there are any. Even if we're not
- * interested in the transaction's contents, it could have manipulated the
- * catalog and we need to update the caches according to that.
+ * Process only cache invalidation messages in this transaction if there
+ * are any. Even if we're not interested in the transaction's contents, it
+ * could have manipulated the catalog and we need to update the caches
+ * according to that.
*/
if (txn->base_snapshot != NULL && txn->ninvalidations > 0)
ReorderBufferImmediateInvalidation(rb, txn->ninvalidations,
@@ -3218,6 +3257,57 @@ ReorderBufferAddNewTupleCids(ReorderBuffer *rb, TransactionId xid,
txn->ntuplecids++;
}
+/*
+ * Add new invalidation messages to the reorder buffer queue.
+ */
+static void
+ReorderBufferQueueInvalidations(ReorderBuffer *rb, TransactionId xid,
+ XLogRecPtr lsn, Size nmsgs,
+ SharedInvalidationMessage *msgs)
+{
+ ReorderBufferChange *change;
+
+ change = ReorderBufferGetChange(rb);
+ change->action = REORDER_BUFFER_CHANGE_INVALIDATION;
+ change->data.inval.ninvalidations = nmsgs;
+ change->data.inval.invalidations = (SharedInvalidationMessage *)
+ palloc(sizeof(SharedInvalidationMessage) * nmsgs);
+ memcpy(change->data.inval.invalidations, msgs,
+ sizeof(SharedInvalidationMessage) * nmsgs);
+
+ ReorderBufferQueueChange(rb, xid, lsn, change, false);
+}
+
+/*
+ * A helper function for ReorderBufferAddInvalidations() and
+ * ReorderBufferAddDistributedInvalidations() to accumulate the invalidation
+ * messages to the **invals_out.
+ */
+static void
+ReorderBufferAccumulateInvalidations(SharedInvalidationMessage **invals_out,
+ uint32 *ninvals_out,
+ SharedInvalidationMessage *msgs_new,
+ Size nmsgs_new)
+{
+ if (*ninvals_out == 0)
+ {
+ *ninvals_out = nmsgs_new;
+ *invals_out = (SharedInvalidationMessage *)
+ palloc(sizeof(SharedInvalidationMessage) * nmsgs_new);
+ memcpy(*invals_out, msgs_new, sizeof(SharedInvalidationMessage) * nmsgs_new);
+ }
+ else
+ {
+ /* Enlarge the array of inval messages */
+ *invals_out = (SharedInvalidationMessage *)
+ repalloc(*invals_out, sizeof(SharedInvalidationMessage) *
+ (*ninvals_out + nmsgs_new));
+ memcpy(*invals_out + *ninvals_out, msgs_new,
+ nmsgs_new * sizeof(SharedInvalidationMessage));
+ *ninvals_out += nmsgs_new;
+ }
+}
+
/*
* Accumulate the invalidations for executing them later.
*
@@ -3238,7 +3328,6 @@ ReorderBufferAddInvalidations(ReorderBuffer *rb, TransactionId xid,
{
ReorderBufferTXN *txn;
MemoryContext oldcontext;
- ReorderBufferChange *change;
txn = ReorderBufferTXNByXid(rb, xid, true, NULL, lsn, true);
@@ -3254,35 +3343,77 @@ ReorderBufferAddInvalidations(ReorderBuffer *rb, TransactionId xid,
Assert(nmsgs > 0);
- /* Accumulate invalidations. */
- if (txn->ninvalidations == 0)
- {
- txn->ninvalidations = nmsgs;
- txn->invalidations = (SharedInvalidationMessage *)
- palloc(sizeof(SharedInvalidationMessage) * nmsgs);
- memcpy(txn->invalidations, msgs,
- sizeof(SharedInvalidationMessage) * nmsgs);
- }
- else
+ ReorderBufferAccumulateInvalidations(&txn->invalidations,
+ &txn->ninvalidations,
+ msgs, nmsgs);
+
+ ReorderBufferQueueInvalidations(rb, xid, lsn, nmsgs, msgs);
+
+ MemoryContextSwitchTo(oldcontext);
+}
+
+/*
+ * Accumulate the invalidations distributed by other committed transactions
+ * for executing them later.
+ *
+ * This function is similar to ReorderBufferAddInvalidations() but stores
+ * the given inval messages to the txn->invalidations_distributed with the
+ * overflow check.
+ *
+ * This needs to be called by committed transactions to distribute their
+ * inval messages to in-progress transactions.
+ */
+void
+ReorderBufferAddDistributedInvalidations(ReorderBuffer *rb, TransactionId xid,
+ XLogRecPtr lsn, Size nmsgs,
+ SharedInvalidationMessage *msgs)
+{
+ ReorderBufferTXN *txn;
+ MemoryContext oldcontext;
+
+ txn = ReorderBufferTXNByXid(rb, xid, true, NULL, lsn, true);
+
+ oldcontext = MemoryContextSwitchTo(rb->context);
+
+ /*
+ * Collect all the invalidations under the top transaction, if available,
+ * so that we can execute them all together. See comments
+ * ReorderBufferAddInvalidations.
+ */
+ if (txn->toptxn)
+ txn = txn->toptxn;
+
+ Assert(nmsgs > 0);
+
+ if (!rbtxn_distr_inval_overflowed(txn))
{
- txn->invalidations = (SharedInvalidationMessage *)
- repalloc(txn->invalidations, sizeof(SharedInvalidationMessage) *
- (txn->ninvalidations + nmsgs));
+ /*
+ * Check the transaction has enough space for storing distributed
+ * invalidation messages.
+ */
+ if (txn->ninvalidations_distributed + nmsgs >= MAX_DISTR_INVAL_MSG_PER_TXN)
+ {
+ /*
+ * Mark the invalidation message as overflowed and free up the
+ * messages accumulated so far.
+ */
+ txn->txn_flags |= RBTXN_DISTR_INVAL_OVERFLOWED;
- memcpy(txn->invalidations + txn->ninvalidations, msgs,
- nmsgs * sizeof(SharedInvalidationMessage));
- txn->ninvalidations += nmsgs;
+ if (txn->invalidations_distributed)
+ {
+ pfree(txn->invalidations_distributed);
+ txn->invalidations_distributed = NULL;
+ txn->ninvalidations_distributed = 0;
+ }
+ }
+ else
+ ReorderBufferAccumulateInvalidations(&txn->invalidations_distributed,
+ &txn->ninvalidations_distributed,
+ msgs, nmsgs);
}
- change = ReorderBufferGetChange(rb);
- change->action = REORDER_BUFFER_CHANGE_INVALIDATION;
- change->data.inval.ninvalidations = nmsgs;
- change->data.inval.invalidations = (SharedInvalidationMessage *)
- palloc(sizeof(SharedInvalidationMessage) * nmsgs);
- memcpy(change->data.inval.invalidations, msgs,
- sizeof(SharedInvalidationMessage) * nmsgs);
-
- ReorderBufferQueueChange(rb, xid, lsn, change, false);
+ /* Queue the invalidation messages into the transaction */
+ ReorderBufferQueueInvalidations(rb, xid, lsn, nmsgs, msgs);
MemoryContextSwitchTo(oldcontext);
}
diff --git a/src/backend/replication/logical/snapbuild.c b/src/backend/replication/logical/snapbuild.c
index 6025f2727ae..67f0b9e089a 100644
--- a/src/backend/replication/logical/snapbuild.c
+++ b/src/backend/replication/logical/snapbuild.c
@@ -916,6 +916,13 @@ SnapBuildDistributeSnapshotAndInval(SnapBuild *builder, XLogRecPtr lsn, Transact
* contents built by the current transaction even after its decoding,
* which should have been invalidated due to concurrent catalog
* changing transaction.
+ *
+ * Distribute only the invalidation messages generated by the current
+ * committed transaction. Invalidation messages received from other
+ * transactions would have already been propagated to the relevant
+ * in-progress transactions. This transaction would have processed
+ * those invalidations, ensuring that subsequent transactions observe
+ * a consistent cache state.
*/
if (txn->xid != xid)
{
@@ -929,8 +936,9 @@ SnapBuildDistributeSnapshotAndInval(SnapBuild *builder, XLogRecPtr lsn, Transact
{
Assert(msgs != NULL);
- ReorderBufferAddInvalidations(builder->reorder, txn->xid, lsn,
- ninvalidations, msgs);
+ ReorderBufferAddDistributedInvalidations(builder->reorder,
+ txn->xid, lsn,
+ ninvalidations, msgs);
}
}
}
diff --git a/src/include/replication/reorderbuffer.h b/src/include/replication/reorderbuffer.h
index d399975e8a5..cb633315d04 100644
--- a/src/include/replication/reorderbuffer.h
+++ b/src/include/replication/reorderbuffer.h
@@ -168,14 +168,15 @@ typedef struct ReorderBufferChange
} ReorderBufferChange;
/* ReorderBufferTXN txn_flags */
-#define RBTXN_HAS_CATALOG_CHANGES 0x0001
-#define RBTXN_IS_SUBXACT 0x0002
-#define RBTXN_IS_SERIALIZED 0x0004
-#define RBTXN_IS_SERIALIZED_CLEAR 0x0008
-#define RBTXN_IS_STREAMED 0x0010
-#define RBTXN_HAS_PARTIAL_CHANGE 0x0020
-#define RBTXN_PREPARE 0x0040
-#define RBTXN_SKIPPED_PREPARE 0x0080
+#define RBTXN_HAS_CATALOG_CHANGES 0x0001
+#define RBTXN_IS_SUBXACT 0x0002
+#define RBTXN_IS_SERIALIZED 0x0004
+#define RBTXN_IS_SERIALIZED_CLEAR 0x0008
+#define RBTXN_IS_STREAMED 0x0010
+#define RBTXN_HAS_PARTIAL_CHANGE 0x0020
+#define RBTXN_PREPARE 0x0040
+#define RBTXN_SKIPPED_PREPARE 0x0080
+#define RBTXN_DISTR_INVAL_OVERFLOWED 0x0100
/* Does the transaction have catalog changes? */
#define rbtxn_has_catalog_changes(txn) \
@@ -233,6 +234,12 @@ typedef struct ReorderBufferChange
((txn)->txn_flags & RBTXN_SKIPPED_PREPARE) != 0 \
)
+/* Is the array of distributed inval messages overflowed? */
+#define rbtxn_distr_inval_overflowed(txn) \
+( \
+ ((txn)->txn_flags & RBTXN_DISTR_INVAL_OVERFLOWED) != 0 \
+)
+
typedef struct ReorderBufferTXN
{
/* See above */
@@ -391,6 +398,12 @@ typedef struct ReorderBufferTXN
* Private data pointer of the output plugin.
*/
void *output_plugin_private;
+
+ /*
+ * Stores cache invalidation messages distributed by other transactions.
+ */
+ uint32 ninvalidations_distributed;
+ SharedInvalidationMessage *invalidations_distributed;
} ReorderBufferTXN;
/* so we can define the callbacks used inside struct ReorderBuffer itself */
@@ -657,6 +670,9 @@ void ReorderBufferAddNewTupleCids(ReorderBuffer *, TransactionId, XLogRecPtr ls
CommandId cmin, CommandId cmax, CommandId combocid);
void ReorderBufferAddInvalidations(ReorderBuffer *, TransactionId, XLogRecPtr lsn,
Size nmsgs, SharedInvalidationMessage *msgs);
+void ReorderBufferAddDistributedInvalidations(ReorderBuffer *rb, TransactionId xid,
+ XLogRecPtr lsn, Size nmsgs,
+ SharedInvalidationMessage *msgs);
void ReorderBufferImmediateInvalidation(ReorderBuffer *, uint32 ninvalidations,
SharedInvalidationMessage *invalidations);
void ReorderBufferProcessXid(ReorderBuffer *, TransactionId xid, XLogRecPtr lsn);
--
2.43.0
[application/octet-stream] v13-PG17-0001-Fix-exponential-memory-allocation-issue-in-.patch (17.9K, ../../CALDaNm0TaTPuza7Fa+DRMzL+mqK3+7RVEvFiRoDJbU2vkJESwg@mail.gmail.com/7-v13-PG17-0001-Fix-exponential-memory-allocation-issue-in-.patch)
download | inline diff:
From 2af67e3d6051320354b3cbf019d041b05e7cb8d0 Mon Sep 17 00:00:00 2001
From: Vignesh <vignesh21@gmail.com>
Date: Sat, 7 Jun 2025 20:34:19 +0530
Subject: [PATCH v13-PG17] Fix exponential memory allocation issue in logical
decoding due to invalidation message distribution
Commit 4909b38af0 added logic to distribute invalidation messages from
one transaction to all concurrent in-progress transactions. However,
since the source of each invalidation message was not tracked, transactions
could redundantly re-distribute already propagated messages. This led to an
exponential increase in invalidation messages and, ultimately, memory
allocation failures.
To fix this, a new field is introduced to track distributed invalidation
messages separately. This ensures that only transactions in the pre-existing
list receive the invalidations, preventing redundant propagation.
The accumulated distributed invalidations are now consumed at the end of
transaction processing in `ReorderBufferProcessTXN()`.
---
.../expected/invalidation_distribution.out | 26 ++-
.../specs/invalidation_distribution.spec | 21 ++
.../replication/logical/reorderbuffer.c | 196 +++++++++++++++---
src/backend/replication/logical/snapbuild.c | 12 +-
src/include/replication/reorderbuffer.h | 34 ++-
5 files changed, 244 insertions(+), 45 deletions(-)
diff --git a/contrib/test_decoding/expected/invalidation_distribution.out b/contrib/test_decoding/expected/invalidation_distribution.out
index ad0a944cbf3..9fe87e2cda9 100644
--- a/contrib/test_decoding/expected/invalidation_distribution.out
+++ b/contrib/test_decoding/expected/invalidation_distribution.out
@@ -1,4 +1,4 @@
-Parsed test spec with 2 sessions
+Parsed test spec with 5 sessions
starting permutation: s1_insert_tbl1 s1_begin s1_insert_tbl1 s2_alter_pub_add_tbl s1_commit s1_insert_tbl1 s2_get_binary_changes
step s1_insert_tbl1: INSERT INTO tbl1 (val1, val2) VALUES (1, 1);
@@ -18,3 +18,27 @@ count
stop
(1 row)
+
+starting permutation: s3i1 s4b s4i1 s5b s5i1 s3a s4i2 s4c s5c s4i3 s3i2 s3_get_binary_changes
+step s3i1: INSERT INTO tbl1 (val1, val2) VALUES (1, 1);
+step s4b: BEGIN;
+step s4i1: INSERT INTO tbl1 (val1, val2) VALUES (2, 2);
+step s5b: BEGIN;
+step s5i1: INSERT INTO tbl1 (val1, val2) VALUES (3, 3);
+step s3a: ALTER PUBLICATION pub ADD TABLE tbl1;
+step s4i2: INSERT INTO tbl1 (val1, val2) VALUES (4, 4);
+step s4c: COMMIT;
+step s5c: COMMIT;
+step s4i3: INSERT INTO tbl1 (val1, val2) VALUES (5, 5);
+step s3i2: INSERT INTO tbl1 (val1, val2) VALUES (6, 6);
+step s3_get_binary_changes: SELECT count(data) FROM pg_logical_slot_get_binary_changes('isolation_slot', NULL, NULL, 'proto_version', '4', 'publication_names', 'pub') WHERE get_byte(data, 0) = 73;
+count
+-----
+ 3
+(1 row)
+
+?column?
+--------
+stop
+(1 row)
+
diff --git a/contrib/test_decoding/specs/invalidation_distribution.spec b/contrib/test_decoding/specs/invalidation_distribution.spec
index decbed627e3..1a90453f1ea 100644
--- a/contrib/test_decoding/specs/invalidation_distribution.spec
+++ b/contrib/test_decoding/specs/invalidation_distribution.spec
@@ -28,5 +28,26 @@ setup { SET synchronous_commit=on; }
step "s2_alter_pub_add_tbl" { ALTER PUBLICATION pub ADD TABLE tbl1; }
step "s2_get_binary_changes" { SELECT count(data) FROM pg_logical_slot_get_binary_changes('isolation_slot', NULL, NULL, 'proto_version', '4', 'publication_names', 'pub') WHERE get_byte(data, 0) = 73; }
+session "s3"
+step "s3i1" { INSERT INTO tbl1 (val1, val2) VALUES (1, 1);}
+step "s3a" { ALTER PUBLICATION pub ADD TABLE tbl1; }
+step "s3i2" { INSERT INTO tbl1 (val1, val2) VALUES (6, 6); }
+step "s3_get_binary_changes" { SELECT count(data) FROM pg_logical_slot_get_binary_changes('isolation_slot', NULL, NULL, 'proto_version', '4', 'publication_names', 'pub') WHERE get_byte(data, 0) = 73; }
+
+session "s4"
+step "s4b" { BEGIN; }
+step "s4i1" { INSERT INTO tbl1 (val1, val2) VALUES (2, 2);}
+step "s4i2" { INSERT INTO tbl1 (val1, val2) VALUES (4, 4); }
+step "s4c" { COMMIT; }
+step "s4i3" { INSERT INTO tbl1 (val1, val2) VALUES (5, 5); }
+
+session "s5"
+step "s5b" { BEGIN; }
+step "s5i1" { INSERT INTO tbl1 (val1, val2) VALUES (3, 3); }
+step "s5c" { COMMIT; }
+
# Expect to get one insert change. LOGICAL_REP_MSG_INSERT = 'I'
permutation "s1_insert_tbl1" "s1_begin" "s1_insert_tbl1" "s2_alter_pub_add_tbl" "s1_commit" "s1_insert_tbl1" "s2_get_binary_changes"
+
+# Expect to get three insert changes with LOGICAL_REP_MSG_INSERT = 'I' that happened after the table tbl1 is added to the publication.
+permutation "s3i1" "s4b" "s4i1" "s5b" "s5i1" "s3a" "s4i2" "s4c" "s5c" "s4i3" "s3i2" "s3_get_binary_changes"
diff --git a/src/backend/replication/logical/reorderbuffer.c b/src/backend/replication/logical/reorderbuffer.c
index 03eb005c39d..d85a9f2bb20 100644
--- a/src/backend/replication/logical/reorderbuffer.c
+++ b/src/backend/replication/logical/reorderbuffer.c
@@ -108,10 +108,22 @@
#include "storage/fd.h"
#include "storage/sinval.h"
#include "utils/builtins.h"
+#include "utils/inval.h"
#include "utils/memutils.h"
#include "utils/rel.h"
#include "utils/relfilenumbermap.h"
+/*
+ * Each transaction has an 8MB limit for invalidation messages distributed from
+ * other transactions. This limit is set considering scenarios with many
+ * concurrent logical decoding operations. When the distributed invalidation
+ * messages reach this threshold, the transaction is marked as
+ * RBTXN_DISTR_INVAL_OVERFLOWED to invalidate the complete cache as we have lost
+ * some inval messages and hence don't know what needs to be invalidated.
+ */
+#define MAX_DISTR_INVAL_MSG_PER_TXN \
+ ((8 * 1024 * 1024) / sizeof(SharedInvalidationMessage))
+
/* entry for a hash table we use to map from xid to our transaction state */
typedef struct ReorderBufferTXNByIdEnt
{
@@ -469,6 +481,12 @@ ReorderBufferReturnTXN(ReorderBuffer *rb, ReorderBufferTXN *txn)
txn->invalidations = NULL;
}
+ if (txn->invalidations_distributed)
+ {
+ pfree(txn->invalidations_distributed);
+ txn->invalidations_distributed = NULL;
+ }
+
/* Reset the toast hash */
ReorderBufferToastReset(rb, txn);
@@ -2574,7 +2592,17 @@ ReorderBufferProcessTXN(ReorderBuffer *rb, ReorderBufferTXN *txn,
AbortCurrentTransaction();
/* make sure there's no cache pollution */
- ReorderBufferExecuteInvalidations(txn->ninvalidations, txn->invalidations);
+ if (rbtxn_distr_inval_overflowed(txn))
+ {
+ Assert(txn->ninvalidations_distributed == 0);
+ InvalidateSystemCaches();
+ }
+ else
+ {
+ ReorderBufferExecuteInvalidations(txn->ninvalidations, txn->invalidations);
+ ReorderBufferExecuteInvalidations(txn->ninvalidations_distributed,
+ txn->invalidations_distributed);
+ }
if (using_subtxn)
RollbackAndReleaseCurrentSubTransaction();
@@ -2620,8 +2648,17 @@ ReorderBufferProcessTXN(ReorderBuffer *rb, ReorderBufferTXN *txn,
AbortCurrentTransaction();
/* make sure there's no cache pollution */
- ReorderBufferExecuteInvalidations(txn->ninvalidations,
- txn->invalidations);
+ if (rbtxn_distr_inval_overflowed(txn))
+ {
+ Assert(txn->ninvalidations_distributed == 0);
+ InvalidateSystemCaches();
+ }
+ else
+ {
+ ReorderBufferExecuteInvalidations(txn->ninvalidations, txn->invalidations);
+ ReorderBufferExecuteInvalidations(txn->ninvalidations_distributed,
+ txn->invalidations_distributed);
+ }
if (using_subtxn)
RollbackAndReleaseCurrentSubTransaction();
@@ -2951,7 +2988,8 @@ ReorderBufferAbort(ReorderBuffer *rb, TransactionId xid, XLogRecPtr lsn,
* We might have decoded changes for this transaction that could load
* the cache as per the current transaction's view (consider DDL's
* happened in this transaction). We don't want the decoding of future
- * transactions to use those cache entries so execute invalidations.
+ * transactions to use those cache entries so execute only the inval
+ * messages in this transaction.
*/
if (txn->ninvalidations > 0)
ReorderBufferImmediateInvalidation(rb, txn->ninvalidations,
@@ -3038,9 +3076,10 @@ ReorderBufferForget(ReorderBuffer *rb, TransactionId xid, XLogRecPtr lsn)
txn->final_lsn = lsn;
/*
- * Process cache invalidation messages if there are any. Even if we're not
- * interested in the transaction's contents, it could have manipulated the
- * catalog and we need to update the caches according to that.
+ * Process only cache invalidation messages in this transaction if there
+ * are any. Even if we're not interested in the transaction's contents, it
+ * could have manipulated the catalog and we need to update the caches
+ * according to that.
*/
if (txn->base_snapshot != NULL && txn->ninvalidations > 0)
ReorderBufferImmediateInvalidation(rb, txn->ninvalidations,
@@ -3312,6 +3351,57 @@ ReorderBufferAddNewTupleCids(ReorderBuffer *rb, TransactionId xid,
txn->ntuplecids++;
}
+/*
+ * Add new invalidation messages to the reorder buffer queue.
+ */
+static void
+ReorderBufferQueueInvalidations(ReorderBuffer *rb, TransactionId xid,
+ XLogRecPtr lsn, Size nmsgs,
+ SharedInvalidationMessage *msgs)
+{
+ ReorderBufferChange *change;
+
+ change = ReorderBufferGetChange(rb);
+ change->action = REORDER_BUFFER_CHANGE_INVALIDATION;
+ change->data.inval.ninvalidations = nmsgs;
+ change->data.inval.invalidations = (SharedInvalidationMessage *)
+ palloc(sizeof(SharedInvalidationMessage) * nmsgs);
+ memcpy(change->data.inval.invalidations, msgs,
+ sizeof(SharedInvalidationMessage) * nmsgs);
+
+ ReorderBufferQueueChange(rb, xid, lsn, change, false);
+}
+
+/*
+ * A helper function for ReorderBufferAddInvalidations() and
+ * ReorderBufferAddDistributedInvalidations() to accumulate the invalidation
+ * messages to the **invals_out.
+ */
+static void
+ReorderBufferAccumulateInvalidations(SharedInvalidationMessage **invals_out,
+ uint32 *ninvals_out,
+ SharedInvalidationMessage *msgs_new,
+ Size nmsgs_new)
+{
+ if (*ninvals_out == 0)
+ {
+ *ninvals_out = nmsgs_new;
+ *invals_out = (SharedInvalidationMessage *)
+ palloc(sizeof(SharedInvalidationMessage) * nmsgs_new);
+ memcpy(*invals_out, msgs_new, sizeof(SharedInvalidationMessage) * nmsgs_new);
+ }
+ else
+ {
+ /* Enlarge the array of inval messages */
+ *invals_out = (SharedInvalidationMessage *)
+ repalloc(*invals_out, sizeof(SharedInvalidationMessage) *
+ (*ninvals_out + nmsgs_new));
+ memcpy(*invals_out + *ninvals_out, msgs_new,
+ nmsgs_new * sizeof(SharedInvalidationMessage));
+ *ninvals_out += nmsgs_new;
+ }
+}
+
/*
* Accumulate the invalidations for executing them later.
*
@@ -3332,7 +3422,6 @@ ReorderBufferAddInvalidations(ReorderBuffer *rb, TransactionId xid,
{
ReorderBufferTXN *txn;
MemoryContext oldcontext;
- ReorderBufferChange *change;
txn = ReorderBufferTXNByXid(rb, xid, true, NULL, lsn, true);
@@ -3347,35 +3436,76 @@ ReorderBufferAddInvalidations(ReorderBuffer *rb, TransactionId xid,
Assert(nmsgs > 0);
- /* Accumulate invalidations. */
- if (txn->ninvalidations == 0)
- {
- txn->ninvalidations = nmsgs;
- txn->invalidations = (SharedInvalidationMessage *)
- palloc(sizeof(SharedInvalidationMessage) * nmsgs);
- memcpy(txn->invalidations, msgs,
- sizeof(SharedInvalidationMessage) * nmsgs);
- }
- else
+ ReorderBufferAccumulateInvalidations(&txn->invalidations,
+ &txn->ninvalidations,
+ msgs, nmsgs);
+
+ ReorderBufferQueueInvalidations(rb, xid, lsn, nmsgs, msgs);
+
+ MemoryContextSwitchTo(oldcontext);
+}
+
+/*
+ * Accumulate the invalidations distributed by other committed transactions
+ * for executing them later.
+ *
+ * This function is similar to ReorderBufferAddInvalidations() but stores
+ * the given inval messages to the txn->invalidations_distributed with the
+ * overflow check.
+ *
+ * This needs to be called by committed transactions to distribute their
+ * inval messages to in-progress transactions.
+ */
+void
+ReorderBufferAddDistributedInvalidations(ReorderBuffer *rb, TransactionId xid,
+ XLogRecPtr lsn, Size nmsgs,
+ SharedInvalidationMessage *msgs)
+{
+ ReorderBufferTXN *txn;
+ MemoryContext oldcontext;
+
+ txn = ReorderBufferTXNByXid(rb, xid, true, NULL, lsn, true);
+
+ oldcontext = MemoryContextSwitchTo(rb->context);
+
+ /*
+ * Collect all the invalidations under the top transaction, if available,
+ * so that we can execute them all together. See comments
+ * ReorderBufferAddInvalidations.
+ */
+ txn = rbtxn_get_toptxn(txn);
+
+ Assert(nmsgs > 0);
+
+ if (!rbtxn_distr_inval_overflowed(txn))
{
- txn->invalidations = (SharedInvalidationMessage *)
- repalloc(txn->invalidations, sizeof(SharedInvalidationMessage) *
- (txn->ninvalidations + nmsgs));
+ /*
+ * Check the transaction has enough space for storing distributed
+ * invalidation messages.
+ */
+ if (txn->ninvalidations_distributed + nmsgs >= MAX_DISTR_INVAL_MSG_PER_TXN)
+ {
+ /*
+ * Mark the invalidation message as overflowed and free up the
+ * messages accumulated so far.
+ */
+ txn->txn_flags |= RBTXN_DISTR_INVAL_OVERFLOWED;
- memcpy(txn->invalidations + txn->ninvalidations, msgs,
- nmsgs * sizeof(SharedInvalidationMessage));
- txn->ninvalidations += nmsgs;
+ if (txn->invalidations_distributed)
+ {
+ pfree(txn->invalidations_distributed);
+ txn->invalidations_distributed = NULL;
+ txn->ninvalidations_distributed = 0;
+ }
+ }
+ else
+ ReorderBufferAccumulateInvalidations(&txn->invalidations_distributed,
+ &txn->ninvalidations_distributed,
+ msgs, nmsgs);
}
- change = ReorderBufferGetChange(rb);
- change->action = REORDER_BUFFER_CHANGE_INVALIDATION;
- change->data.inval.ninvalidations = nmsgs;
- change->data.inval.invalidations = (SharedInvalidationMessage *)
- palloc(sizeof(SharedInvalidationMessage) * nmsgs);
- memcpy(change->data.inval.invalidations, msgs,
- sizeof(SharedInvalidationMessage) * nmsgs);
-
- ReorderBufferQueueChange(rb, xid, lsn, change, false);
+ /* Queue the invalidation messages into the transaction */
+ ReorderBufferQueueInvalidations(rb, xid, lsn, nmsgs, msgs);
MemoryContextSwitchTo(oldcontext);
}
diff --git a/src/backend/replication/logical/snapbuild.c b/src/backend/replication/logical/snapbuild.c
index 110e0b0a044..5c3bbf0e93f 100644
--- a/src/backend/replication/logical/snapbuild.c
+++ b/src/backend/replication/logical/snapbuild.c
@@ -932,6 +932,13 @@ SnapBuildDistributeSnapshotAndInval(SnapBuild *builder, XLogRecPtr lsn, Transact
* contents built by the current transaction even after its decoding,
* which should have been invalidated due to concurrent catalog
* changing transaction.
+ *
+ * Distribute only the invalidation messages generated by the current
+ * committed transaction. Invalidation messages received from other
+ * transactions would have already been propagated to the relevant
+ * in-progress transactions. This transaction would have processed
+ * those invalidations, ensuring that subsequent transactions observe
+ * a consistent cache state.
*/
if (txn->xid != xid)
{
@@ -945,8 +952,9 @@ SnapBuildDistributeSnapshotAndInval(SnapBuild *builder, XLogRecPtr lsn, Transact
{
Assert(msgs != NULL);
- ReorderBufferAddInvalidations(builder->reorder, txn->xid, lsn,
- ninvalidations, msgs);
+ ReorderBufferAddDistributedInvalidations(builder->reorder,
+ txn->xid, lsn,
+ ninvalidations, msgs);
}
}
}
diff --git a/src/include/replication/reorderbuffer.h b/src/include/replication/reorderbuffer.h
index 4c56f219fd8..0cfa9005141 100644
--- a/src/include/replication/reorderbuffer.h
+++ b/src/include/replication/reorderbuffer.h
@@ -159,15 +159,16 @@ typedef struct ReorderBufferChange
} ReorderBufferChange;
/* ReorderBufferTXN txn_flags */
-#define RBTXN_HAS_CATALOG_CHANGES 0x0001
-#define RBTXN_IS_SUBXACT 0x0002
-#define RBTXN_IS_SERIALIZED 0x0004
-#define RBTXN_IS_SERIALIZED_CLEAR 0x0008
-#define RBTXN_IS_STREAMED 0x0010
-#define RBTXN_HAS_PARTIAL_CHANGE 0x0020
-#define RBTXN_PREPARE 0x0040
-#define RBTXN_SKIPPED_PREPARE 0x0080
-#define RBTXN_HAS_STREAMABLE_CHANGE 0x0100
+#define RBTXN_HAS_CATALOG_CHANGES 0x0001
+#define RBTXN_IS_SUBXACT 0x0002
+#define RBTXN_IS_SERIALIZED 0x0004
+#define RBTXN_IS_SERIALIZED_CLEAR 0x0008
+#define RBTXN_IS_STREAMED 0x0010
+#define RBTXN_HAS_PARTIAL_CHANGE 0x0020
+#define RBTXN_PREPARE 0x0040
+#define RBTXN_SKIPPED_PREPARE 0x0080
+#define RBTXN_HAS_STREAMABLE_CHANGE 0x0100
+#define RBTXN_DISTR_INVAL_OVERFLOWED 0x0200
/* Does the transaction have catalog changes? */
#define rbtxn_has_catalog_changes(txn) \
@@ -231,6 +232,12 @@ typedef struct ReorderBufferChange
((txn)->txn_flags & RBTXN_SKIPPED_PREPARE) != 0 \
)
+/* Is the array of distributed inval messages overflowed? */
+#define rbtxn_distr_inval_overflowed(txn) \
+( \
+ ((txn)->txn_flags & RBTXN_DISTR_INVAL_OVERFLOWED) != 0 \
+)
+
/* Is this a top-level transaction? */
#define rbtxn_is_toptxn(txn) \
( \
@@ -422,6 +429,12 @@ typedef struct ReorderBufferTXN
* Private data pointer of the output plugin.
*/
void *output_plugin_private;
+
+ /*
+ * Stores cache invalidation messages distributed by other transactions.
+ */
+ uint32 ninvalidations_distributed;
+ SharedInvalidationMessage *invalidations_distributed;
} ReorderBufferTXN;
/* so we can define the callbacks used inside struct ReorderBuffer itself */
@@ -709,6 +722,9 @@ extern void ReorderBufferAddNewTupleCids(ReorderBuffer *rb, TransactionId xid,
CommandId cmin, CommandId cmax, CommandId combocid);
extern void ReorderBufferAddInvalidations(ReorderBuffer *rb, TransactionId xid, XLogRecPtr lsn,
Size nmsgs, SharedInvalidationMessage *msgs);
+extern void ReorderBufferAddDistributedInvalidations(ReorderBuffer *rb, TransactionId xid,
+ XLogRecPtr lsn, Size nmsgs,
+ SharedInvalidationMessage *msgs);
extern void ReorderBufferImmediateInvalidation(ReorderBuffer *rb, uint32 ninvalidations,
SharedInvalidationMessage *invalidations);
extern void ReorderBufferProcessXid(ReorderBuffer *rb, TransactionId xid, XLogRecPtr lsn);
--
2.43.0
^ permalink raw reply [nested|flat] 94+ messages in thread
* Re: Logical replication 'invalid memory alloc request size 1585837200' after upgrading to 17.5
@ 2025-06-10 19:19 Masahiko Sawada <sawada.mshk@gmail.com>
parent: vignesh C <vignesh21@gmail.com>
1 sibling, 1 reply; 94+ messages in thread
From: Masahiko Sawada @ 2025-06-10 19:19 UTC (permalink / raw)
To: vignesh C <vignesh21@gmail.com>; +Cc: Amit Kapila <amit.kapila16@gmail.com>; Hayato Kuroda (Fujitsu) <kuroda.hayato@fujitsu.com>; Duncan Sands <duncan.sands@deepbluecap.com>; pgsql-bugs@lists.postgresql.org <pgsql-bugs@lists.postgresql.org>
On Sun, Jun 8, 2025 at 6:43 AM vignesh C <vignesh21@gmail.com> wrote:
>
> On Sat, 7 Jun 2025 at 12:47, vignesh C <vignesh21@gmail.com> wrote:
> >
> > On Fri, 6 Jun 2025 at 22:51, Masahiko Sawada <sawada.mshk@gmail.com> wrote:
> > >
> > > On Thu, Jun 5, 2025 at 8:43 PM Amit Kapila <amit.kapila16@gmail.com> wrote:
> > > >
> > > > On Thu, Jun 5, 2025 at 11:14 PM Masahiko Sawada <sawada.mshk@gmail.com> wrote:
> > > > >
> > > > > On Wed, Jun 4, 2025 at 11:36 PM Hayato Kuroda (Fujitsu)
> > > > > <kuroda.hayato@fujitsu.com> wrote:
> > > > > >
> > > > > > Dear Sawada-san,
> > > > > >
> > > > > > > Alternative idea is to lower the constant value when using an
> > > > > > > assertion build. That way, we don't need to rely on injection points
> > > > > > > being enabled.
> > > > > >
> > > > > > Hmm, possible but I prefer current one. Two concerns:
> > > > > >
> > > > > > 1.
> > > > > > USE_ASSERT_CHECKING has not been used to change the value yet. The main usage is
> > > > > > to call debug functions in debug build.
> > > > >
> > > > > I think we have a similar precedent such as MT_NRELS_HASH to improve
> > > > > the test coverages.
> > > > >
> > > > > > 2.
> > > > > > If we add tests which is usable only for debug build, it must be run only when it
> > > > > > is enabled. IIUC such test does not exist yet.
> > > > >
> > > > > I think we need to test cases not to check if we reach a specific code
> > > > > point but to check if we can get the correct results even if we've
> > > > > executed various code paths. As for this bug, it is better to check
> > > > > that it works properly in a variety of cases. That way, we can check
> > > > > overflow cases and non-overflow cases also in test cases added in the
> > > > > future, improving the test coverage more.
> > > > >
> > > >
> > > > This makes sense, but we should see whether some existing tests cover
> > > > this code path after lowering the limit in the overflow code path. One
> > > > minor point to consider is that at the time, the value MT_NRELS_HASH
> > > > was used to cover cases in a debug build, but we didn't have the
> > > > injection_point framework.
> > >
> > > True.
> > >
> > > After thinking about it more, perhaps my proposal would not be a good
> > > idea for this case. I think that the cases where we selectively
> > > invalidate caches is more complex and error-prone than the cases where
> > > we invalidate a complete cache. If we invalidated all caches after
> > > decoding each transaction, we wouldn't have had the original data-loss
> > > issue. Having a lower MAX_DISTR_INVAL_MSG_PER_TXN value when using an
> > > assertio build means that we're going to test the cases using a
> > > simpler invalidation mechanism while productions systems, which has a
> > > higher MAX_DISTR_INVAL_MSG_PER_TXN value, would end up executing
> > > complex cases, which is not great. What do you think?
> > >
> > > BTW, as for a new test case, it might be worth having a case I
> > > mentioned before[1]:
> > >
> > > 1) S1: CREATE TABLE d (data text not null);
> > > 2) S1: INSERT INTO d VALUES ('d1');
> > > 3) S2: BEGIN; INSERT INTO d VALUES ('d2');
> > > 4) S3: BEGIN; INSERT INTO d VALUES ('d3');
> > > 5) S1: ALTER PUBLICATION pb ADD TABLE d;
> > > 6) S2: INSERT INTO d VALUES ('d4');
> > > 7) S2: COMMIT;
> > > 8) S3: COMMIT;
> > > 9) S2: INSERT INTO d VALUES('d5');
> > > 10) S1: INSERT INTO d VALUES ('d6');
> > >
> > > With this case, we can test if we need to execute the distributed
> > > invalidations as well in the non-error path in
> > > ReorderBufferProcessTXN().
> >
> > The attached v13 version patch has the change to include this test case.
>
> Attached are the patches, including those required for the back branches.
Thank you for updating the patches! I'll review them and share comments if any.
Regards,
--
Masahiko Sawada
Amazon Web Services: https://aws.amazon.com
^ permalink raw reply [nested|flat] 94+ messages in thread
* RE: Logical replication 'invalid memory alloc request size 1585837200' after upgrading to 17.5
@ 2025-06-11 01:45 Hayato Kuroda (Fujitsu) <kuroda.hayato@fujitsu.com>
parent: vignesh C <vignesh21@gmail.com>
1 sibling, 1 reply; 94+ messages in thread
From: Hayato Kuroda (Fujitsu) @ 2025-06-11 01:45 UTC (permalink / raw)
To: vignesh C <vignesh21@gmail.com>; Masahiko Sawada <sawada.mshk@gmail.com>; +Cc: Amit Kapila <amit.kapila16@gmail.com>; Duncan Sands <duncan.sands@deepbluecap.com>; pgsql-bugs@lists.postgresql.org <pgsql-bugs@lists.postgresql.org>
Dear hackers,
> Attached are the patches, including those required for the back branches.
While reviewing patch for PG13, I found the doubtful point in ReorderBufferCommit().
```
/*
* Every time the CommandId is incremented, we could
* see new catalog contents, so execute all
* invalidations.
*/
ReorderBufferExecuteInvalidations(txn->ninvalidations,
txn->invalidations);
```
This is called when REORDER_BUFFER_CHANGE_INTERNAL_COMMAND_ID is dequeued from the
change queue, and this part exists only in PG13 codebase.
We are not sure whether we should execute txn->invalidations_distributed as well.
This can affect below case:
txn1: BEGIN; INSERT INTO d VALUES ('d1');
txn2: ALTER PUBLICATION pb ADD TABLE d;
txn1: CREATE TABLE another (id int);
txn1: INSERT INTO d VALUES ('d2');
txn1: COMMIT;
-> PG13 - no output
-> PG13 + v13 patch - no output
-> PG13 + v13 patch + additional inval execution - d2 can be replicated
-> (master - d2 can be replicated)
Personally I think txn->invalidations_distributed is not needed to be executed
because the spec seems bit complex, but I want to know other opinion.
Best regards,
Hayato Kuroda
FUJITSU LIMITED
^ permalink raw reply [nested|flat] 94+ messages in thread
* Re: Logical replication 'invalid memory alloc request size 1585837200' after upgrading to 17.5
@ 2025-06-11 03:19 Amit Kapila <amit.kapila16@gmail.com>
parent: Hayato Kuroda (Fujitsu) <kuroda.hayato@fujitsu.com>
0 siblings, 0 replies; 94+ messages in thread
From: Amit Kapila @ 2025-06-11 03:19 UTC (permalink / raw)
To: Hayato Kuroda (Fujitsu) <kuroda.hayato@fujitsu.com>; +Cc: vignesh C <vignesh21@gmail.com>; Masahiko Sawada <sawada.mshk@gmail.com>; Duncan Sands <duncan.sands@deepbluecap.com>; pgsql-bugs@lists.postgresql.org <pgsql-bugs@lists.postgresql.org>
On Wed, Jun 11, 2025 at 7:15 AM Hayato Kuroda (Fujitsu)
<kuroda.hayato@fujitsu.com> wrote:
>
> While reviewing patch for PG13, I found the doubtful point in ReorderBufferCommit().
>
> ```
> /*
> * Every time the CommandId is incremented, we could
> * see new catalog contents, so execute all
> * invalidations.
> */
> ReorderBufferExecuteInvalidations(txn->ninvalidations,
> txn->invalidations);
> ```
>
> This is called when REORDER_BUFFER_CHANGE_INTERNAL_COMMAND_ID is dequeued from the
> change queue, and this part exists only in PG13 codebase.
> We are not sure whether we should execute txn->invalidations_distributed as well.
> This can affect below case:
>
>
> txn1: BEGIN; INSERT INTO d VALUES ('d1');
> txn2: ALTER PUBLICATION pb ADD TABLE d;
> txn1: CREATE TABLE another (id int);
> txn1: INSERT INTO d VALUES ('d2');
> txn1: COMMIT;
> -> PG13 - no output
> -> PG13 + v13 patch - no output
> -> PG13 + v13 patch + additional inval execution - d2 can be replicated
> -> (master - d2 can be replicated)
>
> Personally I think txn->invalidations_distributed is not needed to be executed
> because the spec seems bit complex, but I want to know other opinion.
>
This is not new; we knew from the time we committed this in PG13 that
concurrent transactions wouldn't pick up DDL changes, but later
transactions should. See commit message of commit
247ee94150b6fe8906da51afadbedf8acf3c17cf in PG13 ("The fix for 13 is
different from what we did in branches 14 and above, such that for 13,
the concurrent DDL changes (from DDL types mentioned earlier) will be
visible for any newly started transactions...). So, the above is
expected behavior for PG-13.
--
With Regards,
Amit Kapila.
^ permalink raw reply [nested|flat] 94+ messages in thread
* Re: Logical replication 'invalid memory alloc request size 1585837200' after upgrading to 17.5
@ 2025-06-11 22:03 Masahiko Sawada <sawada.mshk@gmail.com>
parent: Masahiko Sawada <sawada.mshk@gmail.com>
0 siblings, 2 replies; 94+ messages in thread
From: Masahiko Sawada @ 2025-06-11 22:03 UTC (permalink / raw)
To: vignesh C <vignesh21@gmail.com>; +Cc: Amit Kapila <amit.kapila16@gmail.com>; Hayato Kuroda (Fujitsu) <kuroda.hayato@fujitsu.com>; Duncan Sands <duncan.sands@deepbluecap.com>; pgsql-bugs@lists.postgresql.org <pgsql-bugs@lists.postgresql.org>
On Tue, Jun 10, 2025 at 12:19 PM Masahiko Sawada <sawada.mshk@gmail.com> wrote:
>
> On Sun, Jun 8, 2025 at 6:43 AM vignesh C <vignesh21@gmail.com> wrote:
> >
> > On Sat, 7 Jun 2025 at 12:47, vignesh C <vignesh21@gmail.com> wrote:
> > >
> > > On Fri, 6 Jun 2025 at 22:51, Masahiko Sawada <sawada.mshk@gmail.com> wrote:
> > > >
> > > > On Thu, Jun 5, 2025 at 8:43 PM Amit Kapila <amit.kapila16@gmail.com> wrote:
> > > > >
> > > > > On Thu, Jun 5, 2025 at 11:14 PM Masahiko Sawada <sawada.mshk@gmail.com> wrote:
> > > > > >
> > > > > > On Wed, Jun 4, 2025 at 11:36 PM Hayato Kuroda (Fujitsu)
> > > > > > <kuroda.hayato@fujitsu.com> wrote:
> > > > > > >
> > > > > > > Dear Sawada-san,
> > > > > > >
> > > > > > > > Alternative idea is to lower the constant value when using an
> > > > > > > > assertion build. That way, we don't need to rely on injection points
> > > > > > > > being enabled.
> > > > > > >
> > > > > > > Hmm, possible but I prefer current one. Two concerns:
> > > > > > >
> > > > > > > 1.
> > > > > > > USE_ASSERT_CHECKING has not been used to change the value yet. The main usage is
> > > > > > > to call debug functions in debug build.
> > > > > >
> > > > > > I think we have a similar precedent such as MT_NRELS_HASH to improve
> > > > > > the test coverages.
> > > > > >
> > > > > > > 2.
> > > > > > > If we add tests which is usable only for debug build, it must be run only when it
> > > > > > > is enabled. IIUC such test does not exist yet.
> > > > > >
> > > > > > I think we need to test cases not to check if we reach a specific code
> > > > > > point but to check if we can get the correct results even if we've
> > > > > > executed various code paths. As for this bug, it is better to check
> > > > > > that it works properly in a variety of cases. That way, we can check
> > > > > > overflow cases and non-overflow cases also in test cases added in the
> > > > > > future, improving the test coverage more.
> > > > > >
> > > > >
> > > > > This makes sense, but we should see whether some existing tests cover
> > > > > this code path after lowering the limit in the overflow code path. One
> > > > > minor point to consider is that at the time, the value MT_NRELS_HASH
> > > > > was used to cover cases in a debug build, but we didn't have the
> > > > > injection_point framework.
> > > >
> > > > True.
> > > >
> > > > After thinking about it more, perhaps my proposal would not be a good
> > > > idea for this case. I think that the cases where we selectively
> > > > invalidate caches is more complex and error-prone than the cases where
> > > > we invalidate a complete cache. If we invalidated all caches after
> > > > decoding each transaction, we wouldn't have had the original data-loss
> > > > issue. Having a lower MAX_DISTR_INVAL_MSG_PER_TXN value when using an
> > > > assertio build means that we're going to test the cases using a
> > > > simpler invalidation mechanism while productions systems, which has a
> > > > higher MAX_DISTR_INVAL_MSG_PER_TXN value, would end up executing
> > > > complex cases, which is not great. What do you think?
> > > >
> > > > BTW, as for a new test case, it might be worth having a case I
> > > > mentioned before[1]:
> > > >
> > > > 1) S1: CREATE TABLE d (data text not null);
> > > > 2) S1: INSERT INTO d VALUES ('d1');
> > > > 3) S2: BEGIN; INSERT INTO d VALUES ('d2');
> > > > 4) S3: BEGIN; INSERT INTO d VALUES ('d3');
> > > > 5) S1: ALTER PUBLICATION pb ADD TABLE d;
> > > > 6) S2: INSERT INTO d VALUES ('d4');
> > > > 7) S2: COMMIT;
> > > > 8) S3: COMMIT;
> > > > 9) S2: INSERT INTO d VALUES('d5');
> > > > 10) S1: INSERT INTO d VALUES ('d6');
> > > >
> > > > With this case, we can test if we need to execute the distributed
> > > > invalidations as well in the non-error path in
> > > > ReorderBufferProcessTXN().
> > >
> > > The attached v13 version patch has the change to include this test case.
> >
> > Attached are the patches, including those required for the back branches.
>
> Thank you for updating the patches! I'll review them and share comments if any.
Thank you for updating the patch. I have one comment on the newly added test:
+session "s3"
+step "s3i1" { INSERT INTO tbl1 (val1, val2) VALUES (1, 1);}
+step "s3a" { ALTER PUBLICATION pub ADD TABLE tbl1; }
+step "s3i2" { INSERT INTO tbl1 (val1, val2) VALUES (6, 6); }
+step "s3_get_binary_changes" { SELECT count(data) FROM
pg_logical_slot_get_binary_changes('isolation_slot', NULL, NULL,
'proto_version', '4', 'publication_names', 'pub') WHERE get_byte(data,
0) = 73; }
+
+session "s4"
+step "s4b" { BEGIN; }
+step "s4i1" { INSERT INTO tbl1 (val1, val2) VALUES (2, 2);}
+step "s4i2" { INSERT INTO tbl1 (val1, val2) VALUES (4, 4); }
+step "s4c" { COMMIT; }
+step "s4i3" { INSERT INTO tbl1 (val1, val2) VALUES (5, 5); }
+
+session "s5"
+step "s5b" { BEGIN; }
+step "s5i1" { INSERT INTO tbl1 (val1, val2) VALUES (3, 3); }
+step "s5c" { COMMIT; }
I think we don't necessarily need to add sessions "s4" and "s5". Let's
reuse "s1" and "s2" instead of adding them. I've attached a patch to
change that.
Regards,
--
Masahiko Sawada
Amazon Web Services: https://aws.amazon.com
Attachments:
[application/octet-stream] v13_test_change.patch (4.4K, ../../CAD21AoDCphZ30iiMOG8mZH_eo40m_1KY=4gz7-Yd+9M3FRBySg@mail.gmail.com/2-v13_test_change.patch)
download | inline diff:
diff --git a/contrib/test_decoding/expected/invalidation_distribution.out b/contrib/test_decoding/expected/invalidation_distribution.out
index 9fe87e2cda9..ae53b1e61de 100644
--- a/contrib/test_decoding/expected/invalidation_distribution.out
+++ b/contrib/test_decoding/expected/invalidation_distribution.out
@@ -1,4 +1,4 @@
-Parsed test spec with 5 sessions
+Parsed test spec with 3 sessions
starting permutation: s1_insert_tbl1 s1_begin s1_insert_tbl1 s2_alter_pub_add_tbl s1_commit s1_insert_tbl1 s2_get_binary_changes
step s1_insert_tbl1: INSERT INTO tbl1 (val1, val2) VALUES (1, 1);
@@ -19,22 +19,19 @@ stop
(1 row)
-starting permutation: s3i1 s4b s4i1 s5b s5i1 s3a s4i2 s4c s5c s4i3 s3i2 s3_get_binary_changes
-step s3i1: INSERT INTO tbl1 (val1, val2) VALUES (1, 1);
-step s4b: BEGIN;
-step s4i1: INSERT INTO tbl1 (val1, val2) VALUES (2, 2);
-step s5b: BEGIN;
-step s5i1: INSERT INTO tbl1 (val1, val2) VALUES (3, 3);
-step s3a: ALTER PUBLICATION pub ADD TABLE tbl1;
-step s4i2: INSERT INTO tbl1 (val1, val2) VALUES (4, 4);
-step s4c: COMMIT;
-step s5c: COMMIT;
-step s4i3: INSERT INTO tbl1 (val1, val2) VALUES (5, 5);
-step s3i2: INSERT INTO tbl1 (val1, val2) VALUES (6, 6);
-step s3_get_binary_changes: SELECT count(data) FROM pg_logical_slot_get_binary_changes('isolation_slot', NULL, NULL, 'proto_version', '4', 'publication_names', 'pub') WHERE get_byte(data, 0) = 73;
+starting permutation: s1_begin s1_insert_tbl1 s3_begin s3_insert_tbl1 s2_alter_pub_add_tbl s1_insert_tbl1 s1_commit s3_commit s2_get_binary_changes
+step s1_begin: BEGIN;
+step s1_insert_tbl1: INSERT INTO tbl1 (val1, val2) VALUES (1, 1);
+step s3_begin: BEGIN;
+step s3_insert_tbl1: INSERT INTO tbl1 (val1, val2) VALUES (2, 2);
+step s2_alter_pub_add_tbl: ALTER PUBLICATION pub ADD TABLE tbl1;
+step s1_insert_tbl1: INSERT INTO tbl1 (val1, val2) VALUES (1, 1);
+step s1_commit: COMMIT;
+step s3_commit: COMMIT;
+step s2_get_binary_changes: SELECT count(data) FROM pg_logical_slot_get_binary_changes('isolation_slot', NULL, NULL, 'proto_version', '4', 'publication_names', 'pub') WHERE get_byte(data, 0) = 73;
count
-----
- 3
+ 1
(1 row)
?column?
diff --git a/contrib/test_decoding/specs/invalidation_distribution.spec b/contrib/test_decoding/specs/invalidation_distribution.spec
index 1a90453f1ea..f73cf6e482a 100644
--- a/contrib/test_decoding/specs/invalidation_distribution.spec
+++ b/contrib/test_decoding/specs/invalidation_distribution.spec
@@ -29,25 +29,15 @@ step "s2_alter_pub_add_tbl" { ALTER PUBLICATION pub ADD TABLE tbl1; }
step "s2_get_binary_changes" { SELECT count(data) FROM pg_logical_slot_get_binary_changes('isolation_slot', NULL, NULL, 'proto_version', '4', 'publication_names', 'pub') WHERE get_byte(data, 0) = 73; }
session "s3"
-step "s3i1" { INSERT INTO tbl1 (val1, val2) VALUES (1, 1);}
-step "s3a" { ALTER PUBLICATION pub ADD TABLE tbl1; }
-step "s3i2" { INSERT INTO tbl1 (val1, val2) VALUES (6, 6); }
-step "s3_get_binary_changes" { SELECT count(data) FROM pg_logical_slot_get_binary_changes('isolation_slot', NULL, NULL, 'proto_version', '4', 'publication_names', 'pub') WHERE get_byte(data, 0) = 73; }
-
-session "s4"
-step "s4b" { BEGIN; }
-step "s4i1" { INSERT INTO tbl1 (val1, val2) VALUES (2, 2);}
-step "s4i2" { INSERT INTO tbl1 (val1, val2) VALUES (4, 4); }
-step "s4c" { COMMIT; }
-step "s4i3" { INSERT INTO tbl1 (val1, val2) VALUES (5, 5); }
-
-session "s5"
-step "s5b" { BEGIN; }
-step "s5i1" { INSERT INTO tbl1 (val1, val2) VALUES (3, 3); }
-step "s5c" { COMMIT; }
+setup { SET synchronous_commit=on; }
+step "s3_begin" { BEGIN; }
+step "s3_insert_tbl1" { INSERT INTO tbl1 (val1, val2) VALUES (2, 2); }
+step "s3_commit" { COMMIT; }
# Expect to get one insert change. LOGICAL_REP_MSG_INSERT = 'I'
permutation "s1_insert_tbl1" "s1_begin" "s1_insert_tbl1" "s2_alter_pub_add_tbl" "s1_commit" "s1_insert_tbl1" "s2_get_binary_changes"
-# Expect to get three insert changes with LOGICAL_REP_MSG_INSERT = 'I' that happened after the table tbl1 is added to the publication.
-permutation "s3i1" "s4b" "s4i1" "s5b" "s5i1" "s3a" "s4i2" "s4c" "s5c" "s4i3" "s3i2" "s3_get_binary_changes"
+# Expect to get one insert change with LOGICAL_REP_MSG_INSERT = 'I' from
+# the second "s1_insert_tbl1" executed after adding the table tbl1 the
+# publication in "s2_alter_pub_add_tbl".
+permutation "s1_begin" "s1_insert_tbl1" "s3_begin" "s3_insert_tbl1" "s2_alter_pub_add_tbl" "s1_insert_tbl1" "s1_commit" "s3_commit" "s2_get_binary_changes"
^ permalink raw reply [nested|flat] 94+ messages in thread
* Re: Logical replication 'invalid memory alloc request size 1585837200' after upgrading to 17.5
@ 2025-06-12 06:47 vignesh C <vignesh21@gmail.com>
parent: Masahiko Sawada <sawada.mshk@gmail.com>
1 sibling, 1 reply; 94+ messages in thread
From: vignesh C @ 2025-06-12 06:47 UTC (permalink / raw)
To: Masahiko Sawada <sawada.mshk@gmail.com>; +Cc: Amit Kapila <amit.kapila16@gmail.com>; Hayato Kuroda (Fujitsu) <kuroda.hayato@fujitsu.com>; Duncan Sands <duncan.sands@deepbluecap.com>; pgsql-bugs@lists.postgresql.org <pgsql-bugs@lists.postgresql.org>
On Thu, 12 Jun 2025 at 03:33, Masahiko Sawada <sawada.mshk@gmail.com> wrote:
>
> On Tue, Jun 10, 2025 at 12:19 PM Masahiko Sawada <sawada.mshk@gmail.com> wrote:
> >
> > On Sun, Jun 8, 2025 at 6:43 AM vignesh C <vignesh21@gmail.com> wrote:
> > >
> > > On Sat, 7 Jun 2025 at 12:47, vignesh C <vignesh21@gmail.com> wrote:
> > > >
> > > > On Fri, 6 Jun 2025 at 22:51, Masahiko Sawada <sawada.mshk@gmail.com> wrote:
> > > > >
> > > > > On Thu, Jun 5, 2025 at 8:43 PM Amit Kapila <amit.kapila16@gmail.com> wrote:
> > > > > >
> > > > > > On Thu, Jun 5, 2025 at 11:14 PM Masahiko Sawada <sawada.mshk@gmail.com> wrote:
> > > > > > >
> > > > > > > On Wed, Jun 4, 2025 at 11:36 PM Hayato Kuroda (Fujitsu)
> > > > > > > <kuroda.hayato@fujitsu.com> wrote:
> > > > > > > >
> > > > > > > > Dear Sawada-san,
> > > > > > > >
> > > > > > > > > Alternative idea is to lower the constant value when using an
> > > > > > > > > assertion build. That way, we don't need to rely on injection points
> > > > > > > > > being enabled.
> > > > > > > >
> > > > > > > > Hmm, possible but I prefer current one. Two concerns:
> > > > > > > >
> > > > > > > > 1.
> > > > > > > > USE_ASSERT_CHECKING has not been used to change the value yet. The main usage is
> > > > > > > > to call debug functions in debug build.
> > > > > > >
> > > > > > > I think we have a similar precedent such as MT_NRELS_HASH to improve
> > > > > > > the test coverages.
> > > > > > >
> > > > > > > > 2.
> > > > > > > > If we add tests which is usable only for debug build, it must be run only when it
> > > > > > > > is enabled. IIUC such test does not exist yet.
> > > > > > >
> > > > > > > I think we need to test cases not to check if we reach a specific code
> > > > > > > point but to check if we can get the correct results even if we've
> > > > > > > executed various code paths. As for this bug, it is better to check
> > > > > > > that it works properly in a variety of cases. That way, we can check
> > > > > > > overflow cases and non-overflow cases also in test cases added in the
> > > > > > > future, improving the test coverage more.
> > > > > > >
> > > > > >
> > > > > > This makes sense, but we should see whether some existing tests cover
> > > > > > this code path after lowering the limit in the overflow code path. One
> > > > > > minor point to consider is that at the time, the value MT_NRELS_HASH
> > > > > > was used to cover cases in a debug build, but we didn't have the
> > > > > > injection_point framework.
> > > > >
> > > > > True.
> > > > >
> > > > > After thinking about it more, perhaps my proposal would not be a good
> > > > > idea for this case. I think that the cases where we selectively
> > > > > invalidate caches is more complex and error-prone than the cases where
> > > > > we invalidate a complete cache. If we invalidated all caches after
> > > > > decoding each transaction, we wouldn't have had the original data-loss
> > > > > issue. Having a lower MAX_DISTR_INVAL_MSG_PER_TXN value when using an
> > > > > assertio build means that we're going to test the cases using a
> > > > > simpler invalidation mechanism while productions systems, which has a
> > > > > higher MAX_DISTR_INVAL_MSG_PER_TXN value, would end up executing
> > > > > complex cases, which is not great. What do you think?
> > > > >
> > > > > BTW, as for a new test case, it might be worth having a case I
> > > > > mentioned before[1]:
> > > > >
> > > > > 1) S1: CREATE TABLE d (data text not null);
> > > > > 2) S1: INSERT INTO d VALUES ('d1');
> > > > > 3) S2: BEGIN; INSERT INTO d VALUES ('d2');
> > > > > 4) S3: BEGIN; INSERT INTO d VALUES ('d3');
> > > > > 5) S1: ALTER PUBLICATION pb ADD TABLE d;
> > > > > 6) S2: INSERT INTO d VALUES ('d4');
> > > > > 7) S2: COMMIT;
> > > > > 8) S3: COMMIT;
> > > > > 9) S2: INSERT INTO d VALUES('d5');
> > > > > 10) S1: INSERT INTO d VALUES ('d6');
> > > > >
> > > > > With this case, we can test if we need to execute the distributed
> > > > > invalidations as well in the non-error path in
> > > > > ReorderBufferProcessTXN().
> > > >
> > > > The attached v13 version patch has the change to include this test case.
> > >
> > > Attached are the patches, including those required for the back branches.
> >
> > Thank you for updating the patches! I'll review them and share comments if any.
>
> Thank you for updating the patch. I have one comment on the newly added test:
>
> +session "s3"
> +step "s3i1" { INSERT INTO tbl1 (val1, val2) VALUES (1, 1);}
> +step "s3a" { ALTER PUBLICATION pub ADD TABLE tbl1; }
> +step "s3i2" { INSERT INTO tbl1 (val1, val2) VALUES (6, 6); }
> +step "s3_get_binary_changes" { SELECT count(data) FROM
> pg_logical_slot_get_binary_changes('isolation_slot', NULL, NULL,
> 'proto_version', '4', 'publication_names', 'pub') WHERE get_byte(data,
> 0) = 73; }
> +
> +session "s4"
> +step "s4b" { BEGIN; }
> +step "s4i1" { INSERT INTO tbl1 (val1, val2) VALUES (2, 2);}
> +step "s4i2" { INSERT INTO tbl1 (val1, val2) VALUES (4, 4); }
> +step "s4c" { COMMIT; }
> +step "s4i3" { INSERT INTO tbl1 (val1, val2) VALUES (5, 5); }
> +
> +session "s5"
> +step "s5b" { BEGIN; }
> +step "s5i1" { INSERT INTO tbl1 (val1, val2) VALUES (3, 3); }
> +step "s5c" { COMMIT; }
>
> I think we don't necessarily need to add sessions "s4" and "s5". Let's
> reuse "s1" and "s2" instead of adding them. I've attached a patch to
> change that.
Thanks, this is better.
In the case of PG13 I have slightly changed the test to do the insert
after the commit, because the queuing of invalidation messages into
reorder buffer queue is not supported in PG13. This limitation is due
to the absence of support for REORDER_BUFFER_CHANGE_INVALIDATION which
is already present in >= PG14
The attached v14 version patch has the changes for the same.
Regards,
Vignesh
Attachments:
[text/x-patch] v14-PG14-0001-Fix-exponential-memory-allocation-issue-in-.patch (17.3K, ../../CALDaNm3m6Dq2d6bCBaOjnABYavT3p+=VbJd5oJKok0sCTOSZog@mail.gmail.com/2-v14-PG14-0001-Fix-exponential-memory-allocation-issue-in-.patch)
download | inline diff:
From 6f62afc22ad22f1fcf5882afc32accc93af822e1 Mon Sep 17 00:00:00 2001
From: Vignesh C <vignesh21@gmail.com>
Date: Thu, 12 Jun 2025 11:08:32 +0530
Subject: [PATCH v14-PG14] Fix exponential memory allocation issue in logical
decoding due to invalidation message distribution
Commit 4909b38af0 added logic to distribute invalidation messages from
one transaction to all concurrent in-progress transactions. However,
since the source of each invalidation message was not tracked, transactions
could redundantly re-distribute already propagated messages. This led to an
exponential increase in invalidation messages and, ultimately, memory
allocation failures.
To fix this, a new field is introduced to track distributed invalidation
messages separately. This ensures that only transactions in the pre-existing
list receive the invalidations, preventing redundant propagation.
The accumulated distributed invalidations are now consumed at the end of
transaction processing in `ReorderBufferProcessTXN()`.
---
.../expected/invalidation_distribution.out | 23 +-
.../specs/invalidation_distribution.spec | 11 +
.../replication/logical/reorderbuffer.c | 197 +++++++++++++++---
src/backend/replication/logical/snapbuild.c | 12 +-
src/include/replication/reorderbuffer.h | 32 ++-
5 files changed, 231 insertions(+), 44 deletions(-)
diff --git a/contrib/test_decoding/expected/invalidation_distribution.out b/contrib/test_decoding/expected/invalidation_distribution.out
index c701e290bb9..f2bce5ce75c 100644
--- a/contrib/test_decoding/expected/invalidation_distribution.out
+++ b/contrib/test_decoding/expected/invalidation_distribution.out
@@ -1,4 +1,4 @@
-Parsed test spec with 2 sessions
+Parsed test spec with 3 sessions
starting permutation: s1_insert_tbl1 s1_begin s1_insert_tbl1 s2_alter_pub_add_tbl s1_commit s1_insert_tbl1 s2_get_binary_changes
step s1_insert_tbl1: INSERT INTO tbl1 (val1, val2) VALUES (1, 1);
@@ -18,3 +18,24 @@ count
stop
(1 row)
+
+starting permutation: s1_begin s1_insert_tbl1 s3_begin s3_insert_tbl1 s2_alter_pub_add_tbl s1_insert_tbl1 s1_commit s3_commit s2_get_binary_changes
+step s1_begin: BEGIN;
+step s1_insert_tbl1: INSERT INTO tbl1 (val1, val2) VALUES (1, 1);
+step s3_begin: BEGIN;
+step s3_insert_tbl1: INSERT INTO tbl1 (val1, val2) VALUES (2, 2);
+step s2_alter_pub_add_tbl: ALTER PUBLICATION pub ADD TABLE tbl1;
+step s1_insert_tbl1: INSERT INTO tbl1 (val1, val2) VALUES (1, 1);
+step s1_commit: COMMIT;
+step s3_commit: COMMIT;
+step s2_get_binary_changes: SELECT count(data) FROM pg_logical_slot_get_binary_changes('isolation_slot', NULL, NULL, 'proto_version', '2', 'publication_names', 'pub') WHERE get_byte(data, 0) = 73;
+count
+-----
+ 1
+(1 row)
+
+?column?
+--------
+stop
+(1 row)
+
diff --git a/contrib/test_decoding/specs/invalidation_distribution.spec b/contrib/test_decoding/specs/invalidation_distribution.spec
index b8b14e333a1..89179e0eca1 100644
--- a/contrib/test_decoding/specs/invalidation_distribution.spec
+++ b/contrib/test_decoding/specs/invalidation_distribution.spec
@@ -28,5 +28,16 @@ setup { SET synchronous_commit=on; }
step "s2_alter_pub_add_tbl" { ALTER PUBLICATION pub ADD TABLE tbl1; }
step "s2_get_binary_changes" { SELECT count(data) FROM pg_logical_slot_get_binary_changes('isolation_slot', NULL, NULL, 'proto_version', '2', 'publication_names', 'pub') WHERE get_byte(data, 0) = 73; }
+session "s3"
+setup { SET synchronous_commit=on; }
+step "s3_begin" { BEGIN; }
+step "s3_insert_tbl1" { INSERT INTO tbl1 (val1, val2) VALUES (2, 2); }
+step "s3_commit" { COMMIT; }
+
# Expect to get one insert change. LOGICAL_REP_MSG_INSERT = 'I'
permutation "s1_insert_tbl1" "s1_begin" "s1_insert_tbl1" "s2_alter_pub_add_tbl" "s1_commit" "s1_insert_tbl1" "s2_get_binary_changes"
+
+# Expect to get one insert change with LOGICAL_REP_MSG_INSERT = 'I' from
+# the second "s1_insert_tbl1" executed after adding the table tbl1 the
+# publication in "s2_alter_pub_add_tbl".
+permutation "s1_begin" "s1_insert_tbl1" "s3_begin" "s3_insert_tbl1" "s2_alter_pub_add_tbl" "s1_insert_tbl1" "s1_commit" "s3_commit" "s2_get_binary_changes"
diff --git a/src/backend/replication/logical/reorderbuffer.c b/src/backend/replication/logical/reorderbuffer.c
index a52b51a5e7d..b708877d965 100644
--- a/src/backend/replication/logical/reorderbuffer.c
+++ b/src/backend/replication/logical/reorderbuffer.c
@@ -103,12 +103,24 @@
#include "storage/sinval.h"
#include "utils/builtins.h"
#include "utils/combocid.h"
+#include "utils/inval.h"
#include "utils/memdebug.h"
#include "utils/memutils.h"
#include "utils/rel.h"
#include "utils/relfilenodemap.h"
+/*
+ * Each transaction has an 8MB limit for invalidation messages distributed from
+ * other transactions. This limit is set considering scenarios with many
+ * concurrent logical decoding operations. When the distributed invalidation
+ * messages reach this threshold, the transaction is marked as
+ * RBTXN_DISTR_INVAL_OVERFLOWED to invalidate the complete cache as we have lost
+ * some inval messages and hence don't know what needs to be invalidated.
+ */
+#define MAX_DISTR_INVAL_MSG_PER_TXN \
+ ((8 * 1024 * 1024) / sizeof(SharedInvalidationMessage))
+
/* entry for a hash table we use to map from xid to our transaction state */
typedef struct ReorderBufferTXNByIdEnt
{
@@ -453,6 +465,12 @@ ReorderBufferReturnTXN(ReorderBuffer *rb, ReorderBufferTXN *txn)
txn->invalidations = NULL;
}
+ if (txn->invalidations_distributed)
+ {
+ pfree(txn->invalidations_distributed);
+ txn->invalidations_distributed = NULL;
+ }
+
/* Reset the toast hash */
ReorderBufferToastReset(rb, txn);
@@ -2500,7 +2518,17 @@ ReorderBufferProcessTXN(ReorderBuffer *rb, ReorderBufferTXN *txn,
AbortCurrentTransaction();
/* make sure there's no cache pollution */
- ReorderBufferExecuteInvalidations(txn->ninvalidations, txn->invalidations);
+ if (rbtxn_distr_inval_overflowed(txn))
+ {
+ Assert(txn->ninvalidations_distributed == 0);
+ InvalidateSystemCaches();
+ }
+ else
+ {
+ ReorderBufferExecuteInvalidations(txn->ninvalidations, txn->invalidations);
+ ReorderBufferExecuteInvalidations(txn->ninvalidations_distributed,
+ txn->invalidations_distributed);
+ }
if (using_subtxn)
RollbackAndReleaseCurrentSubTransaction();
@@ -2546,8 +2574,17 @@ ReorderBufferProcessTXN(ReorderBuffer *rb, ReorderBufferTXN *txn,
AbortCurrentTransaction();
/* make sure there's no cache pollution */
- ReorderBufferExecuteInvalidations(txn->ninvalidations,
- txn->invalidations);
+ if (rbtxn_distr_inval_overflowed(txn))
+ {
+ Assert(txn->ninvalidations_distributed == 0);
+ InvalidateSystemCaches();
+ }
+ else
+ {
+ ReorderBufferExecuteInvalidations(txn->ninvalidations, txn->invalidations);
+ ReorderBufferExecuteInvalidations(txn->ninvalidations_distributed,
+ txn->invalidations_distributed);
+ }
if (using_subtxn)
RollbackAndReleaseCurrentSubTransaction();
@@ -2873,7 +2910,8 @@ ReorderBufferAbort(ReorderBuffer *rb, TransactionId xid, XLogRecPtr lsn)
* We might have decoded changes for this transaction that could load
* the cache as per the current transaction's view (consider DDL's
* happened in this transaction). We don't want the decoding of future
- * transactions to use those cache entries so execute invalidations.
+ * transactions to use those cache entries so execute only the inval
+ * messages in this transaction.
*/
if (txn->ninvalidations > 0)
ReorderBufferImmediateInvalidation(rb, txn->ninvalidations,
@@ -2961,9 +2999,10 @@ ReorderBufferForget(ReorderBuffer *rb, TransactionId xid, XLogRecPtr lsn)
txn->final_lsn = lsn;
/*
- * Process cache invalidation messages if there are any. Even if we're not
- * interested in the transaction's contents, it could have manipulated the
- * catalog and we need to update the caches according to that.
+ * Process only cache invalidation messages in this transaction if there
+ * are any. Even if we're not interested in the transaction's contents, it
+ * could have manipulated the catalog and we need to update the caches
+ * according to that.
*/
if (txn->base_snapshot != NULL && txn->ninvalidations > 0)
ReorderBufferImmediateInvalidation(rb, txn->ninvalidations,
@@ -3218,6 +3257,57 @@ ReorderBufferAddNewTupleCids(ReorderBuffer *rb, TransactionId xid,
txn->ntuplecids++;
}
+/*
+ * Add new invalidation messages to the reorder buffer queue.
+ */
+static void
+ReorderBufferQueueInvalidations(ReorderBuffer *rb, TransactionId xid,
+ XLogRecPtr lsn, Size nmsgs,
+ SharedInvalidationMessage *msgs)
+{
+ ReorderBufferChange *change;
+
+ change = ReorderBufferGetChange(rb);
+ change->action = REORDER_BUFFER_CHANGE_INVALIDATION;
+ change->data.inval.ninvalidations = nmsgs;
+ change->data.inval.invalidations = (SharedInvalidationMessage *)
+ palloc(sizeof(SharedInvalidationMessage) * nmsgs);
+ memcpy(change->data.inval.invalidations, msgs,
+ sizeof(SharedInvalidationMessage) * nmsgs);
+
+ ReorderBufferQueueChange(rb, xid, lsn, change, false);
+}
+
+/*
+ * A helper function for ReorderBufferAddInvalidations() and
+ * ReorderBufferAddDistributedInvalidations() to accumulate the invalidation
+ * messages to the **invals_out.
+ */
+static void
+ReorderBufferAccumulateInvalidations(SharedInvalidationMessage **invals_out,
+ uint32 *ninvals_out,
+ SharedInvalidationMessage *msgs_new,
+ Size nmsgs_new)
+{
+ if (*ninvals_out == 0)
+ {
+ *ninvals_out = nmsgs_new;
+ *invals_out = (SharedInvalidationMessage *)
+ palloc(sizeof(SharedInvalidationMessage) * nmsgs_new);
+ memcpy(*invals_out, msgs_new, sizeof(SharedInvalidationMessage) * nmsgs_new);
+ }
+ else
+ {
+ /* Enlarge the array of inval messages */
+ *invals_out = (SharedInvalidationMessage *)
+ repalloc(*invals_out, sizeof(SharedInvalidationMessage) *
+ (*ninvals_out + nmsgs_new));
+ memcpy(*invals_out + *ninvals_out, msgs_new,
+ nmsgs_new * sizeof(SharedInvalidationMessage));
+ *ninvals_out += nmsgs_new;
+ }
+}
+
/*
* Accumulate the invalidations for executing them later.
*
@@ -3238,7 +3328,6 @@ ReorderBufferAddInvalidations(ReorderBuffer *rb, TransactionId xid,
{
ReorderBufferTXN *txn;
MemoryContext oldcontext;
- ReorderBufferChange *change;
txn = ReorderBufferTXNByXid(rb, xid, true, NULL, lsn, true);
@@ -3254,35 +3343,77 @@ ReorderBufferAddInvalidations(ReorderBuffer *rb, TransactionId xid,
Assert(nmsgs > 0);
- /* Accumulate invalidations. */
- if (txn->ninvalidations == 0)
- {
- txn->ninvalidations = nmsgs;
- txn->invalidations = (SharedInvalidationMessage *)
- palloc(sizeof(SharedInvalidationMessage) * nmsgs);
- memcpy(txn->invalidations, msgs,
- sizeof(SharedInvalidationMessage) * nmsgs);
- }
- else
+ ReorderBufferAccumulateInvalidations(&txn->invalidations,
+ &txn->ninvalidations,
+ msgs, nmsgs);
+
+ ReorderBufferQueueInvalidations(rb, xid, lsn, nmsgs, msgs);
+
+ MemoryContextSwitchTo(oldcontext);
+}
+
+/*
+ * Accumulate the invalidations distributed by other committed transactions
+ * for executing them later.
+ *
+ * This function is similar to ReorderBufferAddInvalidations() but stores
+ * the given inval messages to the txn->invalidations_distributed with the
+ * overflow check.
+ *
+ * This needs to be called by committed transactions to distribute their
+ * inval messages to in-progress transactions.
+ */
+void
+ReorderBufferAddDistributedInvalidations(ReorderBuffer *rb, TransactionId xid,
+ XLogRecPtr lsn, Size nmsgs,
+ SharedInvalidationMessage *msgs)
+{
+ ReorderBufferTXN *txn;
+ MemoryContext oldcontext;
+
+ txn = ReorderBufferTXNByXid(rb, xid, true, NULL, lsn, true);
+
+ oldcontext = MemoryContextSwitchTo(rb->context);
+
+ /*
+ * Collect all the invalidations under the top transaction, if available,
+ * so that we can execute them all together. See comments
+ * ReorderBufferAddInvalidations.
+ */
+ if (txn->toptxn)
+ txn = txn->toptxn;
+
+ Assert(nmsgs > 0);
+
+ if (!rbtxn_distr_inval_overflowed(txn))
{
- txn->invalidations = (SharedInvalidationMessage *)
- repalloc(txn->invalidations, sizeof(SharedInvalidationMessage) *
- (txn->ninvalidations + nmsgs));
+ /*
+ * Check the transaction has enough space for storing distributed
+ * invalidation messages.
+ */
+ if (txn->ninvalidations_distributed + nmsgs >= MAX_DISTR_INVAL_MSG_PER_TXN)
+ {
+ /*
+ * Mark the invalidation message as overflowed and free up the
+ * messages accumulated so far.
+ */
+ txn->txn_flags |= RBTXN_DISTR_INVAL_OVERFLOWED;
- memcpy(txn->invalidations + txn->ninvalidations, msgs,
- nmsgs * sizeof(SharedInvalidationMessage));
- txn->ninvalidations += nmsgs;
+ if (txn->invalidations_distributed)
+ {
+ pfree(txn->invalidations_distributed);
+ txn->invalidations_distributed = NULL;
+ txn->ninvalidations_distributed = 0;
+ }
+ }
+ else
+ ReorderBufferAccumulateInvalidations(&txn->invalidations_distributed,
+ &txn->ninvalidations_distributed,
+ msgs, nmsgs);
}
- change = ReorderBufferGetChange(rb);
- change->action = REORDER_BUFFER_CHANGE_INVALIDATION;
- change->data.inval.ninvalidations = nmsgs;
- change->data.inval.invalidations = (SharedInvalidationMessage *)
- palloc(sizeof(SharedInvalidationMessage) * nmsgs);
- memcpy(change->data.inval.invalidations, msgs,
- sizeof(SharedInvalidationMessage) * nmsgs);
-
- ReorderBufferQueueChange(rb, xid, lsn, change, false);
+ /* Queue the invalidation messages into the transaction */
+ ReorderBufferQueueInvalidations(rb, xid, lsn, nmsgs, msgs);
MemoryContextSwitchTo(oldcontext);
}
diff --git a/src/backend/replication/logical/snapbuild.c b/src/backend/replication/logical/snapbuild.c
index 6025f2727ae..67f0b9e089a 100644
--- a/src/backend/replication/logical/snapbuild.c
+++ b/src/backend/replication/logical/snapbuild.c
@@ -916,6 +916,13 @@ SnapBuildDistributeSnapshotAndInval(SnapBuild *builder, XLogRecPtr lsn, Transact
* contents built by the current transaction even after its decoding,
* which should have been invalidated due to concurrent catalog
* changing transaction.
+ *
+ * Distribute only the invalidation messages generated by the current
+ * committed transaction. Invalidation messages received from other
+ * transactions would have already been propagated to the relevant
+ * in-progress transactions. This transaction would have processed
+ * those invalidations, ensuring that subsequent transactions observe
+ * a consistent cache state.
*/
if (txn->xid != xid)
{
@@ -929,8 +936,9 @@ SnapBuildDistributeSnapshotAndInval(SnapBuild *builder, XLogRecPtr lsn, Transact
{
Assert(msgs != NULL);
- ReorderBufferAddInvalidations(builder->reorder, txn->xid, lsn,
- ninvalidations, msgs);
+ ReorderBufferAddDistributedInvalidations(builder->reorder,
+ txn->xid, lsn,
+ ninvalidations, msgs);
}
}
}
diff --git a/src/include/replication/reorderbuffer.h b/src/include/replication/reorderbuffer.h
index d399975e8a5..cb633315d04 100644
--- a/src/include/replication/reorderbuffer.h
+++ b/src/include/replication/reorderbuffer.h
@@ -168,14 +168,15 @@ typedef struct ReorderBufferChange
} ReorderBufferChange;
/* ReorderBufferTXN txn_flags */
-#define RBTXN_HAS_CATALOG_CHANGES 0x0001
-#define RBTXN_IS_SUBXACT 0x0002
-#define RBTXN_IS_SERIALIZED 0x0004
-#define RBTXN_IS_SERIALIZED_CLEAR 0x0008
-#define RBTXN_IS_STREAMED 0x0010
-#define RBTXN_HAS_PARTIAL_CHANGE 0x0020
-#define RBTXN_PREPARE 0x0040
-#define RBTXN_SKIPPED_PREPARE 0x0080
+#define RBTXN_HAS_CATALOG_CHANGES 0x0001
+#define RBTXN_IS_SUBXACT 0x0002
+#define RBTXN_IS_SERIALIZED 0x0004
+#define RBTXN_IS_SERIALIZED_CLEAR 0x0008
+#define RBTXN_IS_STREAMED 0x0010
+#define RBTXN_HAS_PARTIAL_CHANGE 0x0020
+#define RBTXN_PREPARE 0x0040
+#define RBTXN_SKIPPED_PREPARE 0x0080
+#define RBTXN_DISTR_INVAL_OVERFLOWED 0x0100
/* Does the transaction have catalog changes? */
#define rbtxn_has_catalog_changes(txn) \
@@ -233,6 +234,12 @@ typedef struct ReorderBufferChange
((txn)->txn_flags & RBTXN_SKIPPED_PREPARE) != 0 \
)
+/* Is the array of distributed inval messages overflowed? */
+#define rbtxn_distr_inval_overflowed(txn) \
+( \
+ ((txn)->txn_flags & RBTXN_DISTR_INVAL_OVERFLOWED) != 0 \
+)
+
typedef struct ReorderBufferTXN
{
/* See above */
@@ -391,6 +398,12 @@ typedef struct ReorderBufferTXN
* Private data pointer of the output plugin.
*/
void *output_plugin_private;
+
+ /*
+ * Stores cache invalidation messages distributed by other transactions.
+ */
+ uint32 ninvalidations_distributed;
+ SharedInvalidationMessage *invalidations_distributed;
} ReorderBufferTXN;
/* so we can define the callbacks used inside struct ReorderBuffer itself */
@@ -657,6 +670,9 @@ void ReorderBufferAddNewTupleCids(ReorderBuffer *, TransactionId, XLogRecPtr ls
CommandId cmin, CommandId cmax, CommandId combocid);
void ReorderBufferAddInvalidations(ReorderBuffer *, TransactionId, XLogRecPtr lsn,
Size nmsgs, SharedInvalidationMessage *msgs);
+void ReorderBufferAddDistributedInvalidations(ReorderBuffer *rb, TransactionId xid,
+ XLogRecPtr lsn, Size nmsgs,
+ SharedInvalidationMessage *msgs);
void ReorderBufferImmediateInvalidation(ReorderBuffer *, uint32 ninvalidations,
SharedInvalidationMessage *invalidations);
void ReorderBufferProcessXid(ReorderBuffer *, TransactionId xid, XLogRecPtr lsn);
--
2.43.0
[text/x-patch] v14-PG13-0001-Fix-exponential-memory-allocation-issue-in-.patch (16.4K, ../../CALDaNm3m6Dq2d6bCBaOjnABYavT3p+=VbJd5oJKok0sCTOSZog@mail.gmail.com/3-v14-PG13-0001-Fix-exponential-memory-allocation-issue-in-.patch)
download | inline diff:
From 35e4fb4f2804d9eaed53a41b3712e1a71b1fdf64 Mon Sep 17 00:00:00 2001
From: Vignesh C <vignesh21@gmail.com>
Date: Thu, 12 Jun 2025 11:46:35 +0530
Subject: [PATCH v14-PG13] Fix exponential memory allocation issue in logical
decoding due to invalidation message distribution
Commit 4909b38af0 added logic to distribute invalidation messages from
one transaction to all concurrent in-progress transactions. However,
since the source of each invalidation message was not tracked, transactions
could redundantly re-distribute already propagated messages. This led to an
exponential increase in invalidation messages and, ultimately, memory
allocation failures.
To fix this, a new field is introduced to track distributed invalidation
messages separately. This ensures that only transactions in the pre-existing
list receive the invalidations, preventing redundant propagation.
The accumulated distributed invalidations are now consumed at the end of
transaction processing in `ReorderBufferProcessTXN()`.
The test for version 13 differs from that of version 14 because concurrent
DDL changes are not visible in version 13. This limitation is due to the
absence of support for REORDER_BUFFER_CHANGE_INVALIDATION. In version 13, the
test inserts data using a new transaction and verifies that these changes are
selected.
---
.../expected/invalidation_distribution.out | 23 ++-
.../specs/invalidation_distribution.spec | 11 ++
.../replication/logical/reorderbuffer.c | 169 +++++++++++++++---
src/backend/replication/logical/snapbuild.c | 12 +-
src/include/replication/reorderbuffer.h | 22 ++-
5 files changed, 207 insertions(+), 30 deletions(-)
diff --git a/contrib/test_decoding/expected/invalidation_distribution.out b/contrib/test_decoding/expected/invalidation_distribution.out
index eb70eda9042..861d5ac3ef4 100644
--- a/contrib/test_decoding/expected/invalidation_distribution.out
+++ b/contrib/test_decoding/expected/invalidation_distribution.out
@@ -1,4 +1,4 @@
-Parsed test spec with 2 sessions
+Parsed test spec with 3 sessions
starting permutation: s1_insert_tbl1 s1_begin s1_insert_tbl1 s2_alter_pub_add_tbl s1_commit s1_insert_tbl1 s2_get_binary_changes
step s1_insert_tbl1: INSERT INTO tbl1 (val1, val2) VALUES (1, 1);
@@ -18,3 +18,24 @@ count
stop
(1 row)
+
+starting permutation: s1_begin s1_insert_tbl1 s3_begin s3_insert_tbl1 s2_alter_pub_add_tbl s1_commit s3_commit s1_insert_tbl1 s2_get_binary_changes
+step s1_begin: BEGIN;
+step s1_insert_tbl1: INSERT INTO tbl1 (val1, val2) VALUES (1, 1);
+step s3_begin: BEGIN;
+step s3_insert_tbl1: INSERT INTO tbl1 (val1, val2) VALUES (2, 2);
+step s2_alter_pub_add_tbl: ALTER PUBLICATION pub ADD TABLE tbl1;
+step s1_commit: COMMIT;
+step s3_commit: COMMIT;
+step s1_insert_tbl1: INSERT INTO tbl1 (val1, val2) VALUES (1, 1);
+step s2_get_binary_changes: SELECT count(data) FROM pg_logical_slot_get_binary_changes('isolation_slot', NULL, NULL, 'proto_version', '1', 'publication_names', 'pub') WHERE get_byte(data, 0) = 73;
+count
+-----
+ 1
+(1 row)
+
+?column?
+--------
+stop
+(1 row)
+
diff --git a/contrib/test_decoding/specs/invalidation_distribution.spec b/contrib/test_decoding/specs/invalidation_distribution.spec
index ca051fc1e85..dbc1816e421 100644
--- a/contrib/test_decoding/specs/invalidation_distribution.spec
+++ b/contrib/test_decoding/specs/invalidation_distribution.spec
@@ -28,5 +28,16 @@ setup { SET synchronous_commit=on; }
step "s2_alter_pub_add_tbl" { ALTER PUBLICATION pub ADD TABLE tbl1; }
step "s2_get_binary_changes" { SELECT count(data) FROM pg_logical_slot_get_binary_changes('isolation_slot', NULL, NULL, 'proto_version', '1', 'publication_names', 'pub') WHERE get_byte(data, 0) = 73; }
+session "s3"
+setup { SET synchronous_commit=on; }
+step "s3_begin" { BEGIN; }
+step "s3_insert_tbl1" { INSERT INTO tbl1 (val1, val2) VALUES (2, 2); }
+step "s3_commit" { COMMIT; }
+
# Expect to get one insert change. LOGICAL_REP_MSG_INSERT = 'I'
permutation "s1_insert_tbl1" "s1_begin" "s1_insert_tbl1" "s2_alter_pub_add_tbl" "s1_commit" "s1_insert_tbl1" "s2_get_binary_changes"
+
+# Expect to get one insert change with LOGICAL_REP_MSG_INSERT = 'I' from
+# the second "s1_insert_tbl1" executed after adding the table tbl1 the
+# publication in "s2_alter_pub_add_tbl".
+permutation "s1_begin" "s1_insert_tbl1" "s3_begin" "s3_insert_tbl1" "s2_alter_pub_add_tbl" "s1_commit" "s3_commit" "s1_insert_tbl1" "s2_get_binary_changes"
diff --git a/src/backend/replication/logical/reorderbuffer.c b/src/backend/replication/logical/reorderbuffer.c
index fa9413fa2a0..e5fa1d62151 100644
--- a/src/backend/replication/logical/reorderbuffer.c
+++ b/src/backend/replication/logical/reorderbuffer.c
@@ -103,12 +103,24 @@
#include "storage/sinval.h"
#include "utils/builtins.h"
#include "utils/combocid.h"
+#include "utils/inval.h"
#include "utils/memdebug.h"
#include "utils/memutils.h"
#include "utils/rel.h"
#include "utils/relfilenodemap.h"
+/*
+ * Each transaction has an 8MB limit for invalidation messages distributed from
+ * other transactions. This limit is set considering scenarios with many
+ * concurrent logical decoding operations. When the distributed invalidation
+ * messages reach this threshold, the transaction is marked as
+ * RBTXN_DISTR_INVAL_OVERFLOWED to invalidate the complete cache as we have lost
+ * some inval messages and hence don't know what needs to be invalidated.
+ */
+#define MAX_DISTR_INVAL_MSG_PER_TXN \
+ ((8 * 1024 * 1024) / sizeof(SharedInvalidationMessage))
+
/* entry for a hash table we use to map from xid to our transaction state */
typedef struct ReorderBufferTXNByIdEnt
{
@@ -220,7 +232,8 @@ static void ReorderBufferIterTXNInit(ReorderBuffer *rb, ReorderBufferTXN *txn,
static ReorderBufferChange *ReorderBufferIterTXNNext(ReorderBuffer *rb, ReorderBufferIterTXNState *state);
static void ReorderBufferIterTXNFinish(ReorderBuffer *rb,
ReorderBufferIterTXNState *state);
-static void ReorderBufferExecuteInvalidations(ReorderBuffer *rb, ReorderBufferTXN *txn);
+static void ReorderBufferExecuteInvalidations(uint32 nmsgs,
+ SharedInvalidationMessage *msgs);
/*
* ---------------------------------------
@@ -406,6 +419,12 @@ ReorderBufferReturnTXN(ReorderBuffer *rb, ReorderBufferTXN *txn)
txn->invalidations = NULL;
}
+ if (txn->invalidations_distributed)
+ {
+ pfree(txn->invalidations_distributed);
+ txn->invalidations_distributed = NULL;
+ }
+
/* Reset the toast hash */
ReorderBufferToastReset(rb, txn);
@@ -1883,7 +1902,8 @@ ReorderBufferCommit(ReorderBuffer *rb, TransactionId xid,
* see new catalog contents, so execute all
* invalidations.
*/
- ReorderBufferExecuteInvalidations(rb, txn);
+ ReorderBufferExecuteInvalidations(txn->ninvalidations,
+ txn->invalidations);
}
break;
@@ -1921,7 +1941,17 @@ ReorderBufferCommit(ReorderBuffer *rb, TransactionId xid,
AbortCurrentTransaction();
/* make sure there's no cache pollution */
- ReorderBufferExecuteInvalidations(rb, txn);
+ if (rbtxn_distr_inval_overflowed(txn))
+ {
+ Assert(txn->ninvalidations_distributed == 0);
+ InvalidateSystemCaches();
+ }
+ else
+ {
+ ReorderBufferExecuteInvalidations(txn->ninvalidations, txn->invalidations);
+ ReorderBufferExecuteInvalidations(txn->ninvalidations_distributed,
+ txn->invalidations_distributed);
+ }
if (using_subtxn)
RollbackAndReleaseCurrentSubTransaction();
@@ -1947,7 +1977,17 @@ ReorderBufferCommit(ReorderBuffer *rb, TransactionId xid,
AbortCurrentTransaction();
/* make sure there's no cache pollution */
- ReorderBufferExecuteInvalidations(rb, txn);
+ if (rbtxn_distr_inval_overflowed(txn))
+ {
+ Assert(txn->ninvalidations_distributed == 0);
+ InvalidateSystemCaches();
+ }
+ else
+ {
+ ReorderBufferExecuteInvalidations(txn->ninvalidations, txn->invalidations);
+ ReorderBufferExecuteInvalidations(txn->ninvalidations_distributed,
+ txn->invalidations_distributed);
+ }
if (using_subtxn)
RollbackAndReleaseCurrentSubTransaction();
@@ -2060,9 +2100,10 @@ ReorderBufferForget(ReorderBuffer *rb, TransactionId xid, XLogRecPtr lsn)
txn->final_lsn = lsn;
/*
- * Process cache invalidation messages if there are any. Even if we're not
- * interested in the transaction's contents, it could have manipulated the
- * catalog and we need to update the caches according to that.
+ * Process only cache invalidation messages in this transaction if there
+ * are any. Even if we're not interested in the transaction's contents, it
+ * could have manipulated the catalog and we need to update the caches
+ * according to that.
*/
if (txn->base_snapshot != NULL && txn->ninvalidations > 0)
ReorderBufferImmediateInvalidation(rb, txn->ninvalidations,
@@ -2253,6 +2294,36 @@ ReorderBufferAddNewTupleCids(ReorderBuffer *rb, TransactionId xid,
txn->ntuplecids++;
}
+/*
+ * A helper function for ReorderBufferAddInvalidations() and
+ * ReorderBufferAddDistributedInvalidations() to accumulate the invalidation
+ * messages to the **invals_out.
+ */
+static void
+ReorderBufferAccumulateInvalidations(SharedInvalidationMessage **invals_out,
+ uint32 *ninvals_out,
+ SharedInvalidationMessage *msgs_new,
+ Size nmsgs_new)
+{
+ if (*ninvals_out == 0)
+ {
+ *ninvals_out = nmsgs_new;
+ *invals_out = (SharedInvalidationMessage *)
+ palloc(sizeof(SharedInvalidationMessage) * nmsgs_new);
+ memcpy(*invals_out, msgs_new, sizeof(SharedInvalidationMessage) * nmsgs_new);
+ }
+ else
+ {
+ /* Enlarge the array of inval messages */
+ *invals_out = (SharedInvalidationMessage *)
+ repalloc(*invals_out, sizeof(SharedInvalidationMessage) *
+ (*ninvals_out + nmsgs_new));
+ memcpy(*invals_out + *ninvals_out, msgs_new,
+ nmsgs_new * sizeof(SharedInvalidationMessage));
+ *ninvals_out += nmsgs_new;
+ }
+}
+
/*
* Setup the invalidation of the toplevel transaction.
*
@@ -2282,24 +2353,74 @@ ReorderBufferAddInvalidations(ReorderBuffer *rb, TransactionId xid,
Assert(nmsgs > 0);
- /* Accumulate invalidations. */
- if (txn->ninvalidations == 0)
+ ReorderBufferAccumulateInvalidations(&txn->invalidations,
+ &txn->ninvalidations,
+ msgs, nmsgs);
+
+ MemoryContextSwitchTo(oldcontext);
+}
+
+
+/*
+ * Accumulate the invalidations distributed by other committed transactions
+ * for executing them later.
+ *
+ * This function is similar to ReorderBufferAddInvalidations() but stores
+ * the given inval messages to the txn->invalidations_distributed with the
+ * overflow check.
+ *
+ * This needs to be called by committed transactions to distribute their
+ * inval messages to in-progress transactions.
+ */
+void
+ReorderBufferAddDistributedInvalidations(ReorderBuffer *rb, TransactionId xid,
+ XLogRecPtr lsn, Size nmsgs,
+ SharedInvalidationMessage *msgs)
+{
+ ReorderBufferTXN *txn;
+ MemoryContext oldcontext;
+
+ txn = ReorderBufferTXNByXid(rb, xid, true, NULL, lsn, true);
+
+ oldcontext = MemoryContextSwitchTo(rb->context);
+
+ /*
+ * Collect all the invalidations under the top transaction, if available,
+ * so that we can execute them all together.
+ */
+ if (txn->toplevel_xid)
{
- txn->ninvalidations = nmsgs;
- txn->invalidations = (SharedInvalidationMessage *)
- palloc(sizeof(SharedInvalidationMessage) * nmsgs);
- memcpy(txn->invalidations, msgs,
- sizeof(SharedInvalidationMessage) * nmsgs);
+ txn = ReorderBufferTXNByXid(rb, txn->toplevel_xid, true, NULL, lsn,
+ true);
}
- else
+
+ Assert(nmsgs > 0);
+
+ if (!rbtxn_distr_inval_overflowed(txn))
{
- txn->invalidations = (SharedInvalidationMessage *)
- repalloc(txn->invalidations, sizeof(SharedInvalidationMessage) *
- (txn->ninvalidations + nmsgs));
+ /*
+ * Check the transaction has enough space for storing distributed
+ * invalidation messages.
+ */
+ if (txn->ninvalidations_distributed + nmsgs >= MAX_DISTR_INVAL_MSG_PER_TXN)
+ {
+ /*
+ * Mark the invalidation message as overflowed and free up the
+ * messages accumulated so far.
+ */
+ txn->txn_flags |= RBTXN_DISTR_INVAL_OVERFLOWED;
- memcpy(txn->invalidations + txn->ninvalidations, msgs,
- nmsgs * sizeof(SharedInvalidationMessage));
- txn->ninvalidations += nmsgs;
+ if (txn->invalidations_distributed)
+ {
+ pfree(txn->invalidations_distributed);
+ txn->invalidations_distributed = NULL;
+ txn->ninvalidations_distributed = 0;
+ }
+ }
+ else
+ ReorderBufferAccumulateInvalidations(&txn->invalidations_distributed,
+ &txn->ninvalidations_distributed,
+ msgs, nmsgs);
}
MemoryContextSwitchTo(oldcontext);
@@ -2310,12 +2431,12 @@ ReorderBufferAddInvalidations(ReorderBuffer *rb, TransactionId xid,
* in the changestream but we don't know which those are.
*/
static void
-ReorderBufferExecuteInvalidations(ReorderBuffer *rb, ReorderBufferTXN *txn)
+ReorderBufferExecuteInvalidations(uint32 nmsgs, SharedInvalidationMessage *msgs)
{
int i;
- for (i = 0; i < txn->ninvalidations; i++)
- LocalExecuteInvalidationMessage(&txn->invalidations[i]);
+ for (i = 0; i < nmsgs; i++)
+ LocalExecuteInvalidationMessage(&msgs[i]);
}
/*
diff --git a/src/backend/replication/logical/snapbuild.c b/src/backend/replication/logical/snapbuild.c
index 3bda41c5251..69b083a36fe 100644
--- a/src/backend/replication/logical/snapbuild.c
+++ b/src/backend/replication/logical/snapbuild.c
@@ -927,6 +927,13 @@ SnapBuildDistributeSnapshotAndInval(SnapBuild *builder, XLogRecPtr lsn, Transact
* contents built by the current transaction even after its decoding,
* which should have been invalidated due to concurrent catalog
* changing transaction.
+ *
+ * Distribute only the invalidation messages generated by the current
+ * committed transaction. Invalidation messages received from other
+ * transactions would have already been propagated to the relevant
+ * in-progress transactions. This transaction would have processed
+ * those invalidations, ensuring that subsequent transactions observe
+ * a consistent cache state.
*/
if (txn->xid != xid)
{
@@ -940,8 +947,9 @@ SnapBuildDistributeSnapshotAndInval(SnapBuild *builder, XLogRecPtr lsn, Transact
{
Assert(msgs != NULL);
- ReorderBufferAddInvalidations(builder->reorder, txn->xid, lsn,
- ninvalidations, msgs);
+ ReorderBufferAddDistributedInvalidations(builder->reorder,
+ txn->xid, lsn,
+ ninvalidations, msgs);
}
}
}
diff --git a/src/include/replication/reorderbuffer.h b/src/include/replication/reorderbuffer.h
index 545cee891ed..2321bb0b406 100644
--- a/src/include/replication/reorderbuffer.h
+++ b/src/include/replication/reorderbuffer.h
@@ -160,9 +160,10 @@ typedef struct ReorderBufferChange
} ReorderBufferChange;
/* ReorderBufferTXN txn_flags */
-#define RBTXN_HAS_CATALOG_CHANGES 0x0001
-#define RBTXN_IS_SUBXACT 0x0002
-#define RBTXN_IS_SERIALIZED 0x0004
+#define RBTXN_HAS_CATALOG_CHANGES 0x0001
+#define RBTXN_IS_SUBXACT 0x0002
+#define RBTXN_IS_SERIALIZED 0x0004
+#define RBTXN_DISTR_INVAL_OVERFLOWED 0x0008
/* Does the transaction have catalog changes? */
#define rbtxn_has_catalog_changes(txn) \
@@ -182,6 +183,12 @@ typedef struct ReorderBufferChange
((txn)->txn_flags & RBTXN_IS_SERIALIZED) != 0 \
)
+/* Is the array of distributed inval messages overflowed? */
+#define rbtxn_distr_inval_overflowed(txn) \
+( \
+ ((txn)->txn_flags & RBTXN_DISTR_INVAL_OVERFLOWED) != 0 \
+)
+
typedef struct ReorderBufferTXN
{
/* See above */
@@ -311,6 +318,12 @@ typedef struct ReorderBufferTXN
* Size of this transaction (changes currently in memory, in bytes).
*/
Size size;
+
+ /*
+ * Stores cache invalidation messages distributed by other transactions.
+ */
+ uint32 ninvalidations_distributed;
+ SharedInvalidationMessage *invalidations_distributed;
} ReorderBufferTXN;
/* so we can define the callbacks used inside struct ReorderBuffer itself */
@@ -451,6 +464,9 @@ void ReorderBufferAddNewTupleCids(ReorderBuffer *, TransactionId, XLogRecPtr ls
CommandId cmin, CommandId cmax, CommandId combocid);
void ReorderBufferAddInvalidations(ReorderBuffer *, TransactionId, XLogRecPtr lsn,
Size nmsgs, SharedInvalidationMessage *msgs);
+void ReorderBufferAddDistributedInvalidations(ReorderBuffer *rb, TransactionId xid,
+ XLogRecPtr lsn, Size nmsgs,
+ SharedInvalidationMessage *msgs);
void ReorderBufferImmediateInvalidation(ReorderBuffer *, uint32 ninvalidations,
SharedInvalidationMessage *invalidations);
void ReorderBufferProcessXid(ReorderBuffer *, TransactionId xid, XLogRecPtr lsn);
--
2.43.0
[text/x-patch] v14-PG15-0001-Fix-exponential-memory-allocation-issue-in-.patch (17.3K, ../../CALDaNm3m6Dq2d6bCBaOjnABYavT3p+=VbJd5oJKok0sCTOSZog@mail.gmail.com/4-v14-PG15-0001-Fix-exponential-memory-allocation-issue-in-.patch)
download | inline diff:
From b36559c655dc5ae1422670d431641afff9035776 Mon Sep 17 00:00:00 2001
From: Vignesh C <vignesh21@gmail.com>
Date: Thu, 12 Jun 2025 10:23:05 +0530
Subject: [PATCH v14-PG15] Fix exponential memory allocation issue in logical
decoding due to invalidation message distribution
Commit 4909b38af0 added logic to distribute invalidation messages from
one transaction to all concurrent in-progress transactions. However,
since the source of each invalidation message was not tracked, transactions
could redundantly re-distribute already propagated messages. This led to an
exponential increase in invalidation messages and, ultimately, memory
allocation failures.
To fix this, a new field is introduced to track distributed invalidation
messages separately. This ensures that only transactions in the pre-existing
list receive the invalidations, preventing redundant propagation.
The accumulated distributed invalidations are now consumed at the end of
transaction processing in `ReorderBufferProcessTXN()`.
---
.../expected/invalidation_distribution.out | 23 +-
.../specs/invalidation_distribution.spec | 11 +
.../replication/logical/reorderbuffer.c | 197 +++++++++++++++---
src/backend/replication/logical/snapbuild.c | 12 +-
src/include/replication/reorderbuffer.h | 32 ++-
5 files changed, 231 insertions(+), 44 deletions(-)
diff --git a/contrib/test_decoding/expected/invalidation_distribution.out b/contrib/test_decoding/expected/invalidation_distribution.out
index 24190ebe570..49c1f60cc4f 100644
--- a/contrib/test_decoding/expected/invalidation_distribution.out
+++ b/contrib/test_decoding/expected/invalidation_distribution.out
@@ -1,4 +1,4 @@
-Parsed test spec with 2 sessions
+Parsed test spec with 3 sessions
starting permutation: s1_insert_tbl1 s1_begin s1_insert_tbl1 s2_alter_pub_add_tbl s1_commit s1_insert_tbl1 s2_get_binary_changes
step s1_insert_tbl1: INSERT INTO tbl1 (val1, val2) VALUES (1, 1);
@@ -18,3 +18,24 @@ count
stop
(1 row)
+
+starting permutation: s1_begin s1_insert_tbl1 s3_begin s3_insert_tbl1 s2_alter_pub_add_tbl s1_insert_tbl1 s1_commit s3_commit s2_get_binary_changes
+step s1_begin: BEGIN;
+step s1_insert_tbl1: INSERT INTO tbl1 (val1, val2) VALUES (1, 1);
+step s3_begin: BEGIN;
+step s3_insert_tbl1: INSERT INTO tbl1 (val1, val2) VALUES (2, 2);
+step s2_alter_pub_add_tbl: ALTER PUBLICATION pub ADD TABLE tbl1;
+step s1_insert_tbl1: INSERT INTO tbl1 (val1, val2) VALUES (1, 1);
+step s1_commit: COMMIT;
+step s3_commit: COMMIT;
+step s2_get_binary_changes: SELECT count(data) FROM pg_logical_slot_get_binary_changes('isolation_slot', NULL, NULL, 'proto_version', '3', 'publication_names', 'pub') WHERE get_byte(data, 0) = 73;
+count
+-----
+ 1
+(1 row)
+
+?column?
+--------
+stop
+(1 row)
+
diff --git a/contrib/test_decoding/specs/invalidation_distribution.spec b/contrib/test_decoding/specs/invalidation_distribution.spec
index f63aba3ce96..6dd2c84adbf 100644
--- a/contrib/test_decoding/specs/invalidation_distribution.spec
+++ b/contrib/test_decoding/specs/invalidation_distribution.spec
@@ -28,5 +28,16 @@ setup { SET synchronous_commit=on; }
step "s2_alter_pub_add_tbl" { ALTER PUBLICATION pub ADD TABLE tbl1; }
step "s2_get_binary_changes" { SELECT count(data) FROM pg_logical_slot_get_binary_changes('isolation_slot', NULL, NULL, 'proto_version', '3', 'publication_names', 'pub') WHERE get_byte(data, 0) = 73; }
+session "s3"
+setup { SET synchronous_commit=on; }
+step "s3_begin" { BEGIN; }
+step "s3_insert_tbl1" { INSERT INTO tbl1 (val1, val2) VALUES (2, 2); }
+step "s3_commit" { COMMIT; }
+
# Expect to get one insert change. LOGICAL_REP_MSG_INSERT = 'I'
permutation "s1_insert_tbl1" "s1_begin" "s1_insert_tbl1" "s2_alter_pub_add_tbl" "s1_commit" "s1_insert_tbl1" "s2_get_binary_changes"
+
+# Expect to get one insert change with LOGICAL_REP_MSG_INSERT = 'I' from
+# the second "s1_insert_tbl1" executed after adding the table tbl1 the
+# publication in "s2_alter_pub_add_tbl".
+permutation "s1_begin" "s1_insert_tbl1" "s3_begin" "s3_insert_tbl1" "s2_alter_pub_add_tbl" "s1_insert_tbl1" "s1_commit" "s3_commit" "s2_get_binary_changes"
diff --git a/src/backend/replication/logical/reorderbuffer.c b/src/backend/replication/logical/reorderbuffer.c
index 835180e12f6..59efa73930f 100644
--- a/src/backend/replication/logical/reorderbuffer.c
+++ b/src/backend/replication/logical/reorderbuffer.c
@@ -103,12 +103,24 @@
#include "storage/sinval.h"
#include "utils/builtins.h"
#include "utils/combocid.h"
+#include "utils/inval.h"
#include "utils/memdebug.h"
#include "utils/memutils.h"
#include "utils/rel.h"
#include "utils/relfilenodemap.h"
+/*
+ * Each transaction has an 8MB limit for invalidation messages distributed from
+ * other transactions. This limit is set considering scenarios with many
+ * concurrent logical decoding operations. When the distributed invalidation
+ * messages reach this threshold, the transaction is marked as
+ * RBTXN_DISTR_INVAL_OVERFLOWED to invalidate the complete cache as we have lost
+ * some inval messages and hence don't know what needs to be invalidated.
+ */
+#define MAX_DISTR_INVAL_MSG_PER_TXN \
+ ((8 * 1024 * 1024) / sizeof(SharedInvalidationMessage))
+
/* entry for a hash table we use to map from xid to our transaction state */
typedef struct ReorderBufferTXNByIdEnt
{
@@ -457,6 +469,12 @@ ReorderBufferReturnTXN(ReorderBuffer *rb, ReorderBufferTXN *txn)
txn->invalidations = NULL;
}
+ if (txn->invalidations_distributed)
+ {
+ pfree(txn->invalidations_distributed);
+ txn->invalidations_distributed = NULL;
+ }
+
/* Reset the toast hash */
ReorderBufferToastReset(rb, txn);
@@ -2503,7 +2521,17 @@ ReorderBufferProcessTXN(ReorderBuffer *rb, ReorderBufferTXN *txn,
AbortCurrentTransaction();
/* make sure there's no cache pollution */
- ReorderBufferExecuteInvalidations(txn->ninvalidations, txn->invalidations);
+ if (rbtxn_distr_inval_overflowed(txn))
+ {
+ Assert(txn->ninvalidations_distributed == 0);
+ InvalidateSystemCaches();
+ }
+ else
+ {
+ ReorderBufferExecuteInvalidations(txn->ninvalidations, txn->invalidations);
+ ReorderBufferExecuteInvalidations(txn->ninvalidations_distributed,
+ txn->invalidations_distributed);
+ }
if (using_subtxn)
RollbackAndReleaseCurrentSubTransaction();
@@ -2549,8 +2577,17 @@ ReorderBufferProcessTXN(ReorderBuffer *rb, ReorderBufferTXN *txn,
AbortCurrentTransaction();
/* make sure there's no cache pollution */
- ReorderBufferExecuteInvalidations(txn->ninvalidations,
- txn->invalidations);
+ if (rbtxn_distr_inval_overflowed(txn))
+ {
+ Assert(txn->ninvalidations_distributed == 0);
+ InvalidateSystemCaches();
+ }
+ else
+ {
+ ReorderBufferExecuteInvalidations(txn->ninvalidations, txn->invalidations);
+ ReorderBufferExecuteInvalidations(txn->ninvalidations_distributed,
+ txn->invalidations_distributed);
+ }
if (using_subtxn)
RollbackAndReleaseCurrentSubTransaction();
@@ -2877,7 +2914,8 @@ ReorderBufferAbort(ReorderBuffer *rb, TransactionId xid, XLogRecPtr lsn)
* We might have decoded changes for this transaction that could load
* the cache as per the current transaction's view (consider DDL's
* happened in this transaction). We don't want the decoding of future
- * transactions to use those cache entries so execute invalidations.
+ * transactions to use those cache entries so execute only the inval
+ * messages in this transaction.
*/
if (txn->ninvalidations > 0)
ReorderBufferImmediateInvalidation(rb, txn->ninvalidations,
@@ -2965,9 +3003,10 @@ ReorderBufferForget(ReorderBuffer *rb, TransactionId xid, XLogRecPtr lsn)
txn->final_lsn = lsn;
/*
- * Process cache invalidation messages if there are any. Even if we're not
- * interested in the transaction's contents, it could have manipulated the
- * catalog and we need to update the caches according to that.
+ * Process only cache invalidation messages in this transaction if there
+ * are any. Even if we're not interested in the transaction's contents, it
+ * could have manipulated the catalog and we need to update the caches
+ * according to that.
*/
if (txn->base_snapshot != NULL && txn->ninvalidations > 0)
ReorderBufferImmediateInvalidation(rb, txn->ninvalidations,
@@ -3222,6 +3261,57 @@ ReorderBufferAddNewTupleCids(ReorderBuffer *rb, TransactionId xid,
txn->ntuplecids++;
}
+/*
+ * Add new invalidation messages to the reorder buffer queue.
+ */
+static void
+ReorderBufferQueueInvalidations(ReorderBuffer *rb, TransactionId xid,
+ XLogRecPtr lsn, Size nmsgs,
+ SharedInvalidationMessage *msgs)
+{
+ ReorderBufferChange *change;
+
+ change = ReorderBufferGetChange(rb);
+ change->action = REORDER_BUFFER_CHANGE_INVALIDATION;
+ change->data.inval.ninvalidations = nmsgs;
+ change->data.inval.invalidations = (SharedInvalidationMessage *)
+ palloc(sizeof(SharedInvalidationMessage) * nmsgs);
+ memcpy(change->data.inval.invalidations, msgs,
+ sizeof(SharedInvalidationMessage) * nmsgs);
+
+ ReorderBufferQueueChange(rb, xid, lsn, change, false);
+}
+
+/*
+ * A helper function for ReorderBufferAddInvalidations() and
+ * ReorderBufferAddDistributedInvalidations() to accumulate the invalidation
+ * messages to the **invals_out.
+ */
+static void
+ReorderBufferAccumulateInvalidations(SharedInvalidationMessage **invals_out,
+ uint32 *ninvals_out,
+ SharedInvalidationMessage *msgs_new,
+ Size nmsgs_new)
+{
+ if (*ninvals_out == 0)
+ {
+ *ninvals_out = nmsgs_new;
+ *invals_out = (SharedInvalidationMessage *)
+ palloc(sizeof(SharedInvalidationMessage) * nmsgs_new);
+ memcpy(*invals_out, msgs_new, sizeof(SharedInvalidationMessage) * nmsgs_new);
+ }
+ else
+ {
+ /* Enlarge the array of inval messages */
+ *invals_out = (SharedInvalidationMessage *)
+ repalloc(*invals_out, sizeof(SharedInvalidationMessage) *
+ (*ninvals_out + nmsgs_new));
+ memcpy(*invals_out + *ninvals_out, msgs_new,
+ nmsgs_new * sizeof(SharedInvalidationMessage));
+ *ninvals_out += nmsgs_new;
+ }
+}
+
/*
* Accumulate the invalidations for executing them later.
*
@@ -3242,7 +3332,6 @@ ReorderBufferAddInvalidations(ReorderBuffer *rb, TransactionId xid,
{
ReorderBufferTXN *txn;
MemoryContext oldcontext;
- ReorderBufferChange *change;
txn = ReorderBufferTXNByXid(rb, xid, true, NULL, lsn, true);
@@ -3258,35 +3347,77 @@ ReorderBufferAddInvalidations(ReorderBuffer *rb, TransactionId xid,
Assert(nmsgs > 0);
- /* Accumulate invalidations. */
- if (txn->ninvalidations == 0)
- {
- txn->ninvalidations = nmsgs;
- txn->invalidations = (SharedInvalidationMessage *)
- palloc(sizeof(SharedInvalidationMessage) * nmsgs);
- memcpy(txn->invalidations, msgs,
- sizeof(SharedInvalidationMessage) * nmsgs);
- }
- else
+ ReorderBufferAccumulateInvalidations(&txn->invalidations,
+ &txn->ninvalidations,
+ msgs, nmsgs);
+
+ ReorderBufferQueueInvalidations(rb, xid, lsn, nmsgs, msgs);
+
+ MemoryContextSwitchTo(oldcontext);
+}
+
+/*
+ * Accumulate the invalidations distributed by other committed transactions
+ * for executing them later.
+ *
+ * This function is similar to ReorderBufferAddInvalidations() but stores
+ * the given inval messages to the txn->invalidations_distributed with the
+ * overflow check.
+ *
+ * This needs to be called by committed transactions to distribute their
+ * inval messages to in-progress transactions.
+ */
+void
+ReorderBufferAddDistributedInvalidations(ReorderBuffer *rb, TransactionId xid,
+ XLogRecPtr lsn, Size nmsgs,
+ SharedInvalidationMessage *msgs)
+{
+ ReorderBufferTXN *txn;
+ MemoryContext oldcontext;
+
+ txn = ReorderBufferTXNByXid(rb, xid, true, NULL, lsn, true);
+
+ oldcontext = MemoryContextSwitchTo(rb->context);
+
+ /*
+ * Collect all the invalidations under the top transaction, if available,
+ * so that we can execute them all together. See comments
+ * ReorderBufferAddInvalidations.
+ */
+ if (txn->toptxn)
+ txn = txn->toptxn;
+
+ Assert(nmsgs > 0);
+
+ if (!rbtxn_distr_inval_overflowed(txn))
{
- txn->invalidations = (SharedInvalidationMessage *)
- repalloc(txn->invalidations, sizeof(SharedInvalidationMessage) *
- (txn->ninvalidations + nmsgs));
+ /*
+ * Check the transaction has enough space for storing distributed
+ * invalidation messages.
+ */
+ if (txn->ninvalidations_distributed + nmsgs >= MAX_DISTR_INVAL_MSG_PER_TXN)
+ {
+ /*
+ * Mark the invalidation message as overflowed and free up the
+ * messages accumulated so far.
+ */
+ txn->txn_flags |= RBTXN_DISTR_INVAL_OVERFLOWED;
- memcpy(txn->invalidations + txn->ninvalidations, msgs,
- nmsgs * sizeof(SharedInvalidationMessage));
- txn->ninvalidations += nmsgs;
+ if (txn->invalidations_distributed)
+ {
+ pfree(txn->invalidations_distributed);
+ txn->invalidations_distributed = NULL;
+ txn->ninvalidations_distributed = 0;
+ }
+ }
+ else
+ ReorderBufferAccumulateInvalidations(&txn->invalidations_distributed,
+ &txn->ninvalidations_distributed,
+ msgs, nmsgs);
}
- change = ReorderBufferGetChange(rb);
- change->action = REORDER_BUFFER_CHANGE_INVALIDATION;
- change->data.inval.ninvalidations = nmsgs;
- change->data.inval.invalidations = (SharedInvalidationMessage *)
- palloc(sizeof(SharedInvalidationMessage) * nmsgs);
- memcpy(change->data.inval.invalidations, msgs,
- sizeof(SharedInvalidationMessage) * nmsgs);
-
- ReorderBufferQueueChange(rb, xid, lsn, change, false);
+ /* Queue the invalidation messages into the transaction */
+ ReorderBufferQueueInvalidations(rb, xid, lsn, nmsgs, msgs);
MemoryContextSwitchTo(oldcontext);
}
diff --git a/src/backend/replication/logical/snapbuild.c b/src/backend/replication/logical/snapbuild.c
index 2ca8e6e9aff..2dd4782f086 100644
--- a/src/backend/replication/logical/snapbuild.c
+++ b/src/backend/replication/logical/snapbuild.c
@@ -925,6 +925,13 @@ SnapBuildDistributeSnapshotAndInval(SnapBuild *builder, XLogRecPtr lsn, Transact
* contents built by the current transaction even after its decoding,
* which should have been invalidated due to concurrent catalog
* changing transaction.
+ *
+ * Distribute only the invalidation messages generated by the current
+ * committed transaction. Invalidation messages received from other
+ * transactions would have already been propagated to the relevant
+ * in-progress transactions. This transaction would have processed
+ * those invalidations, ensuring that subsequent transactions observe
+ * a consistent cache state.
*/
if (txn->xid != xid)
{
@@ -938,8 +945,9 @@ SnapBuildDistributeSnapshotAndInval(SnapBuild *builder, XLogRecPtr lsn, Transact
{
Assert(msgs != NULL);
- ReorderBufferAddInvalidations(builder->reorder, txn->xid, lsn,
- ninvalidations, msgs);
+ ReorderBufferAddDistributedInvalidations(builder->reorder,
+ txn->xid, lsn,
+ ninvalidations, msgs);
}
}
}
diff --git a/src/include/replication/reorderbuffer.h b/src/include/replication/reorderbuffer.h
index 402bb7a2728..af154587618 100644
--- a/src/include/replication/reorderbuffer.h
+++ b/src/include/replication/reorderbuffer.h
@@ -168,14 +168,15 @@ typedef struct ReorderBufferChange
} ReorderBufferChange;
/* ReorderBufferTXN txn_flags */
-#define RBTXN_HAS_CATALOG_CHANGES 0x0001
-#define RBTXN_IS_SUBXACT 0x0002
-#define RBTXN_IS_SERIALIZED 0x0004
-#define RBTXN_IS_SERIALIZED_CLEAR 0x0008
-#define RBTXN_IS_STREAMED 0x0010
-#define RBTXN_HAS_PARTIAL_CHANGE 0x0020
-#define RBTXN_PREPARE 0x0040
-#define RBTXN_SKIPPED_PREPARE 0x0080
+#define RBTXN_HAS_CATALOG_CHANGES 0x0001
+#define RBTXN_IS_SUBXACT 0x0002
+#define RBTXN_IS_SERIALIZED 0x0004
+#define RBTXN_IS_SERIALIZED_CLEAR 0x0008
+#define RBTXN_IS_STREAMED 0x0010
+#define RBTXN_HAS_PARTIAL_CHANGE 0x0020
+#define RBTXN_PREPARE 0x0040
+#define RBTXN_SKIPPED_PREPARE 0x0080
+#define RBTXN_DISTR_INVAL_OVERFLOWED 0x0100
/* Does the transaction have catalog changes? */
#define rbtxn_has_catalog_changes(txn) \
@@ -233,6 +234,12 @@ typedef struct ReorderBufferChange
((txn)->txn_flags & RBTXN_SKIPPED_PREPARE) != 0 \
)
+/* Is the array of distributed inval messages overflowed? */
+#define rbtxn_distr_inval_overflowed(txn) \
+( \
+ ((txn)->txn_flags & RBTXN_DISTR_INVAL_OVERFLOWED) != 0 \
+)
+
typedef struct ReorderBufferTXN
{
/* See above */
@@ -395,6 +402,12 @@ typedef struct ReorderBufferTXN
* Private data pointer of the output plugin.
*/
void *output_plugin_private;
+
+ /*
+ * Stores cache invalidation messages distributed by other transactions.
+ */
+ uint32 ninvalidations_distributed;
+ SharedInvalidationMessage *invalidations_distributed;
} ReorderBufferTXN;
/* so we can define the callbacks used inside struct ReorderBuffer itself */
@@ -661,6 +674,9 @@ extern void ReorderBufferAddNewTupleCids(ReorderBuffer *, TransactionId, XLogRec
CommandId cmin, CommandId cmax, CommandId combocid);
extern void ReorderBufferAddInvalidations(ReorderBuffer *, TransactionId, XLogRecPtr lsn,
Size nmsgs, SharedInvalidationMessage *msgs);
+extern void ReorderBufferAddDistributedInvalidations(ReorderBuffer *rb, TransactionId xid,
+ XLogRecPtr lsn, Size nmsgs,
+ SharedInvalidationMessage *msgs);
extern void ReorderBufferImmediateInvalidation(ReorderBuffer *, uint32 ninvalidations,
SharedInvalidationMessage *invalidations);
extern void ReorderBufferProcessXid(ReorderBuffer *, TransactionId xid, XLogRecPtr lsn);
--
2.43.0
[text/x-patch] v14-PG16-0001-Fix-exponential-memory-allocation-issue-in-.patch (17.4K, ../../CALDaNm3m6Dq2d6bCBaOjnABYavT3p+=VbJd5oJKok0sCTOSZog@mail.gmail.com/5-v14-PG16-0001-Fix-exponential-memory-allocation-issue-in-.patch)
download | inline diff:
From 5168c582b2a503f10074674723efbf3cc430637c Mon Sep 17 00:00:00 2001
From: Vignesh C <vignesh21@gmail.com>
Date: Thu, 12 Jun 2025 09:51:04 +0530
Subject: [PATCH v14-PG16] Fix exponential memory allocation issue in logical
decoding due to invalidation message distribution
Commit 4909b38af0 added logic to distribute invalidation messages from
one transaction to all concurrent in-progress transactions. However,
since the source of each invalidation message was not tracked, transactions
could redundantly re-distribute already propagated messages. This led to an
exponential increase in invalidation messages and, ultimately, memory
allocation failures.
To fix this, a new field is introduced to track distributed invalidation
messages separately. This ensures that only transactions in the pre-existing
list receive the invalidations, preventing redundant propagation.
The accumulated distributed invalidations are now consumed at the end of
transaction processing in `ReorderBufferProcessTXN()`.
---
.../expected/invalidation_distribution.out | 23 ++-
.../specs/invalidation_distribution.spec | 11 +
.../replication/logical/reorderbuffer.c | 195 +++++++++++++++---
src/backend/replication/logical/snapbuild.c | 12 +-
src/include/replication/reorderbuffer.h | 34 ++-
5 files changed, 230 insertions(+), 45 deletions(-)
diff --git a/contrib/test_decoding/expected/invalidation_distribution.out b/contrib/test_decoding/expected/invalidation_distribution.out
index ad0a944cbf3..ae53b1e61de 100644
--- a/contrib/test_decoding/expected/invalidation_distribution.out
+++ b/contrib/test_decoding/expected/invalidation_distribution.out
@@ -1,4 +1,4 @@
-Parsed test spec with 2 sessions
+Parsed test spec with 3 sessions
starting permutation: s1_insert_tbl1 s1_begin s1_insert_tbl1 s2_alter_pub_add_tbl s1_commit s1_insert_tbl1 s2_get_binary_changes
step s1_insert_tbl1: INSERT INTO tbl1 (val1, val2) VALUES (1, 1);
@@ -18,3 +18,24 @@ count
stop
(1 row)
+
+starting permutation: s1_begin s1_insert_tbl1 s3_begin s3_insert_tbl1 s2_alter_pub_add_tbl s1_insert_tbl1 s1_commit s3_commit s2_get_binary_changes
+step s1_begin: BEGIN;
+step s1_insert_tbl1: INSERT INTO tbl1 (val1, val2) VALUES (1, 1);
+step s3_begin: BEGIN;
+step s3_insert_tbl1: INSERT INTO tbl1 (val1, val2) VALUES (2, 2);
+step s2_alter_pub_add_tbl: ALTER PUBLICATION pub ADD TABLE tbl1;
+step s1_insert_tbl1: INSERT INTO tbl1 (val1, val2) VALUES (1, 1);
+step s1_commit: COMMIT;
+step s3_commit: COMMIT;
+step s2_get_binary_changes: SELECT count(data) FROM pg_logical_slot_get_binary_changes('isolation_slot', NULL, NULL, 'proto_version', '4', 'publication_names', 'pub') WHERE get_byte(data, 0) = 73;
+count
+-----
+ 1
+(1 row)
+
+?column?
+--------
+stop
+(1 row)
+
diff --git a/contrib/test_decoding/specs/invalidation_distribution.spec b/contrib/test_decoding/specs/invalidation_distribution.spec
index decbed627e3..f73cf6e482a 100644
--- a/contrib/test_decoding/specs/invalidation_distribution.spec
+++ b/contrib/test_decoding/specs/invalidation_distribution.spec
@@ -28,5 +28,16 @@ setup { SET synchronous_commit=on; }
step "s2_alter_pub_add_tbl" { ALTER PUBLICATION pub ADD TABLE tbl1; }
step "s2_get_binary_changes" { SELECT count(data) FROM pg_logical_slot_get_binary_changes('isolation_slot', NULL, NULL, 'proto_version', '4', 'publication_names', 'pub') WHERE get_byte(data, 0) = 73; }
+session "s3"
+setup { SET synchronous_commit=on; }
+step "s3_begin" { BEGIN; }
+step "s3_insert_tbl1" { INSERT INTO tbl1 (val1, val2) VALUES (2, 2); }
+step "s3_commit" { COMMIT; }
+
# Expect to get one insert change. LOGICAL_REP_MSG_INSERT = 'I'
permutation "s1_insert_tbl1" "s1_begin" "s1_insert_tbl1" "s2_alter_pub_add_tbl" "s1_commit" "s1_insert_tbl1" "s2_get_binary_changes"
+
+# Expect to get one insert change with LOGICAL_REP_MSG_INSERT = 'I' from
+# the second "s1_insert_tbl1" executed after adding the table tbl1 the
+# publication in "s2_alter_pub_add_tbl".
+permutation "s1_begin" "s1_insert_tbl1" "s3_begin" "s3_insert_tbl1" "s2_alter_pub_add_tbl" "s1_insert_tbl1" "s1_commit" "s3_commit" "s2_get_binary_changes"
diff --git a/src/backend/replication/logical/reorderbuffer.c b/src/backend/replication/logical/reorderbuffer.c
index fa04e829cc9..6464a926c20 100644
--- a/src/backend/replication/logical/reorderbuffer.c
+++ b/src/backend/replication/logical/reorderbuffer.c
@@ -103,11 +103,22 @@
#include "storage/sinval.h"
#include "utils/builtins.h"
#include "utils/combocid.h"
+#include "utils/inval.h"
#include "utils/memdebug.h"
#include "utils/memutils.h"
#include "utils/rel.h"
#include "utils/relfilenumbermap.h"
+/*
+ * Each transaction has an 8MB limit for invalidation messages distributed from
+ * other transactions. This limit is set considering scenarios with many
+ * concurrent logical decoding operations. When the distributed invalidation
+ * messages reach this threshold, the transaction is marked as
+ * RBTXN_DISTR_INVAL_OVERFLOWED to invalidate the complete cache as we have lost
+ * some inval messages and hence don't know what needs to be invalidated.
+ */
+#define MAX_DISTR_INVAL_MSG_PER_TXN \
+ ((8 * 1024 * 1024) / sizeof(SharedInvalidationMessage))
/* entry for a hash table we use to map from xid to our transaction state */
typedef struct ReorderBufferTXNByIdEnt
@@ -461,6 +472,12 @@ ReorderBufferReturnTXN(ReorderBuffer *rb, ReorderBufferTXN *txn)
txn->invalidations = NULL;
}
+ if (txn->invalidations_distributed)
+ {
+ pfree(txn->invalidations_distributed);
+ txn->invalidations_distributed = NULL;
+ }
+
/* Reset the toast hash */
ReorderBufferToastReset(rb, txn);
@@ -2539,7 +2556,17 @@ ReorderBufferProcessTXN(ReorderBuffer *rb, ReorderBufferTXN *txn,
AbortCurrentTransaction();
/* make sure there's no cache pollution */
- ReorderBufferExecuteInvalidations(txn->ninvalidations, txn->invalidations);
+ if (rbtxn_distr_inval_overflowed(txn))
+ {
+ Assert(txn->ninvalidations_distributed == 0);
+ InvalidateSystemCaches();
+ }
+ else
+ {
+ ReorderBufferExecuteInvalidations(txn->ninvalidations, txn->invalidations);
+ ReorderBufferExecuteInvalidations(txn->ninvalidations_distributed,
+ txn->invalidations_distributed);
+ }
if (using_subtxn)
RollbackAndReleaseCurrentSubTransaction();
@@ -2585,8 +2612,17 @@ ReorderBufferProcessTXN(ReorderBuffer *rb, ReorderBufferTXN *txn,
AbortCurrentTransaction();
/* make sure there's no cache pollution */
- ReorderBufferExecuteInvalidations(txn->ninvalidations,
- txn->invalidations);
+ if (rbtxn_distr_inval_overflowed(txn))
+ {
+ Assert(txn->ninvalidations_distributed == 0);
+ InvalidateSystemCaches();
+ }
+ else
+ {
+ ReorderBufferExecuteInvalidations(txn->ninvalidations, txn->invalidations);
+ ReorderBufferExecuteInvalidations(txn->ninvalidations_distributed,
+ txn->invalidations_distributed);
+ }
if (using_subtxn)
RollbackAndReleaseCurrentSubTransaction();
@@ -2916,7 +2952,8 @@ ReorderBufferAbort(ReorderBuffer *rb, TransactionId xid, XLogRecPtr lsn,
* We might have decoded changes for this transaction that could load
* the cache as per the current transaction's view (consider DDL's
* happened in this transaction). We don't want the decoding of future
- * transactions to use those cache entries so execute invalidations.
+ * transactions to use those cache entries so execute only the inval
+ * messages in this transaction.
*/
if (txn->ninvalidations > 0)
ReorderBufferImmediateInvalidation(rb, txn->ninvalidations,
@@ -3003,9 +3040,10 @@ ReorderBufferForget(ReorderBuffer *rb, TransactionId xid, XLogRecPtr lsn)
txn->final_lsn = lsn;
/*
- * Process cache invalidation messages if there are any. Even if we're not
- * interested in the transaction's contents, it could have manipulated the
- * catalog and we need to update the caches according to that.
+ * Process only cache invalidation messages in this transaction if there
+ * are any. Even if we're not interested in the transaction's contents, it
+ * could have manipulated the catalog and we need to update the caches
+ * according to that.
*/
if (txn->base_snapshot != NULL && txn->ninvalidations > 0)
ReorderBufferImmediateInvalidation(rb, txn->ninvalidations,
@@ -3257,6 +3295,57 @@ ReorderBufferAddNewTupleCids(ReorderBuffer *rb, TransactionId xid,
txn->ntuplecids++;
}
+/*
+ * Add new invalidation messages to the reorder buffer queue.
+ */
+static void
+ReorderBufferQueueInvalidations(ReorderBuffer *rb, TransactionId xid,
+ XLogRecPtr lsn, Size nmsgs,
+ SharedInvalidationMessage *msgs)
+{
+ ReorderBufferChange *change;
+
+ change = ReorderBufferGetChange(rb);
+ change->action = REORDER_BUFFER_CHANGE_INVALIDATION;
+ change->data.inval.ninvalidations = nmsgs;
+ change->data.inval.invalidations = (SharedInvalidationMessage *)
+ palloc(sizeof(SharedInvalidationMessage) * nmsgs);
+ memcpy(change->data.inval.invalidations, msgs,
+ sizeof(SharedInvalidationMessage) * nmsgs);
+
+ ReorderBufferQueueChange(rb, xid, lsn, change, false);
+}
+
+/*
+ * A helper function for ReorderBufferAddInvalidations() and
+ * ReorderBufferAddDistributedInvalidations() to accumulate the invalidation
+ * messages to the **invals_out.
+ */
+static void
+ReorderBufferAccumulateInvalidations(SharedInvalidationMessage **invals_out,
+ uint32 *ninvals_out,
+ SharedInvalidationMessage *msgs_new,
+ Size nmsgs_new)
+{
+ if (*ninvals_out == 0)
+ {
+ *ninvals_out = nmsgs_new;
+ *invals_out = (SharedInvalidationMessage *)
+ palloc(sizeof(SharedInvalidationMessage) * nmsgs_new);
+ memcpy(*invals_out, msgs_new, sizeof(SharedInvalidationMessage) * nmsgs_new);
+ }
+ else
+ {
+ /* Enlarge the array of inval messages */
+ *invals_out = (SharedInvalidationMessage *)
+ repalloc(*invals_out, sizeof(SharedInvalidationMessage) *
+ (*ninvals_out + nmsgs_new));
+ memcpy(*invals_out + *ninvals_out, msgs_new,
+ nmsgs_new * sizeof(SharedInvalidationMessage));
+ *ninvals_out += nmsgs_new;
+ }
+}
+
/*
* Accumulate the invalidations for executing them later.
*
@@ -3277,7 +3366,6 @@ ReorderBufferAddInvalidations(ReorderBuffer *rb, TransactionId xid,
{
ReorderBufferTXN *txn;
MemoryContext oldcontext;
- ReorderBufferChange *change;
txn = ReorderBufferTXNByXid(rb, xid, true, NULL, lsn, true);
@@ -3292,35 +3380,76 @@ ReorderBufferAddInvalidations(ReorderBuffer *rb, TransactionId xid,
Assert(nmsgs > 0);
- /* Accumulate invalidations. */
- if (txn->ninvalidations == 0)
- {
- txn->ninvalidations = nmsgs;
- txn->invalidations = (SharedInvalidationMessage *)
- palloc(sizeof(SharedInvalidationMessage) * nmsgs);
- memcpy(txn->invalidations, msgs,
- sizeof(SharedInvalidationMessage) * nmsgs);
- }
- else
+ ReorderBufferAccumulateInvalidations(&txn->invalidations,
+ &txn->ninvalidations,
+ msgs, nmsgs);
+
+ ReorderBufferQueueInvalidations(rb, xid, lsn, nmsgs, msgs);
+
+ MemoryContextSwitchTo(oldcontext);
+}
+
+/*
+ * Accumulate the invalidations distributed by other committed transactions
+ * for executing them later.
+ *
+ * This function is similar to ReorderBufferAddInvalidations() but stores
+ * the given inval messages to the txn->invalidations_distributed with the
+ * overflow check.
+ *
+ * This needs to be called by committed transactions to distribute their
+ * inval messages to in-progress transactions.
+ */
+void
+ReorderBufferAddDistributedInvalidations(ReorderBuffer *rb, TransactionId xid,
+ XLogRecPtr lsn, Size nmsgs,
+ SharedInvalidationMessage *msgs)
+{
+ ReorderBufferTXN *txn;
+ MemoryContext oldcontext;
+
+ txn = ReorderBufferTXNByXid(rb, xid, true, NULL, lsn, true);
+
+ oldcontext = MemoryContextSwitchTo(rb->context);
+
+ /*
+ * Collect all the invalidations under the top transaction, if available,
+ * so that we can execute them all together. See comments
+ * ReorderBufferAddInvalidations.
+ */
+ txn = rbtxn_get_toptxn(txn);
+
+ Assert(nmsgs > 0);
+
+ if (!rbtxn_distr_inval_overflowed(txn))
{
- txn->invalidations = (SharedInvalidationMessage *)
- repalloc(txn->invalidations, sizeof(SharedInvalidationMessage) *
- (txn->ninvalidations + nmsgs));
+ /*
+ * Check the transaction has enough space for storing distributed
+ * invalidation messages.
+ */
+ if (txn->ninvalidations_distributed + nmsgs >= MAX_DISTR_INVAL_MSG_PER_TXN)
+ {
+ /*
+ * Mark the invalidation message as overflowed and free up the
+ * messages accumulated so far.
+ */
+ txn->txn_flags |= RBTXN_DISTR_INVAL_OVERFLOWED;
- memcpy(txn->invalidations + txn->ninvalidations, msgs,
- nmsgs * sizeof(SharedInvalidationMessage));
- txn->ninvalidations += nmsgs;
+ if (txn->invalidations_distributed)
+ {
+ pfree(txn->invalidations_distributed);
+ txn->invalidations_distributed = NULL;
+ txn->ninvalidations_distributed = 0;
+ }
+ }
+ else
+ ReorderBufferAccumulateInvalidations(&txn->invalidations_distributed,
+ &txn->ninvalidations_distributed,
+ msgs, nmsgs);
}
- change = ReorderBufferGetChange(rb);
- change->action = REORDER_BUFFER_CHANGE_INVALIDATION;
- change->data.inval.ninvalidations = nmsgs;
- change->data.inval.invalidations = (SharedInvalidationMessage *)
- palloc(sizeof(SharedInvalidationMessage) * nmsgs);
- memcpy(change->data.inval.invalidations, msgs,
- sizeof(SharedInvalidationMessage) * nmsgs);
-
- ReorderBufferQueueChange(rb, xid, lsn, change, false);
+ /* Queue the invalidation messages into the transaction */
+ ReorderBufferQueueInvalidations(rb, xid, lsn, nmsgs, msgs);
MemoryContextSwitchTo(oldcontext);
}
diff --git a/src/backend/replication/logical/snapbuild.c b/src/backend/replication/logical/snapbuild.c
index d48ebb8337b..6c49dda75b6 100644
--- a/src/backend/replication/logical/snapbuild.c
+++ b/src/backend/replication/logical/snapbuild.c
@@ -918,6 +918,13 @@ SnapBuildDistributeSnapshotAndInval(SnapBuild *builder, XLogRecPtr lsn, Transact
* contents built by the current transaction even after its decoding,
* which should have been invalidated due to concurrent catalog
* changing transaction.
+ *
+ * Distribute only the invalidation messages generated by the current
+ * committed transaction. Invalidation messages received from other
+ * transactions would have already been propagated to the relevant
+ * in-progress transactions. This transaction would have processed
+ * those invalidations, ensuring that subsequent transactions observe
+ * a consistent cache state.
*/
if (txn->xid != xid)
{
@@ -931,8 +938,9 @@ SnapBuildDistributeSnapshotAndInval(SnapBuild *builder, XLogRecPtr lsn, Transact
{
Assert(msgs != NULL);
- ReorderBufferAddInvalidations(builder->reorder, txn->xid, lsn,
- ninvalidations, msgs);
+ ReorderBufferAddDistributedInvalidations(builder->reorder,
+ txn->xid, lsn,
+ ninvalidations, msgs);
}
}
}
diff --git a/src/include/replication/reorderbuffer.h b/src/include/replication/reorderbuffer.h
index 8a32bbe28df..1606e0ea007 100644
--- a/src/include/replication/reorderbuffer.h
+++ b/src/include/replication/reorderbuffer.h
@@ -177,15 +177,16 @@ typedef struct ReorderBufferChange
} ReorderBufferChange;
/* ReorderBufferTXN txn_flags */
-#define RBTXN_HAS_CATALOG_CHANGES 0x0001
-#define RBTXN_IS_SUBXACT 0x0002
-#define RBTXN_IS_SERIALIZED 0x0004
-#define RBTXN_IS_SERIALIZED_CLEAR 0x0008
-#define RBTXN_IS_STREAMED 0x0010
-#define RBTXN_HAS_PARTIAL_CHANGE 0x0020
-#define RBTXN_PREPARE 0x0040
-#define RBTXN_SKIPPED_PREPARE 0x0080
-#define RBTXN_HAS_STREAMABLE_CHANGE 0x0100
+#define RBTXN_HAS_CATALOG_CHANGES 0x0001
+#define RBTXN_IS_SUBXACT 0x0002
+#define RBTXN_IS_SERIALIZED 0x0004
+#define RBTXN_IS_SERIALIZED_CLEAR 0x0008
+#define RBTXN_IS_STREAMED 0x0010
+#define RBTXN_HAS_PARTIAL_CHANGE 0x0020
+#define RBTXN_PREPARE 0x0040
+#define RBTXN_SKIPPED_PREPARE 0x0080
+#define RBTXN_HAS_STREAMABLE_CHANGE 0x0100
+#define RBTXN_DISTR_INVAL_OVERFLOWED 0x0200
/* Does the transaction have catalog changes? */
#define rbtxn_has_catalog_changes(txn) \
@@ -249,6 +250,12 @@ typedef struct ReorderBufferChange
((txn)->txn_flags & RBTXN_SKIPPED_PREPARE) != 0 \
)
+/* Is the array of distributed inval messages overflowed? */
+#define rbtxn_distr_inval_overflowed(txn) \
+( \
+ ((txn)->txn_flags & RBTXN_DISTR_INVAL_OVERFLOWED) != 0 \
+)
+
/* Is this a top-level transaction? */
#define rbtxn_is_toptxn(txn) \
( \
@@ -435,6 +442,12 @@ typedef struct ReorderBufferTXN
* Private data pointer of the output plugin.
*/
void *output_plugin_private;
+
+ /*
+ * Stores cache invalidation messages distributed by other transactions.
+ */
+ uint32 ninvalidations_distributed;
+ SharedInvalidationMessage *invalidations_distributed;
} ReorderBufferTXN;
/* so we can define the callbacks used inside struct ReorderBuffer itself */
@@ -728,6 +741,9 @@ extern void ReorderBufferAddNewTupleCids(ReorderBuffer *rb, TransactionId xid,
CommandId cmin, CommandId cmax, CommandId combocid);
extern void ReorderBufferAddInvalidations(ReorderBuffer *rb, TransactionId xid, XLogRecPtr lsn,
Size nmsgs, SharedInvalidationMessage *msgs);
+extern void ReorderBufferAddDistributedInvalidations(ReorderBuffer *rb, TransactionId xid,
+ XLogRecPtr lsn, Size nmsgs,
+ SharedInvalidationMessage *msgs);
extern void ReorderBufferImmediateInvalidation(ReorderBuffer *rb, uint32 ninvalidations,
SharedInvalidationMessage *invalidations);
extern void ReorderBufferProcessXid(ReorderBuffer *rb, TransactionId xid, XLogRecPtr lsn);
--
2.43.0
[text/x-patch] v14-PG17-0001-Fix-exponential-memory-allocation-issue-in-.patch (17.4K, ../../CALDaNm3m6Dq2d6bCBaOjnABYavT3p+=VbJd5oJKok0sCTOSZog@mail.gmail.com/6-v14-PG17-0001-Fix-exponential-memory-allocation-issue-in-.patch)
download | inline diff:
From ab0775142ead69a4bb3a5d8dec0cd28dd5d14f03 Mon Sep 17 00:00:00 2001
From: Vignesh C <vignesh21@gmail.com>
Date: Thu, 12 Jun 2025 09:38:58 +0530
Subject: [PATCH v14-PG17] Fix exponential memory allocation issue in logical
decoding due to invalidation message distribution
Commit 4909b38af0 added logic to distribute invalidation messages from
one transaction to all concurrent in-progress transactions. However,
since the source of each invalidation message was not tracked, transactions
could redundantly re-distribute already propagated messages. This led to an
exponential increase in invalidation messages and, ultimately, memory
allocation failures.
To fix this, a new field is introduced to track distributed invalidation
messages separately. This ensures that only transactions in the pre-existing
list receive the invalidations, preventing redundant propagation.
The accumulated distributed invalidations are now consumed at the end of
transaction processing in `ReorderBufferProcessTXN()`.
---
.../expected/invalidation_distribution.out | 23 +-
.../specs/invalidation_distribution.spec | 11 +
.../replication/logical/reorderbuffer.c | 196 +++++++++++++++---
src/backend/replication/logical/snapbuild.c | 12 +-
src/include/replication/reorderbuffer.h | 34 ++-
5 files changed, 231 insertions(+), 45 deletions(-)
diff --git a/contrib/test_decoding/expected/invalidation_distribution.out b/contrib/test_decoding/expected/invalidation_distribution.out
index ad0a944cbf3..ae53b1e61de 100644
--- a/contrib/test_decoding/expected/invalidation_distribution.out
+++ b/contrib/test_decoding/expected/invalidation_distribution.out
@@ -1,4 +1,4 @@
-Parsed test spec with 2 sessions
+Parsed test spec with 3 sessions
starting permutation: s1_insert_tbl1 s1_begin s1_insert_tbl1 s2_alter_pub_add_tbl s1_commit s1_insert_tbl1 s2_get_binary_changes
step s1_insert_tbl1: INSERT INTO tbl1 (val1, val2) VALUES (1, 1);
@@ -18,3 +18,24 @@ count
stop
(1 row)
+
+starting permutation: s1_begin s1_insert_tbl1 s3_begin s3_insert_tbl1 s2_alter_pub_add_tbl s1_insert_tbl1 s1_commit s3_commit s2_get_binary_changes
+step s1_begin: BEGIN;
+step s1_insert_tbl1: INSERT INTO tbl1 (val1, val2) VALUES (1, 1);
+step s3_begin: BEGIN;
+step s3_insert_tbl1: INSERT INTO tbl1 (val1, val2) VALUES (2, 2);
+step s2_alter_pub_add_tbl: ALTER PUBLICATION pub ADD TABLE tbl1;
+step s1_insert_tbl1: INSERT INTO tbl1 (val1, val2) VALUES (1, 1);
+step s1_commit: COMMIT;
+step s3_commit: COMMIT;
+step s2_get_binary_changes: SELECT count(data) FROM pg_logical_slot_get_binary_changes('isolation_slot', NULL, NULL, 'proto_version', '4', 'publication_names', 'pub') WHERE get_byte(data, 0) = 73;
+count
+-----
+ 1
+(1 row)
+
+?column?
+--------
+stop
+(1 row)
+
diff --git a/contrib/test_decoding/specs/invalidation_distribution.spec b/contrib/test_decoding/specs/invalidation_distribution.spec
index decbed627e3..f73cf6e482a 100644
--- a/contrib/test_decoding/specs/invalidation_distribution.spec
+++ b/contrib/test_decoding/specs/invalidation_distribution.spec
@@ -28,5 +28,16 @@ setup { SET synchronous_commit=on; }
step "s2_alter_pub_add_tbl" { ALTER PUBLICATION pub ADD TABLE tbl1; }
step "s2_get_binary_changes" { SELECT count(data) FROM pg_logical_slot_get_binary_changes('isolation_slot', NULL, NULL, 'proto_version', '4', 'publication_names', 'pub') WHERE get_byte(data, 0) = 73; }
+session "s3"
+setup { SET synchronous_commit=on; }
+step "s3_begin" { BEGIN; }
+step "s3_insert_tbl1" { INSERT INTO tbl1 (val1, val2) VALUES (2, 2); }
+step "s3_commit" { COMMIT; }
+
# Expect to get one insert change. LOGICAL_REP_MSG_INSERT = 'I'
permutation "s1_insert_tbl1" "s1_begin" "s1_insert_tbl1" "s2_alter_pub_add_tbl" "s1_commit" "s1_insert_tbl1" "s2_get_binary_changes"
+
+# Expect to get one insert change with LOGICAL_REP_MSG_INSERT = 'I' from
+# the second "s1_insert_tbl1" executed after adding the table tbl1 the
+# publication in "s2_alter_pub_add_tbl".
+permutation "s1_begin" "s1_insert_tbl1" "s3_begin" "s3_insert_tbl1" "s2_alter_pub_add_tbl" "s1_insert_tbl1" "s1_commit" "s3_commit" "s2_get_binary_changes"
diff --git a/src/backend/replication/logical/reorderbuffer.c b/src/backend/replication/logical/reorderbuffer.c
index 03eb005c39d..d85a9f2bb20 100644
--- a/src/backend/replication/logical/reorderbuffer.c
+++ b/src/backend/replication/logical/reorderbuffer.c
@@ -108,10 +108,22 @@
#include "storage/fd.h"
#include "storage/sinval.h"
#include "utils/builtins.h"
+#include "utils/inval.h"
#include "utils/memutils.h"
#include "utils/rel.h"
#include "utils/relfilenumbermap.h"
+/*
+ * Each transaction has an 8MB limit for invalidation messages distributed from
+ * other transactions. This limit is set considering scenarios with many
+ * concurrent logical decoding operations. When the distributed invalidation
+ * messages reach this threshold, the transaction is marked as
+ * RBTXN_DISTR_INVAL_OVERFLOWED to invalidate the complete cache as we have lost
+ * some inval messages and hence don't know what needs to be invalidated.
+ */
+#define MAX_DISTR_INVAL_MSG_PER_TXN \
+ ((8 * 1024 * 1024) / sizeof(SharedInvalidationMessage))
+
/* entry for a hash table we use to map from xid to our transaction state */
typedef struct ReorderBufferTXNByIdEnt
{
@@ -469,6 +481,12 @@ ReorderBufferReturnTXN(ReorderBuffer *rb, ReorderBufferTXN *txn)
txn->invalidations = NULL;
}
+ if (txn->invalidations_distributed)
+ {
+ pfree(txn->invalidations_distributed);
+ txn->invalidations_distributed = NULL;
+ }
+
/* Reset the toast hash */
ReorderBufferToastReset(rb, txn);
@@ -2574,7 +2592,17 @@ ReorderBufferProcessTXN(ReorderBuffer *rb, ReorderBufferTXN *txn,
AbortCurrentTransaction();
/* make sure there's no cache pollution */
- ReorderBufferExecuteInvalidations(txn->ninvalidations, txn->invalidations);
+ if (rbtxn_distr_inval_overflowed(txn))
+ {
+ Assert(txn->ninvalidations_distributed == 0);
+ InvalidateSystemCaches();
+ }
+ else
+ {
+ ReorderBufferExecuteInvalidations(txn->ninvalidations, txn->invalidations);
+ ReorderBufferExecuteInvalidations(txn->ninvalidations_distributed,
+ txn->invalidations_distributed);
+ }
if (using_subtxn)
RollbackAndReleaseCurrentSubTransaction();
@@ -2620,8 +2648,17 @@ ReorderBufferProcessTXN(ReorderBuffer *rb, ReorderBufferTXN *txn,
AbortCurrentTransaction();
/* make sure there's no cache pollution */
- ReorderBufferExecuteInvalidations(txn->ninvalidations,
- txn->invalidations);
+ if (rbtxn_distr_inval_overflowed(txn))
+ {
+ Assert(txn->ninvalidations_distributed == 0);
+ InvalidateSystemCaches();
+ }
+ else
+ {
+ ReorderBufferExecuteInvalidations(txn->ninvalidations, txn->invalidations);
+ ReorderBufferExecuteInvalidations(txn->ninvalidations_distributed,
+ txn->invalidations_distributed);
+ }
if (using_subtxn)
RollbackAndReleaseCurrentSubTransaction();
@@ -2951,7 +2988,8 @@ ReorderBufferAbort(ReorderBuffer *rb, TransactionId xid, XLogRecPtr lsn,
* We might have decoded changes for this transaction that could load
* the cache as per the current transaction's view (consider DDL's
* happened in this transaction). We don't want the decoding of future
- * transactions to use those cache entries so execute invalidations.
+ * transactions to use those cache entries so execute only the inval
+ * messages in this transaction.
*/
if (txn->ninvalidations > 0)
ReorderBufferImmediateInvalidation(rb, txn->ninvalidations,
@@ -3038,9 +3076,10 @@ ReorderBufferForget(ReorderBuffer *rb, TransactionId xid, XLogRecPtr lsn)
txn->final_lsn = lsn;
/*
- * Process cache invalidation messages if there are any. Even if we're not
- * interested in the transaction's contents, it could have manipulated the
- * catalog and we need to update the caches according to that.
+ * Process only cache invalidation messages in this transaction if there
+ * are any. Even if we're not interested in the transaction's contents, it
+ * could have manipulated the catalog and we need to update the caches
+ * according to that.
*/
if (txn->base_snapshot != NULL && txn->ninvalidations > 0)
ReorderBufferImmediateInvalidation(rb, txn->ninvalidations,
@@ -3312,6 +3351,57 @@ ReorderBufferAddNewTupleCids(ReorderBuffer *rb, TransactionId xid,
txn->ntuplecids++;
}
+/*
+ * Add new invalidation messages to the reorder buffer queue.
+ */
+static void
+ReorderBufferQueueInvalidations(ReorderBuffer *rb, TransactionId xid,
+ XLogRecPtr lsn, Size nmsgs,
+ SharedInvalidationMessage *msgs)
+{
+ ReorderBufferChange *change;
+
+ change = ReorderBufferGetChange(rb);
+ change->action = REORDER_BUFFER_CHANGE_INVALIDATION;
+ change->data.inval.ninvalidations = nmsgs;
+ change->data.inval.invalidations = (SharedInvalidationMessage *)
+ palloc(sizeof(SharedInvalidationMessage) * nmsgs);
+ memcpy(change->data.inval.invalidations, msgs,
+ sizeof(SharedInvalidationMessage) * nmsgs);
+
+ ReorderBufferQueueChange(rb, xid, lsn, change, false);
+}
+
+/*
+ * A helper function for ReorderBufferAddInvalidations() and
+ * ReorderBufferAddDistributedInvalidations() to accumulate the invalidation
+ * messages to the **invals_out.
+ */
+static void
+ReorderBufferAccumulateInvalidations(SharedInvalidationMessage **invals_out,
+ uint32 *ninvals_out,
+ SharedInvalidationMessage *msgs_new,
+ Size nmsgs_new)
+{
+ if (*ninvals_out == 0)
+ {
+ *ninvals_out = nmsgs_new;
+ *invals_out = (SharedInvalidationMessage *)
+ palloc(sizeof(SharedInvalidationMessage) * nmsgs_new);
+ memcpy(*invals_out, msgs_new, sizeof(SharedInvalidationMessage) * nmsgs_new);
+ }
+ else
+ {
+ /* Enlarge the array of inval messages */
+ *invals_out = (SharedInvalidationMessage *)
+ repalloc(*invals_out, sizeof(SharedInvalidationMessage) *
+ (*ninvals_out + nmsgs_new));
+ memcpy(*invals_out + *ninvals_out, msgs_new,
+ nmsgs_new * sizeof(SharedInvalidationMessage));
+ *ninvals_out += nmsgs_new;
+ }
+}
+
/*
* Accumulate the invalidations for executing them later.
*
@@ -3332,7 +3422,6 @@ ReorderBufferAddInvalidations(ReorderBuffer *rb, TransactionId xid,
{
ReorderBufferTXN *txn;
MemoryContext oldcontext;
- ReorderBufferChange *change;
txn = ReorderBufferTXNByXid(rb, xid, true, NULL, lsn, true);
@@ -3347,35 +3436,76 @@ ReorderBufferAddInvalidations(ReorderBuffer *rb, TransactionId xid,
Assert(nmsgs > 0);
- /* Accumulate invalidations. */
- if (txn->ninvalidations == 0)
- {
- txn->ninvalidations = nmsgs;
- txn->invalidations = (SharedInvalidationMessage *)
- palloc(sizeof(SharedInvalidationMessage) * nmsgs);
- memcpy(txn->invalidations, msgs,
- sizeof(SharedInvalidationMessage) * nmsgs);
- }
- else
+ ReorderBufferAccumulateInvalidations(&txn->invalidations,
+ &txn->ninvalidations,
+ msgs, nmsgs);
+
+ ReorderBufferQueueInvalidations(rb, xid, lsn, nmsgs, msgs);
+
+ MemoryContextSwitchTo(oldcontext);
+}
+
+/*
+ * Accumulate the invalidations distributed by other committed transactions
+ * for executing them later.
+ *
+ * This function is similar to ReorderBufferAddInvalidations() but stores
+ * the given inval messages to the txn->invalidations_distributed with the
+ * overflow check.
+ *
+ * This needs to be called by committed transactions to distribute their
+ * inval messages to in-progress transactions.
+ */
+void
+ReorderBufferAddDistributedInvalidations(ReorderBuffer *rb, TransactionId xid,
+ XLogRecPtr lsn, Size nmsgs,
+ SharedInvalidationMessage *msgs)
+{
+ ReorderBufferTXN *txn;
+ MemoryContext oldcontext;
+
+ txn = ReorderBufferTXNByXid(rb, xid, true, NULL, lsn, true);
+
+ oldcontext = MemoryContextSwitchTo(rb->context);
+
+ /*
+ * Collect all the invalidations under the top transaction, if available,
+ * so that we can execute them all together. See comments
+ * ReorderBufferAddInvalidations.
+ */
+ txn = rbtxn_get_toptxn(txn);
+
+ Assert(nmsgs > 0);
+
+ if (!rbtxn_distr_inval_overflowed(txn))
{
- txn->invalidations = (SharedInvalidationMessage *)
- repalloc(txn->invalidations, sizeof(SharedInvalidationMessage) *
- (txn->ninvalidations + nmsgs));
+ /*
+ * Check the transaction has enough space for storing distributed
+ * invalidation messages.
+ */
+ if (txn->ninvalidations_distributed + nmsgs >= MAX_DISTR_INVAL_MSG_PER_TXN)
+ {
+ /*
+ * Mark the invalidation message as overflowed and free up the
+ * messages accumulated so far.
+ */
+ txn->txn_flags |= RBTXN_DISTR_INVAL_OVERFLOWED;
- memcpy(txn->invalidations + txn->ninvalidations, msgs,
- nmsgs * sizeof(SharedInvalidationMessage));
- txn->ninvalidations += nmsgs;
+ if (txn->invalidations_distributed)
+ {
+ pfree(txn->invalidations_distributed);
+ txn->invalidations_distributed = NULL;
+ txn->ninvalidations_distributed = 0;
+ }
+ }
+ else
+ ReorderBufferAccumulateInvalidations(&txn->invalidations_distributed,
+ &txn->ninvalidations_distributed,
+ msgs, nmsgs);
}
- change = ReorderBufferGetChange(rb);
- change->action = REORDER_BUFFER_CHANGE_INVALIDATION;
- change->data.inval.ninvalidations = nmsgs;
- change->data.inval.invalidations = (SharedInvalidationMessage *)
- palloc(sizeof(SharedInvalidationMessage) * nmsgs);
- memcpy(change->data.inval.invalidations, msgs,
- sizeof(SharedInvalidationMessage) * nmsgs);
-
- ReorderBufferQueueChange(rb, xid, lsn, change, false);
+ /* Queue the invalidation messages into the transaction */
+ ReorderBufferQueueInvalidations(rb, xid, lsn, nmsgs, msgs);
MemoryContextSwitchTo(oldcontext);
}
diff --git a/src/backend/replication/logical/snapbuild.c b/src/backend/replication/logical/snapbuild.c
index 110e0b0a044..5c3bbf0e93f 100644
--- a/src/backend/replication/logical/snapbuild.c
+++ b/src/backend/replication/logical/snapbuild.c
@@ -932,6 +932,13 @@ SnapBuildDistributeSnapshotAndInval(SnapBuild *builder, XLogRecPtr lsn, Transact
* contents built by the current transaction even after its decoding,
* which should have been invalidated due to concurrent catalog
* changing transaction.
+ *
+ * Distribute only the invalidation messages generated by the current
+ * committed transaction. Invalidation messages received from other
+ * transactions would have already been propagated to the relevant
+ * in-progress transactions. This transaction would have processed
+ * those invalidations, ensuring that subsequent transactions observe
+ * a consistent cache state.
*/
if (txn->xid != xid)
{
@@ -945,8 +952,9 @@ SnapBuildDistributeSnapshotAndInval(SnapBuild *builder, XLogRecPtr lsn, Transact
{
Assert(msgs != NULL);
- ReorderBufferAddInvalidations(builder->reorder, txn->xid, lsn,
- ninvalidations, msgs);
+ ReorderBufferAddDistributedInvalidations(builder->reorder,
+ txn->xid, lsn,
+ ninvalidations, msgs);
}
}
}
diff --git a/src/include/replication/reorderbuffer.h b/src/include/replication/reorderbuffer.h
index 4c56f219fd8..0cfa9005141 100644
--- a/src/include/replication/reorderbuffer.h
+++ b/src/include/replication/reorderbuffer.h
@@ -159,15 +159,16 @@ typedef struct ReorderBufferChange
} ReorderBufferChange;
/* ReorderBufferTXN txn_flags */
-#define RBTXN_HAS_CATALOG_CHANGES 0x0001
-#define RBTXN_IS_SUBXACT 0x0002
-#define RBTXN_IS_SERIALIZED 0x0004
-#define RBTXN_IS_SERIALIZED_CLEAR 0x0008
-#define RBTXN_IS_STREAMED 0x0010
-#define RBTXN_HAS_PARTIAL_CHANGE 0x0020
-#define RBTXN_PREPARE 0x0040
-#define RBTXN_SKIPPED_PREPARE 0x0080
-#define RBTXN_HAS_STREAMABLE_CHANGE 0x0100
+#define RBTXN_HAS_CATALOG_CHANGES 0x0001
+#define RBTXN_IS_SUBXACT 0x0002
+#define RBTXN_IS_SERIALIZED 0x0004
+#define RBTXN_IS_SERIALIZED_CLEAR 0x0008
+#define RBTXN_IS_STREAMED 0x0010
+#define RBTXN_HAS_PARTIAL_CHANGE 0x0020
+#define RBTXN_PREPARE 0x0040
+#define RBTXN_SKIPPED_PREPARE 0x0080
+#define RBTXN_HAS_STREAMABLE_CHANGE 0x0100
+#define RBTXN_DISTR_INVAL_OVERFLOWED 0x0200
/* Does the transaction have catalog changes? */
#define rbtxn_has_catalog_changes(txn) \
@@ -231,6 +232,12 @@ typedef struct ReorderBufferChange
((txn)->txn_flags & RBTXN_SKIPPED_PREPARE) != 0 \
)
+/* Is the array of distributed inval messages overflowed? */
+#define rbtxn_distr_inval_overflowed(txn) \
+( \
+ ((txn)->txn_flags & RBTXN_DISTR_INVAL_OVERFLOWED) != 0 \
+)
+
/* Is this a top-level transaction? */
#define rbtxn_is_toptxn(txn) \
( \
@@ -422,6 +429,12 @@ typedef struct ReorderBufferTXN
* Private data pointer of the output plugin.
*/
void *output_plugin_private;
+
+ /*
+ * Stores cache invalidation messages distributed by other transactions.
+ */
+ uint32 ninvalidations_distributed;
+ SharedInvalidationMessage *invalidations_distributed;
} ReorderBufferTXN;
/* so we can define the callbacks used inside struct ReorderBuffer itself */
@@ -709,6 +722,9 @@ extern void ReorderBufferAddNewTupleCids(ReorderBuffer *rb, TransactionId xid,
CommandId cmin, CommandId cmax, CommandId combocid);
extern void ReorderBufferAddInvalidations(ReorderBuffer *rb, TransactionId xid, XLogRecPtr lsn,
Size nmsgs, SharedInvalidationMessage *msgs);
+extern void ReorderBufferAddDistributedInvalidations(ReorderBuffer *rb, TransactionId xid,
+ XLogRecPtr lsn, Size nmsgs,
+ SharedInvalidationMessage *msgs);
extern void ReorderBufferImmediateInvalidation(ReorderBuffer *rb, uint32 ninvalidations,
SharedInvalidationMessage *invalidations);
extern void ReorderBufferProcessXid(ReorderBuffer *rb, TransactionId xid, XLogRecPtr lsn);
--
2.43.0
[text/x-patch] v14-master-0001-Fix-exponential-memory-allocation-issue-i.patch (16.7K, ../../CALDaNm3m6Dq2d6bCBaOjnABYavT3p+=VbJd5oJKok0sCTOSZog@mail.gmail.com/7-v14-master-0001-Fix-exponential-memory-allocation-issue-i.patch)
download | inline diff:
From 76a93cb7b827c7fee04a2752009f835d93525042 Mon Sep 17 00:00:00 2001
From: Vignesh C <vignesh21@gmail.com>
Date: Thu, 12 Jun 2025 08:48:57 +0530
Subject: [PATCH v14-master] Fix exponential memory allocation issue in logical
decoding due to invalidation message distribution
Commit 4909b38af0 added logic to distribute invalidation messages from
one transaction to all concurrent in-progress transactions. However,
since the source of each invalidation message was not tracked, transactions
could redundantly re-distribute already propagated messages. This led to an
exponential increase in invalidation messages and, ultimately, memory
allocation failures.
To fix this, a new field is introduced to track distributed invalidation
messages separately. This ensures that only transactions in the pre-existing
list receive the invalidations, preventing redundant propagation.
The accumulated distributed invalidations are now consumed at the end of
transaction processing in `ReorderBufferProcessTXN()`.
---
.../expected/invalidation_distribution.out | 23 +-
.../specs/invalidation_distribution.spec | 11 +
.../replication/logical/reorderbuffer.c | 196 +++++++++++++++---
src/backend/replication/logical/snapbuild.c | 12 +-
src/include/replication/reorderbuffer.h | 16 ++
5 files changed, 222 insertions(+), 36 deletions(-)
diff --git a/contrib/test_decoding/expected/invalidation_distribution.out b/contrib/test_decoding/expected/invalidation_distribution.out
index ad0a944cbf3..ae53b1e61de 100644
--- a/contrib/test_decoding/expected/invalidation_distribution.out
+++ b/contrib/test_decoding/expected/invalidation_distribution.out
@@ -1,4 +1,4 @@
-Parsed test spec with 2 sessions
+Parsed test spec with 3 sessions
starting permutation: s1_insert_tbl1 s1_begin s1_insert_tbl1 s2_alter_pub_add_tbl s1_commit s1_insert_tbl1 s2_get_binary_changes
step s1_insert_tbl1: INSERT INTO tbl1 (val1, val2) VALUES (1, 1);
@@ -18,3 +18,24 @@ count
stop
(1 row)
+
+starting permutation: s1_begin s1_insert_tbl1 s3_begin s3_insert_tbl1 s2_alter_pub_add_tbl s1_insert_tbl1 s1_commit s3_commit s2_get_binary_changes
+step s1_begin: BEGIN;
+step s1_insert_tbl1: INSERT INTO tbl1 (val1, val2) VALUES (1, 1);
+step s3_begin: BEGIN;
+step s3_insert_tbl1: INSERT INTO tbl1 (val1, val2) VALUES (2, 2);
+step s2_alter_pub_add_tbl: ALTER PUBLICATION pub ADD TABLE tbl1;
+step s1_insert_tbl1: INSERT INTO tbl1 (val1, val2) VALUES (1, 1);
+step s1_commit: COMMIT;
+step s3_commit: COMMIT;
+step s2_get_binary_changes: SELECT count(data) FROM pg_logical_slot_get_binary_changes('isolation_slot', NULL, NULL, 'proto_version', '4', 'publication_names', 'pub') WHERE get_byte(data, 0) = 73;
+count
+-----
+ 1
+(1 row)
+
+?column?
+--------
+stop
+(1 row)
+
diff --git a/contrib/test_decoding/specs/invalidation_distribution.spec b/contrib/test_decoding/specs/invalidation_distribution.spec
index decbed627e3..f73cf6e482a 100644
--- a/contrib/test_decoding/specs/invalidation_distribution.spec
+++ b/contrib/test_decoding/specs/invalidation_distribution.spec
@@ -28,5 +28,16 @@ setup { SET synchronous_commit=on; }
step "s2_alter_pub_add_tbl" { ALTER PUBLICATION pub ADD TABLE tbl1; }
step "s2_get_binary_changes" { SELECT count(data) FROM pg_logical_slot_get_binary_changes('isolation_slot', NULL, NULL, 'proto_version', '4', 'publication_names', 'pub') WHERE get_byte(data, 0) = 73; }
+session "s3"
+setup { SET synchronous_commit=on; }
+step "s3_begin" { BEGIN; }
+step "s3_insert_tbl1" { INSERT INTO tbl1 (val1, val2) VALUES (2, 2); }
+step "s3_commit" { COMMIT; }
+
# Expect to get one insert change. LOGICAL_REP_MSG_INSERT = 'I'
permutation "s1_insert_tbl1" "s1_begin" "s1_insert_tbl1" "s2_alter_pub_add_tbl" "s1_commit" "s1_insert_tbl1" "s2_get_binary_changes"
+
+# Expect to get one insert change with LOGICAL_REP_MSG_INSERT = 'I' from
+# the second "s1_insert_tbl1" executed after adding the table tbl1 the
+# publication in "s2_alter_pub_add_tbl".
+permutation "s1_begin" "s1_insert_tbl1" "s3_begin" "s3_insert_tbl1" "s2_alter_pub_add_tbl" "s1_insert_tbl1" "s1_commit" "s3_commit" "s2_get_binary_changes"
diff --git a/src/backend/replication/logical/reorderbuffer.c b/src/backend/replication/logical/reorderbuffer.c
index 67655111875..c4299c76fb1 100644
--- a/src/backend/replication/logical/reorderbuffer.c
+++ b/src/backend/replication/logical/reorderbuffer.c
@@ -109,10 +109,22 @@
#include "storage/procarray.h"
#include "storage/sinval.h"
#include "utils/builtins.h"
+#include "utils/inval.h"
#include "utils/memutils.h"
#include "utils/rel.h"
#include "utils/relfilenumbermap.h"
+/*
+ * Each transaction has an 8MB limit for invalidation messages distributed from
+ * other transactions. This limit is set considering scenarios with many
+ * concurrent logical decoding operations. When the distributed invalidation
+ * messages reach this threshold, the transaction is marked as
+ * RBTXN_DISTR_INVAL_OVERFLOWED to invalidate the complete cache as we have lost
+ * some inval messages and hence don't know what needs to be invalidated.
+ */
+#define MAX_DISTR_INVAL_MSG_PER_TXN \
+ ((8 * 1024 * 1024) / sizeof(SharedInvalidationMessage))
+
/* entry for a hash table we use to map from xid to our transaction state */
typedef struct ReorderBufferTXNByIdEnt
{
@@ -472,6 +484,12 @@ ReorderBufferFreeTXN(ReorderBuffer *rb, ReorderBufferTXN *txn)
txn->invalidations = NULL;
}
+ if (txn->invalidations_distributed)
+ {
+ pfree(txn->invalidations_distributed);
+ txn->invalidations_distributed = NULL;
+ }
+
/* Reset the toast hash */
ReorderBufferToastReset(rb, txn);
@@ -2661,7 +2679,17 @@ ReorderBufferProcessTXN(ReorderBuffer *rb, ReorderBufferTXN *txn,
AbortCurrentTransaction();
/* make sure there's no cache pollution */
- ReorderBufferExecuteInvalidations(txn->ninvalidations, txn->invalidations);
+ if (rbtxn_distr_inval_overflowed(txn))
+ {
+ Assert(txn->ninvalidations_distributed == 0);
+ InvalidateSystemCaches();
+ }
+ else
+ {
+ ReorderBufferExecuteInvalidations(txn->ninvalidations, txn->invalidations);
+ ReorderBufferExecuteInvalidations(txn->ninvalidations_distributed,
+ txn->invalidations_distributed);
+ }
if (using_subtxn)
RollbackAndReleaseCurrentSubTransaction();
@@ -2710,8 +2738,17 @@ ReorderBufferProcessTXN(ReorderBuffer *rb, ReorderBufferTXN *txn,
AbortCurrentTransaction();
/* make sure there's no cache pollution */
- ReorderBufferExecuteInvalidations(txn->ninvalidations,
- txn->invalidations);
+ if (rbtxn_distr_inval_overflowed(txn))
+ {
+ Assert(txn->ninvalidations_distributed == 0);
+ InvalidateSystemCaches();
+ }
+ else
+ {
+ ReorderBufferExecuteInvalidations(txn->ninvalidations, txn->invalidations);
+ ReorderBufferExecuteInvalidations(txn->ninvalidations_distributed,
+ txn->invalidations_distributed);
+ }
if (using_subtxn)
RollbackAndReleaseCurrentSubTransaction();
@@ -3060,7 +3097,8 @@ ReorderBufferAbort(ReorderBuffer *rb, TransactionId xid, XLogRecPtr lsn,
* We might have decoded changes for this transaction that could load
* the cache as per the current transaction's view (consider DDL's
* happened in this transaction). We don't want the decoding of future
- * transactions to use those cache entries so execute invalidations.
+ * transactions to use those cache entries so execute only the inval
+ * messages in this transaction.
*/
if (txn->ninvalidations > 0)
ReorderBufferImmediateInvalidation(rb, txn->ninvalidations,
@@ -3147,9 +3185,10 @@ ReorderBufferForget(ReorderBuffer *rb, TransactionId xid, XLogRecPtr lsn)
txn->final_lsn = lsn;
/*
- * Process cache invalidation messages if there are any. Even if we're not
- * interested in the transaction's contents, it could have manipulated the
- * catalog and we need to update the caches according to that.
+ * Process only cache invalidation messages in this transaction if there
+ * are any. Even if we're not interested in the transaction's contents, it
+ * could have manipulated the catalog and we need to update the caches
+ * according to that.
*/
if (txn->base_snapshot != NULL && txn->ninvalidations > 0)
ReorderBufferImmediateInvalidation(rb, txn->ninvalidations,
@@ -3421,6 +3460,57 @@ ReorderBufferAddNewTupleCids(ReorderBuffer *rb, TransactionId xid,
txn->ntuplecids++;
}
+/*
+ * Add new invalidation messages to the reorder buffer queue.
+ */
+static void
+ReorderBufferQueueInvalidations(ReorderBuffer *rb, TransactionId xid,
+ XLogRecPtr lsn, Size nmsgs,
+ SharedInvalidationMessage *msgs)
+{
+ ReorderBufferChange *change;
+
+ change = ReorderBufferAllocChange(rb);
+ change->action = REORDER_BUFFER_CHANGE_INVALIDATION;
+ change->data.inval.ninvalidations = nmsgs;
+ change->data.inval.invalidations = (SharedInvalidationMessage *)
+ palloc(sizeof(SharedInvalidationMessage) * nmsgs);
+ memcpy(change->data.inval.invalidations, msgs,
+ sizeof(SharedInvalidationMessage) * nmsgs);
+
+ ReorderBufferQueueChange(rb, xid, lsn, change, false);
+}
+
+/*
+ * A helper function for ReorderBufferAddInvalidations() and
+ * ReorderBufferAddDistributedInvalidations() to accumulate the invalidation
+ * messages to the **invals_out.
+ */
+static void
+ReorderBufferAccumulateInvalidations(SharedInvalidationMessage **invals_out,
+ uint32 *ninvals_out,
+ SharedInvalidationMessage *msgs_new,
+ Size nmsgs_new)
+{
+ if (*ninvals_out == 0)
+ {
+ *ninvals_out = nmsgs_new;
+ *invals_out = (SharedInvalidationMessage *)
+ palloc(sizeof(SharedInvalidationMessage) * nmsgs_new);
+ memcpy(*invals_out, msgs_new, sizeof(SharedInvalidationMessage) * nmsgs_new);
+ }
+ else
+ {
+ /* Enlarge the array of inval messages */
+ *invals_out = (SharedInvalidationMessage *)
+ repalloc(*invals_out, sizeof(SharedInvalidationMessage) *
+ (*ninvals_out + nmsgs_new));
+ memcpy(*invals_out + *ninvals_out, msgs_new,
+ nmsgs_new * sizeof(SharedInvalidationMessage));
+ *ninvals_out += nmsgs_new;
+ }
+}
+
/*
* Accumulate the invalidations for executing them later.
*
@@ -3441,7 +3531,6 @@ ReorderBufferAddInvalidations(ReorderBuffer *rb, TransactionId xid,
{
ReorderBufferTXN *txn;
MemoryContext oldcontext;
- ReorderBufferChange *change;
txn = ReorderBufferTXNByXid(rb, xid, true, NULL, lsn, true);
@@ -3456,35 +3545,76 @@ ReorderBufferAddInvalidations(ReorderBuffer *rb, TransactionId xid,
Assert(nmsgs > 0);
- /* Accumulate invalidations. */
- if (txn->ninvalidations == 0)
- {
- txn->ninvalidations = nmsgs;
- txn->invalidations = (SharedInvalidationMessage *)
- palloc(sizeof(SharedInvalidationMessage) * nmsgs);
- memcpy(txn->invalidations, msgs,
- sizeof(SharedInvalidationMessage) * nmsgs);
- }
- else
+ ReorderBufferAccumulateInvalidations(&txn->invalidations,
+ &txn->ninvalidations,
+ msgs, nmsgs);
+
+ ReorderBufferQueueInvalidations(rb, xid, lsn, nmsgs, msgs);
+
+ MemoryContextSwitchTo(oldcontext);
+}
+
+/*
+ * Accumulate the invalidations distributed by other committed transactions
+ * for executing them later.
+ *
+ * This function is similar to ReorderBufferAddInvalidations() but stores
+ * the given inval messages to the txn->invalidations_distributed with the
+ * overflow check.
+ *
+ * This needs to be called by committed transactions to distribute their
+ * inval messages to in-progress transactions.
+ */
+void
+ReorderBufferAddDistributedInvalidations(ReorderBuffer *rb, TransactionId xid,
+ XLogRecPtr lsn, Size nmsgs,
+ SharedInvalidationMessage *msgs)
+{
+ ReorderBufferTXN *txn;
+ MemoryContext oldcontext;
+
+ txn = ReorderBufferTXNByXid(rb, xid, true, NULL, lsn, true);
+
+ oldcontext = MemoryContextSwitchTo(rb->context);
+
+ /*
+ * Collect all the invalidations under the top transaction, if available,
+ * so that we can execute them all together. See comments
+ * ReorderBufferAddInvalidations.
+ */
+ txn = rbtxn_get_toptxn(txn);
+
+ Assert(nmsgs > 0);
+
+ if (!rbtxn_distr_inval_overflowed(txn))
{
- txn->invalidations = (SharedInvalidationMessage *)
- repalloc(txn->invalidations, sizeof(SharedInvalidationMessage) *
- (txn->ninvalidations + nmsgs));
+ /*
+ * Check the transaction has enough space for storing distributed
+ * invalidation messages.
+ */
+ if (txn->ninvalidations_distributed + nmsgs >= MAX_DISTR_INVAL_MSG_PER_TXN)
+ {
+ /*
+ * Mark the invalidation message as overflowed and free up the
+ * messages accumulated so far.
+ */
+ txn->txn_flags |= RBTXN_DISTR_INVAL_OVERFLOWED;
- memcpy(txn->invalidations + txn->ninvalidations, msgs,
- nmsgs * sizeof(SharedInvalidationMessage));
- txn->ninvalidations += nmsgs;
+ if (txn->invalidations_distributed)
+ {
+ pfree(txn->invalidations_distributed);
+ txn->invalidations_distributed = NULL;
+ txn->ninvalidations_distributed = 0;
+ }
+ }
+ else
+ ReorderBufferAccumulateInvalidations(&txn->invalidations_distributed,
+ &txn->ninvalidations_distributed,
+ msgs, nmsgs);
}
- change = ReorderBufferAllocChange(rb);
- change->action = REORDER_BUFFER_CHANGE_INVALIDATION;
- change->data.inval.ninvalidations = nmsgs;
- change->data.inval.invalidations = (SharedInvalidationMessage *)
- palloc(sizeof(SharedInvalidationMessage) * nmsgs);
- memcpy(change->data.inval.invalidations, msgs,
- sizeof(SharedInvalidationMessage) * nmsgs);
-
- ReorderBufferQueueChange(rb, xid, lsn, change, false);
+ /* Queue the invalidation messages into the transaction */
+ ReorderBufferQueueInvalidations(rb, xid, lsn, nmsgs, msgs);
MemoryContextSwitchTo(oldcontext);
}
diff --git a/src/backend/replication/logical/snapbuild.c b/src/backend/replication/logical/snapbuild.c
index 0d7bddbe4ed..adf18c397db 100644
--- a/src/backend/replication/logical/snapbuild.c
+++ b/src/backend/replication/logical/snapbuild.c
@@ -794,6 +794,13 @@ SnapBuildDistributeSnapshotAndInval(SnapBuild *builder, XLogRecPtr lsn, Transact
* contents built by the current transaction even after its decoding,
* which should have been invalidated due to concurrent catalog
* changing transaction.
+ *
+ * Distribute only the invalidation messages generated by the current
+ * committed transaction. Invalidation messages received from other
+ * transactions would have already been propagated to the relevant
+ * in-progress transactions. This transaction would have processed
+ * those invalidations, ensuring that subsequent transactions observe
+ * a consistent cache state.
*/
if (txn->xid != xid)
{
@@ -807,8 +814,9 @@ SnapBuildDistributeSnapshotAndInval(SnapBuild *builder, XLogRecPtr lsn, Transact
{
Assert(msgs != NULL);
- ReorderBufferAddInvalidations(builder->reorder, txn->xid, lsn,
- ninvalidations, msgs);
+ ReorderBufferAddDistributedInvalidations(builder->reorder,
+ txn->xid, lsn,
+ ninvalidations, msgs);
}
}
}
diff --git a/src/include/replication/reorderbuffer.h b/src/include/replication/reorderbuffer.h
index 24e88c409ba..fa0745552f8 100644
--- a/src/include/replication/reorderbuffer.h
+++ b/src/include/replication/reorderbuffer.h
@@ -176,6 +176,7 @@ typedef struct ReorderBufferChange
#define RBTXN_SENT_PREPARE 0x0200
#define RBTXN_IS_COMMITTED 0x0400
#define RBTXN_IS_ABORTED 0x0800
+#define RBTXN_DISTR_INVAL_OVERFLOWED 0x1000
#define RBTXN_PREPARE_STATUS_MASK (RBTXN_IS_PREPARED | RBTXN_SKIPPED_PREPARE | RBTXN_SENT_PREPARE)
@@ -265,6 +266,12 @@ typedef struct ReorderBufferChange
((txn)->txn_flags & RBTXN_SKIPPED_PREPARE) != 0 \
)
+/* Is the array of distributed inval messages overflowed? */
+#define rbtxn_distr_inval_overflowed(txn) \
+( \
+ ((txn)->txn_flags & RBTXN_DISTR_INVAL_OVERFLOWED) != 0 \
+)
+
/* Is this a top-level transaction? */
#define rbtxn_is_toptxn(txn) \
( \
@@ -422,6 +429,12 @@ typedef struct ReorderBufferTXN
uint32 ninvalidations;
SharedInvalidationMessage *invalidations;
+ /*
+ * Stores cache invalidation messages distributed by other transactions.
+ */
+ uint32 ninvalidations_distributed;
+ SharedInvalidationMessage *invalidations_distributed;
+
/* ---
* Position in one of two lists:
* * list of subtransactions if we are *known* to be subxact
@@ -738,6 +751,9 @@ extern void ReorderBufferAddNewTupleCids(ReorderBuffer *rb, TransactionId xid,
CommandId cmin, CommandId cmax, CommandId combocid);
extern void ReorderBufferAddInvalidations(ReorderBuffer *rb, TransactionId xid, XLogRecPtr lsn,
Size nmsgs, SharedInvalidationMessage *msgs);
+extern void ReorderBufferAddDistributedInvalidations(ReorderBuffer *rb, TransactionId xid,
+ XLogRecPtr lsn, Size nmsgs,
+ SharedInvalidationMessage *msgs);
extern void ReorderBufferImmediateInvalidation(ReorderBuffer *rb, uint32 ninvalidations,
SharedInvalidationMessage *invalidations);
extern void ReorderBufferProcessXid(ReorderBuffer *rb, TransactionId xid, XLogRecPtr lsn);
--
2.43.0
^ permalink raw reply [nested|flat] 94+ messages in thread
* Re: Logical replication 'invalid memory alloc request size 1585837200' after upgrading to 17.5
@ 2025-06-13 23:52 Masahiko Sawada <sawada.mshk@gmail.com>
parent: vignesh C <vignesh21@gmail.com>
0 siblings, 1 reply; 94+ messages in thread
From: Masahiko Sawada @ 2025-06-13 23:52 UTC (permalink / raw)
To: vignesh C <vignesh21@gmail.com>; +Cc: Amit Kapila <amit.kapila16@gmail.com>; Hayato Kuroda (Fujitsu) <kuroda.hayato@fujitsu.com>; Duncan Sands <duncan.sands@deepbluecap.com>; pgsql-bugs@lists.postgresql.org <pgsql-bugs@lists.postgresql.org>
On Wed, Jun 11, 2025 at 11:47 PM vignesh C <vignesh21@gmail.com> wrote:
>
> On Thu, 12 Jun 2025 at 03:33, Masahiko Sawada <sawada.mshk@gmail.com> wrote:
> >
> > On Tue, Jun 10, 2025 at 12:19 PM Masahiko Sawada <sawada.mshk@gmail.com> wrote:
> > >
> > > On Sun, Jun 8, 2025 at 6:43 AM vignesh C <vignesh21@gmail.com> wrote:
> > > >
> > > > On Sat, 7 Jun 2025 at 12:47, vignesh C <vignesh21@gmail.com> wrote:
> > > > >
> > > > > On Fri, 6 Jun 2025 at 22:51, Masahiko Sawada <sawada.mshk@gmail.com> wrote:
> > > > > >
> > > > > > On Thu, Jun 5, 2025 at 8:43 PM Amit Kapila <amit.kapila16@gmail.com> wrote:
> > > > > > >
> > > > > > > On Thu, Jun 5, 2025 at 11:14 PM Masahiko Sawada <sawada.mshk@gmail.com> wrote:
> > > > > > > >
> > > > > > > > On Wed, Jun 4, 2025 at 11:36 PM Hayato Kuroda (Fujitsu)
> > > > > > > > <kuroda.hayato@fujitsu.com> wrote:
> > > > > > > > >
> > > > > > > > > Dear Sawada-san,
> > > > > > > > >
> > > > > > > > > > Alternative idea is to lower the constant value when using an
> > > > > > > > > > assertion build. That way, we don't need to rely on injection points
> > > > > > > > > > being enabled.
> > > > > > > > >
> > > > > > > > > Hmm, possible but I prefer current one. Two concerns:
> > > > > > > > >
> > > > > > > > > 1.
> > > > > > > > > USE_ASSERT_CHECKING has not been used to change the value yet. The main usage is
> > > > > > > > > to call debug functions in debug build.
> > > > > > > >
> > > > > > > > I think we have a similar precedent such as MT_NRELS_HASH to improve
> > > > > > > > the test coverages.
> > > > > > > >
> > > > > > > > > 2.
> > > > > > > > > If we add tests which is usable only for debug build, it must be run only when it
> > > > > > > > > is enabled. IIUC such test does not exist yet.
> > > > > > > >
> > > > > > > > I think we need to test cases not to check if we reach a specific code
> > > > > > > > point but to check if we can get the correct results even if we've
> > > > > > > > executed various code paths. As for this bug, it is better to check
> > > > > > > > that it works properly in a variety of cases. That way, we can check
> > > > > > > > overflow cases and non-overflow cases also in test cases added in the
> > > > > > > > future, improving the test coverage more.
> > > > > > > >
> > > > > > >
> > > > > > > This makes sense, but we should see whether some existing tests cover
> > > > > > > this code path after lowering the limit in the overflow code path. One
> > > > > > > minor point to consider is that at the time, the value MT_NRELS_HASH
> > > > > > > was used to cover cases in a debug build, but we didn't have the
> > > > > > > injection_point framework.
> > > > > >
> > > > > > True.
> > > > > >
> > > > > > After thinking about it more, perhaps my proposal would not be a good
> > > > > > idea for this case. I think that the cases where we selectively
> > > > > > invalidate caches is more complex and error-prone than the cases where
> > > > > > we invalidate a complete cache. If we invalidated all caches after
> > > > > > decoding each transaction, we wouldn't have had the original data-loss
> > > > > > issue. Having a lower MAX_DISTR_INVAL_MSG_PER_TXN value when using an
> > > > > > assertio build means that we're going to test the cases using a
> > > > > > simpler invalidation mechanism while productions systems, which has a
> > > > > > higher MAX_DISTR_INVAL_MSG_PER_TXN value, would end up executing
> > > > > > complex cases, which is not great. What do you think?
> > > > > >
> > > > > > BTW, as for a new test case, it might be worth having a case I
> > > > > > mentioned before[1]:
> > > > > >
> > > > > > 1) S1: CREATE TABLE d (data text not null);
> > > > > > 2) S1: INSERT INTO d VALUES ('d1');
> > > > > > 3) S2: BEGIN; INSERT INTO d VALUES ('d2');
> > > > > > 4) S3: BEGIN; INSERT INTO d VALUES ('d3');
> > > > > > 5) S1: ALTER PUBLICATION pb ADD TABLE d;
> > > > > > 6) S2: INSERT INTO d VALUES ('d4');
> > > > > > 7) S2: COMMIT;
> > > > > > 8) S3: COMMIT;
> > > > > > 9) S2: INSERT INTO d VALUES('d5');
> > > > > > 10) S1: INSERT INTO d VALUES ('d6');
> > > > > >
> > > > > > With this case, we can test if we need to execute the distributed
> > > > > > invalidations as well in the non-error path in
> > > > > > ReorderBufferProcessTXN().
> > > > >
> > > > > The attached v13 version patch has the change to include this test case.
> > > >
> > > > Attached are the patches, including those required for the back branches.
> > >
> > > Thank you for updating the patches! I'll review them and share comments if any.
> >
> > Thank you for updating the patch. I have one comment on the newly added test:
> >
> > +session "s3"
> > +step "s3i1" { INSERT INTO tbl1 (val1, val2) VALUES (1, 1);}
> > +step "s3a" { ALTER PUBLICATION pub ADD TABLE tbl1; }
> > +step "s3i2" { INSERT INTO tbl1 (val1, val2) VALUES (6, 6); }
> > +step "s3_get_binary_changes" { SELECT count(data) FROM
> > pg_logical_slot_get_binary_changes('isolation_slot', NULL, NULL,
> > 'proto_version', '4', 'publication_names', 'pub') WHERE get_byte(data,
> > 0) = 73; }
> > +
> > +session "s4"
> > +step "s4b" { BEGIN; }
> > +step "s4i1" { INSERT INTO tbl1 (val1, val2) VALUES (2, 2);}
> > +step "s4i2" { INSERT INTO tbl1 (val1, val2) VALUES (4, 4); }
> > +step "s4c" { COMMIT; }
> > +step "s4i3" { INSERT INTO tbl1 (val1, val2) VALUES (5, 5); }
> > +
> > +session "s5"
> > +step "s5b" { BEGIN; }
> > +step "s5i1" { INSERT INTO tbl1 (val1, val2) VALUES (3, 3); }
> > +step "s5c" { COMMIT; }
> >
> > I think we don't necessarily need to add sessions "s4" and "s5". Let's
> > reuse "s1" and "s2" instead of adding them. I've attached a patch to
> > change that.
>
> Thanks, this is better.
> In the case of PG13 I have slightly changed the test to do the insert
> after the commit, because the queuing of invalidation messages into
> reorder buffer queue is not supported in PG13. This limitation is due
> to the absence of support for REORDER_BUFFER_CHANGE_INVALIDATION which
> is already present in >= PG14
> The attached v14 version patch has the changes for the same.
Hmm, but the modified test is essentially the same as what we already
have in invalidation_distribution.spec. I think that it's good to have
the same test case in all branches even though we get a different
result in v13 as we currently don't have the test to check such
differences.
I've attached the updated patches accordingly and the commit messages.
I'm going to push them to all supported branches early next week,
barring any objections and further review comments.
Regards,
--
Masahiko Sawada
Amazon Web Services: https://aws.amazon.com
Attachments:
[application/octet-stream] REL15_v15-0001-Fix-re-distributing-previously-distributed-inval.patch (19.1K, ../../CAD21AoCHqKXVhUZbxPKBve-uEHJWKxV9SVC5deiB5VUONTH=6w@mail.gmail.com/2-REL15_v15-0001-Fix-re-distributing-previously-distributed-inval.patch)
download | inline diff:
From a6a84af588365673dd64a93e437c33ae5a2df2c2 Mon Sep 17 00:00:00 2001
From: Vignesh C <vignesh21@gmail.com>
Date: Thu, 12 Jun 2025 10:23:05 +0530
Subject: [PATCH v15] Fix re-distributing previously distributed invalidation
messages during logical decoding.
Commit 4909b38af0 introduced logic to distribute invalidation messages
from catalog-modifying transactions to all concurrent in-progress
transactions. However, since each transaction distributes not only its
original invalidation messages but also previously distributed
messages to other transactions, this leads to an exponential increase
in allocation request size for invalidation messages,
ultimatelycausing memory allocation failure.
This commit fixes this issue by tracking distributed invalidation
messages separately per decoded transaction and not redistributing
these messages to other in-progress transactions. The maximum size of
distributed invalidation messages that one transaction can store is
limited to MAX_DISTR_INVAL_MSG_PER_TXN (8MB). Once the size of the
distributed invalidation messages exceeds this threshold, we
invalidate all caches in locations where distributed invalidation
messages need to be executed.
Back-patch to all supported versions where we introduced the fix by
commit 4909b38af0.
Note that this commit adds two new fields to ReorderBufferTXN to store
the distributed transactions. This change breaks ABI compatibility in
back branches, affecting third-party extensions that depend on the
size of the ReorderBufferTXN struct, though this scenario seems
unlikely.
Additionally, it adds a new flag to the txn_flags field of
ReorderBufferTXN to indicate distributed invalidation message
overflow. This should not affect existing implementations, as it is
unlikely that third-party extensions use unused bits in the txn_flags
field.
Bug: #18938 #18942
Author: vignesh C <vignesh21@gmail.com>
Reported-by: Duncan Sands <duncan.sands@deepbluecap.com>
Reported-by: John Hutchins <john.hutchins@wicourts.gov>
Reported-by: Laurence Parry <greenreaper@hotmail.com>
Reported-by: Max Madden <maxmmadden@gmail.com>
Reported-by: Braulio Fdo Gonzalez <brauliofg@gmail.com>
Reviewed-by: Masahiko Sawada <sawada.mshk@gmail.com>
Reviewed-by: Amit Kapila <amit.kapila16@gmail.com>
Reviewed-by: Hayato Kuroda <kuroda.hayato@fujitsu.com>
Discussion: https://postgr.es/m/680bdaf6-f7d1-4536-b580-05c2760c67c6@deepbluecap.com
Discussion: https://postgr.es/m/18942-0ab1e5ae156613ad@postgresql.org
Discussion: https://postgr.es/m/18938-57c9a1c463b68ce0@postgresql.org
Discussion: https://postgr.es/m/CAD1FGCT2sYrP_70RTuo56QTizyc+J3wJdtn2gtO3VttQFpdMZg@mail.gmail.com
Discussion: https://postgr.es/m/CANO2=B=2BT1hSYCE=nuuTnVTnjidMg0+-FfnRnqM6kd23qoygg@mail.gmail.com
Backpatch-through: 13
---
.../expected/invalidation_distribution.out | 23 +-
.../specs/invalidation_distribution.spec | 11 +
.../replication/logical/reorderbuffer.c | 197 +++++++++++++++---
src/backend/replication/logical/snapbuild.c | 12 +-
src/include/replication/reorderbuffer.h | 32 ++-
5 files changed, 231 insertions(+), 44 deletions(-)
diff --git a/contrib/test_decoding/expected/invalidation_distribution.out b/contrib/test_decoding/expected/invalidation_distribution.out
index 24190ebe570..49c1f60cc4f 100644
--- a/contrib/test_decoding/expected/invalidation_distribution.out
+++ b/contrib/test_decoding/expected/invalidation_distribution.out
@@ -1,4 +1,4 @@
-Parsed test spec with 2 sessions
+Parsed test spec with 3 sessions
starting permutation: s1_insert_tbl1 s1_begin s1_insert_tbl1 s2_alter_pub_add_tbl s1_commit s1_insert_tbl1 s2_get_binary_changes
step s1_insert_tbl1: INSERT INTO tbl1 (val1, val2) VALUES (1, 1);
@@ -18,3 +18,24 @@ count
stop
(1 row)
+
+starting permutation: s1_begin s1_insert_tbl1 s3_begin s3_insert_tbl1 s2_alter_pub_add_tbl s1_insert_tbl1 s1_commit s3_commit s2_get_binary_changes
+step s1_begin: BEGIN;
+step s1_insert_tbl1: INSERT INTO tbl1 (val1, val2) VALUES (1, 1);
+step s3_begin: BEGIN;
+step s3_insert_tbl1: INSERT INTO tbl1 (val1, val2) VALUES (2, 2);
+step s2_alter_pub_add_tbl: ALTER PUBLICATION pub ADD TABLE tbl1;
+step s1_insert_tbl1: INSERT INTO tbl1 (val1, val2) VALUES (1, 1);
+step s1_commit: COMMIT;
+step s3_commit: COMMIT;
+step s2_get_binary_changes: SELECT count(data) FROM pg_logical_slot_get_binary_changes('isolation_slot', NULL, NULL, 'proto_version', '3', 'publication_names', 'pub') WHERE get_byte(data, 0) = 73;
+count
+-----
+ 1
+(1 row)
+
+?column?
+--------
+stop
+(1 row)
+
diff --git a/contrib/test_decoding/specs/invalidation_distribution.spec b/contrib/test_decoding/specs/invalidation_distribution.spec
index f63aba3ce96..c85f452938c 100644
--- a/contrib/test_decoding/specs/invalidation_distribution.spec
+++ b/contrib/test_decoding/specs/invalidation_distribution.spec
@@ -28,5 +28,16 @@ setup { SET synchronous_commit=on; }
step "s2_alter_pub_add_tbl" { ALTER PUBLICATION pub ADD TABLE tbl1; }
step "s2_get_binary_changes" { SELECT count(data) FROM pg_logical_slot_get_binary_changes('isolation_slot', NULL, NULL, 'proto_version', '3', 'publication_names', 'pub') WHERE get_byte(data, 0) = 73; }
+session "s3"
+setup { SET synchronous_commit=on; }
+step "s3_begin" { BEGIN; }
+step "s3_insert_tbl1" { INSERT INTO tbl1 (val1, val2) VALUES (2, 2); }
+step "s3_commit" { COMMIT; }
+
# Expect to get one insert change. LOGICAL_REP_MSG_INSERT = 'I'
permutation "s1_insert_tbl1" "s1_begin" "s1_insert_tbl1" "s2_alter_pub_add_tbl" "s1_commit" "s1_insert_tbl1" "s2_get_binary_changes"
+
+# Expect to get one insert change with LOGICAL_REP_MSG_INSERT = 'I' from
+# the second "s1_insert_tbl1" executed after adding the table tbl1 to the
+# publication in "s2_alter_pub_add_tbl".
+permutation "s1_begin" "s1_insert_tbl1" "s3_begin" "s3_insert_tbl1" "s2_alter_pub_add_tbl" "s1_insert_tbl1" "s1_commit" "s3_commit" "s2_get_binary_changes"
diff --git a/src/backend/replication/logical/reorderbuffer.c b/src/backend/replication/logical/reorderbuffer.c
index 835180e12f6..59efa73930f 100644
--- a/src/backend/replication/logical/reorderbuffer.c
+++ b/src/backend/replication/logical/reorderbuffer.c
@@ -103,12 +103,24 @@
#include "storage/sinval.h"
#include "utils/builtins.h"
#include "utils/combocid.h"
+#include "utils/inval.h"
#include "utils/memdebug.h"
#include "utils/memutils.h"
#include "utils/rel.h"
#include "utils/relfilenodemap.h"
+/*
+ * Each transaction has an 8MB limit for invalidation messages distributed from
+ * other transactions. This limit is set considering scenarios with many
+ * concurrent logical decoding operations. When the distributed invalidation
+ * messages reach this threshold, the transaction is marked as
+ * RBTXN_DISTR_INVAL_OVERFLOWED to invalidate the complete cache as we have lost
+ * some inval messages and hence don't know what needs to be invalidated.
+ */
+#define MAX_DISTR_INVAL_MSG_PER_TXN \
+ ((8 * 1024 * 1024) / sizeof(SharedInvalidationMessage))
+
/* entry for a hash table we use to map from xid to our transaction state */
typedef struct ReorderBufferTXNByIdEnt
{
@@ -457,6 +469,12 @@ ReorderBufferReturnTXN(ReorderBuffer *rb, ReorderBufferTXN *txn)
txn->invalidations = NULL;
}
+ if (txn->invalidations_distributed)
+ {
+ pfree(txn->invalidations_distributed);
+ txn->invalidations_distributed = NULL;
+ }
+
/* Reset the toast hash */
ReorderBufferToastReset(rb, txn);
@@ -2503,7 +2521,17 @@ ReorderBufferProcessTXN(ReorderBuffer *rb, ReorderBufferTXN *txn,
AbortCurrentTransaction();
/* make sure there's no cache pollution */
- ReorderBufferExecuteInvalidations(txn->ninvalidations, txn->invalidations);
+ if (rbtxn_distr_inval_overflowed(txn))
+ {
+ Assert(txn->ninvalidations_distributed == 0);
+ InvalidateSystemCaches();
+ }
+ else
+ {
+ ReorderBufferExecuteInvalidations(txn->ninvalidations, txn->invalidations);
+ ReorderBufferExecuteInvalidations(txn->ninvalidations_distributed,
+ txn->invalidations_distributed);
+ }
if (using_subtxn)
RollbackAndReleaseCurrentSubTransaction();
@@ -2549,8 +2577,17 @@ ReorderBufferProcessTXN(ReorderBuffer *rb, ReorderBufferTXN *txn,
AbortCurrentTransaction();
/* make sure there's no cache pollution */
- ReorderBufferExecuteInvalidations(txn->ninvalidations,
- txn->invalidations);
+ if (rbtxn_distr_inval_overflowed(txn))
+ {
+ Assert(txn->ninvalidations_distributed == 0);
+ InvalidateSystemCaches();
+ }
+ else
+ {
+ ReorderBufferExecuteInvalidations(txn->ninvalidations, txn->invalidations);
+ ReorderBufferExecuteInvalidations(txn->ninvalidations_distributed,
+ txn->invalidations_distributed);
+ }
if (using_subtxn)
RollbackAndReleaseCurrentSubTransaction();
@@ -2877,7 +2914,8 @@ ReorderBufferAbort(ReorderBuffer *rb, TransactionId xid, XLogRecPtr lsn)
* We might have decoded changes for this transaction that could load
* the cache as per the current transaction's view (consider DDL's
* happened in this transaction). We don't want the decoding of future
- * transactions to use those cache entries so execute invalidations.
+ * transactions to use those cache entries so execute only the inval
+ * messages in this transaction.
*/
if (txn->ninvalidations > 0)
ReorderBufferImmediateInvalidation(rb, txn->ninvalidations,
@@ -2965,9 +3003,10 @@ ReorderBufferForget(ReorderBuffer *rb, TransactionId xid, XLogRecPtr lsn)
txn->final_lsn = lsn;
/*
- * Process cache invalidation messages if there are any. Even if we're not
- * interested in the transaction's contents, it could have manipulated the
- * catalog and we need to update the caches according to that.
+ * Process only cache invalidation messages in this transaction if there
+ * are any. Even if we're not interested in the transaction's contents, it
+ * could have manipulated the catalog and we need to update the caches
+ * according to that.
*/
if (txn->base_snapshot != NULL && txn->ninvalidations > 0)
ReorderBufferImmediateInvalidation(rb, txn->ninvalidations,
@@ -3222,6 +3261,57 @@ ReorderBufferAddNewTupleCids(ReorderBuffer *rb, TransactionId xid,
txn->ntuplecids++;
}
+/*
+ * Add new invalidation messages to the reorder buffer queue.
+ */
+static void
+ReorderBufferQueueInvalidations(ReorderBuffer *rb, TransactionId xid,
+ XLogRecPtr lsn, Size nmsgs,
+ SharedInvalidationMessage *msgs)
+{
+ ReorderBufferChange *change;
+
+ change = ReorderBufferGetChange(rb);
+ change->action = REORDER_BUFFER_CHANGE_INVALIDATION;
+ change->data.inval.ninvalidations = nmsgs;
+ change->data.inval.invalidations = (SharedInvalidationMessage *)
+ palloc(sizeof(SharedInvalidationMessage) * nmsgs);
+ memcpy(change->data.inval.invalidations, msgs,
+ sizeof(SharedInvalidationMessage) * nmsgs);
+
+ ReorderBufferQueueChange(rb, xid, lsn, change, false);
+}
+
+/*
+ * A helper function for ReorderBufferAddInvalidations() and
+ * ReorderBufferAddDistributedInvalidations() to accumulate the invalidation
+ * messages to the **invals_out.
+ */
+static void
+ReorderBufferAccumulateInvalidations(SharedInvalidationMessage **invals_out,
+ uint32 *ninvals_out,
+ SharedInvalidationMessage *msgs_new,
+ Size nmsgs_new)
+{
+ if (*ninvals_out == 0)
+ {
+ *ninvals_out = nmsgs_new;
+ *invals_out = (SharedInvalidationMessage *)
+ palloc(sizeof(SharedInvalidationMessage) * nmsgs_new);
+ memcpy(*invals_out, msgs_new, sizeof(SharedInvalidationMessage) * nmsgs_new);
+ }
+ else
+ {
+ /* Enlarge the array of inval messages */
+ *invals_out = (SharedInvalidationMessage *)
+ repalloc(*invals_out, sizeof(SharedInvalidationMessage) *
+ (*ninvals_out + nmsgs_new));
+ memcpy(*invals_out + *ninvals_out, msgs_new,
+ nmsgs_new * sizeof(SharedInvalidationMessage));
+ *ninvals_out += nmsgs_new;
+ }
+}
+
/*
* Accumulate the invalidations for executing them later.
*
@@ -3242,7 +3332,6 @@ ReorderBufferAddInvalidations(ReorderBuffer *rb, TransactionId xid,
{
ReorderBufferTXN *txn;
MemoryContext oldcontext;
- ReorderBufferChange *change;
txn = ReorderBufferTXNByXid(rb, xid, true, NULL, lsn, true);
@@ -3258,35 +3347,77 @@ ReorderBufferAddInvalidations(ReorderBuffer *rb, TransactionId xid,
Assert(nmsgs > 0);
- /* Accumulate invalidations. */
- if (txn->ninvalidations == 0)
- {
- txn->ninvalidations = nmsgs;
- txn->invalidations = (SharedInvalidationMessage *)
- palloc(sizeof(SharedInvalidationMessage) * nmsgs);
- memcpy(txn->invalidations, msgs,
- sizeof(SharedInvalidationMessage) * nmsgs);
- }
- else
+ ReorderBufferAccumulateInvalidations(&txn->invalidations,
+ &txn->ninvalidations,
+ msgs, nmsgs);
+
+ ReorderBufferQueueInvalidations(rb, xid, lsn, nmsgs, msgs);
+
+ MemoryContextSwitchTo(oldcontext);
+}
+
+/*
+ * Accumulate the invalidations distributed by other committed transactions
+ * for executing them later.
+ *
+ * This function is similar to ReorderBufferAddInvalidations() but stores
+ * the given inval messages to the txn->invalidations_distributed with the
+ * overflow check.
+ *
+ * This needs to be called by committed transactions to distribute their
+ * inval messages to in-progress transactions.
+ */
+void
+ReorderBufferAddDistributedInvalidations(ReorderBuffer *rb, TransactionId xid,
+ XLogRecPtr lsn, Size nmsgs,
+ SharedInvalidationMessage *msgs)
+{
+ ReorderBufferTXN *txn;
+ MemoryContext oldcontext;
+
+ txn = ReorderBufferTXNByXid(rb, xid, true, NULL, lsn, true);
+
+ oldcontext = MemoryContextSwitchTo(rb->context);
+
+ /*
+ * Collect all the invalidations under the top transaction, if available,
+ * so that we can execute them all together. See comments
+ * ReorderBufferAddInvalidations.
+ */
+ if (txn->toptxn)
+ txn = txn->toptxn;
+
+ Assert(nmsgs > 0);
+
+ if (!rbtxn_distr_inval_overflowed(txn))
{
- txn->invalidations = (SharedInvalidationMessage *)
- repalloc(txn->invalidations, sizeof(SharedInvalidationMessage) *
- (txn->ninvalidations + nmsgs));
+ /*
+ * Check the transaction has enough space for storing distributed
+ * invalidation messages.
+ */
+ if (txn->ninvalidations_distributed + nmsgs >= MAX_DISTR_INVAL_MSG_PER_TXN)
+ {
+ /*
+ * Mark the invalidation message as overflowed and free up the
+ * messages accumulated so far.
+ */
+ txn->txn_flags |= RBTXN_DISTR_INVAL_OVERFLOWED;
- memcpy(txn->invalidations + txn->ninvalidations, msgs,
- nmsgs * sizeof(SharedInvalidationMessage));
- txn->ninvalidations += nmsgs;
+ if (txn->invalidations_distributed)
+ {
+ pfree(txn->invalidations_distributed);
+ txn->invalidations_distributed = NULL;
+ txn->ninvalidations_distributed = 0;
+ }
+ }
+ else
+ ReorderBufferAccumulateInvalidations(&txn->invalidations_distributed,
+ &txn->ninvalidations_distributed,
+ msgs, nmsgs);
}
- change = ReorderBufferGetChange(rb);
- change->action = REORDER_BUFFER_CHANGE_INVALIDATION;
- change->data.inval.ninvalidations = nmsgs;
- change->data.inval.invalidations = (SharedInvalidationMessage *)
- palloc(sizeof(SharedInvalidationMessage) * nmsgs);
- memcpy(change->data.inval.invalidations, msgs,
- sizeof(SharedInvalidationMessage) * nmsgs);
-
- ReorderBufferQueueChange(rb, xid, lsn, change, false);
+ /* Queue the invalidation messages into the transaction */
+ ReorderBufferQueueInvalidations(rb, xid, lsn, nmsgs, msgs);
MemoryContextSwitchTo(oldcontext);
}
diff --git a/src/backend/replication/logical/snapbuild.c b/src/backend/replication/logical/snapbuild.c
index 2ca8e6e9aff..2dd4782f086 100644
--- a/src/backend/replication/logical/snapbuild.c
+++ b/src/backend/replication/logical/snapbuild.c
@@ -925,6 +925,13 @@ SnapBuildDistributeSnapshotAndInval(SnapBuild *builder, XLogRecPtr lsn, Transact
* contents built by the current transaction even after its decoding,
* which should have been invalidated due to concurrent catalog
* changing transaction.
+ *
+ * Distribute only the invalidation messages generated by the current
+ * committed transaction. Invalidation messages received from other
+ * transactions would have already been propagated to the relevant
+ * in-progress transactions. This transaction would have processed
+ * those invalidations, ensuring that subsequent transactions observe
+ * a consistent cache state.
*/
if (txn->xid != xid)
{
@@ -938,8 +945,9 @@ SnapBuildDistributeSnapshotAndInval(SnapBuild *builder, XLogRecPtr lsn, Transact
{
Assert(msgs != NULL);
- ReorderBufferAddInvalidations(builder->reorder, txn->xid, lsn,
- ninvalidations, msgs);
+ ReorderBufferAddDistributedInvalidations(builder->reorder,
+ txn->xid, lsn,
+ ninvalidations, msgs);
}
}
}
diff --git a/src/include/replication/reorderbuffer.h b/src/include/replication/reorderbuffer.h
index 402bb7a2728..af154587618 100644
--- a/src/include/replication/reorderbuffer.h
+++ b/src/include/replication/reorderbuffer.h
@@ -168,14 +168,15 @@ typedef struct ReorderBufferChange
} ReorderBufferChange;
/* ReorderBufferTXN txn_flags */
-#define RBTXN_HAS_CATALOG_CHANGES 0x0001
-#define RBTXN_IS_SUBXACT 0x0002
-#define RBTXN_IS_SERIALIZED 0x0004
-#define RBTXN_IS_SERIALIZED_CLEAR 0x0008
-#define RBTXN_IS_STREAMED 0x0010
-#define RBTXN_HAS_PARTIAL_CHANGE 0x0020
-#define RBTXN_PREPARE 0x0040
-#define RBTXN_SKIPPED_PREPARE 0x0080
+#define RBTXN_HAS_CATALOG_CHANGES 0x0001
+#define RBTXN_IS_SUBXACT 0x0002
+#define RBTXN_IS_SERIALIZED 0x0004
+#define RBTXN_IS_SERIALIZED_CLEAR 0x0008
+#define RBTXN_IS_STREAMED 0x0010
+#define RBTXN_HAS_PARTIAL_CHANGE 0x0020
+#define RBTXN_PREPARE 0x0040
+#define RBTXN_SKIPPED_PREPARE 0x0080
+#define RBTXN_DISTR_INVAL_OVERFLOWED 0x0100
/* Does the transaction have catalog changes? */
#define rbtxn_has_catalog_changes(txn) \
@@ -233,6 +234,12 @@ typedef struct ReorderBufferChange
((txn)->txn_flags & RBTXN_SKIPPED_PREPARE) != 0 \
)
+/* Is the array of distributed inval messages overflowed? */
+#define rbtxn_distr_inval_overflowed(txn) \
+( \
+ ((txn)->txn_flags & RBTXN_DISTR_INVAL_OVERFLOWED) != 0 \
+)
+
typedef struct ReorderBufferTXN
{
/* See above */
@@ -395,6 +402,12 @@ typedef struct ReorderBufferTXN
* Private data pointer of the output plugin.
*/
void *output_plugin_private;
+
+ /*
+ * Stores cache invalidation messages distributed by other transactions.
+ */
+ uint32 ninvalidations_distributed;
+ SharedInvalidationMessage *invalidations_distributed;
} ReorderBufferTXN;
/* so we can define the callbacks used inside struct ReorderBuffer itself */
@@ -661,6 +674,9 @@ extern void ReorderBufferAddNewTupleCids(ReorderBuffer *, TransactionId, XLogRec
CommandId cmin, CommandId cmax, CommandId combocid);
extern void ReorderBufferAddInvalidations(ReorderBuffer *, TransactionId, XLogRecPtr lsn,
Size nmsgs, SharedInvalidationMessage *msgs);
+extern void ReorderBufferAddDistributedInvalidations(ReorderBuffer *rb, TransactionId xid,
+ XLogRecPtr lsn, Size nmsgs,
+ SharedInvalidationMessage *msgs);
extern void ReorderBufferImmediateInvalidation(ReorderBuffer *, uint32 ninvalidations,
SharedInvalidationMessage *invalidations);
extern void ReorderBufferProcessXid(ReorderBuffer *, TransactionId xid, XLogRecPtr lsn);
--
2.43.5
[application/octet-stream] master_v15-0001-Fix-re-distributing-previously-distributed-inval.patch (18.4K, ../../CAD21AoCHqKXVhUZbxPKBve-uEHJWKxV9SVC5deiB5VUONTH=6w@mail.gmail.com/3-master_v15-0001-Fix-re-distributing-previously-distributed-inval.patch)
download | inline diff:
From 1cd2695085d01bb8de6f763671177ad108a53ef7 Mon Sep 17 00:00:00 2001
From: Masahiko Sawada <sawada.mshk@gmail.com>
Date: Fri, 13 Jun 2025 14:45:03 -0700
Subject: [PATCH v15] Fix re-distributing previously distributed invalidation
messages during logical decoding.
Commit 4909b38af0 introduced logic to distribute invalidation messages
from catalog-modifying transactions to all concurrent in-progress
transactions. However, since each transaction distributes not only its
original invalidation messages but also previously distributed
messages to other transactions, this leads to an exponential increase
in allocation request size for invalidation messages, ultimately
causing memory allocation failure.
This commit fixes this issue by tracking distributed invalidation
messages separately per decoded transaction and not redistributing
these messages to other in-progress transactions. The maximum size of
distributed invalidation messages that one transaction can store is
limited to MAX_DISTR_INVAL_MSG_PER_TXN (8MB). Once the size of the
distributed invalidation messages exceeds this threshold, we
invalidate all caches in locations where distributed invalidation
messages need to be executed.
Back-patch to all supported versions where we introduced the fix by
commit 4909b38af0.
Note that this commit adds two new fields to ReorderBufferTXN to store
the distributed transactions. This change breaks ABI compatibility in
back branches, affecting third-party extensions that depend on the
size of the ReorderBufferTXN struct, though this scenario seems
unlikely.
Additionally, it adds a new flag to the txn_flags field of
ReorderBufferTXN to indicate distributed invalidation message
overflow. This should not affect existing implementations, as it is
unlikely that third-party extensions use unused bits in the txn_flags
field.
Bug: #18938 #18942
Author: vignesh C <vignesh21@gmail.com>
Reported-by: Duncan Sands <duncan.sands@deepbluecap.com>
Reported-by: John Hutchins <john.hutchins@wicourts.gov>
Reported-by: Laurence Parry <greenreaper@hotmail.com>
Reported-by: Max Madden <maxmmadden@gmail.com>
Reported-by: Braulio Fdo Gonzalez <brauliofg@gmail.com>
Reviewed-by: Masahiko Sawada <sawada.mshk@gmail.com>
Reviewed-by: Amit Kapila <amit.kapila16@gmail.com>
Reviewed-by: Hayato Kuroda <kuroda.hayato@fujitsu.com>
Discussion: https://postgr.es/m/680bdaf6-f7d1-4536-b580-05c2760c67c6@deepbluecap.com
Discussion: https://postgr.es/m/18942-0ab1e5ae156613ad@postgresql.org
Discussion: https://postgr.es/m/18938-57c9a1c463b68ce0@postgresql.org
Discussion: https://postgr.es/m/CAD1FGCT2sYrP_70RTuo56QTizyc+J3wJdtn2gtO3VttQFpdMZg@mail.gmail.com
Discussion: https://postgr.es/m/CANO2=B=2BT1hSYCE=nuuTnVTnjidMg0+-FfnRnqM6kd23qoygg@mail.gmail.com
Backpatch-through: 13
---
.../expected/invalidation_distribution.out | 23 +-
.../specs/invalidation_distribution.spec | 11 +
.../replication/logical/reorderbuffer.c | 196 +++++++++++++++---
src/backend/replication/logical/snapbuild.c | 12 +-
src/include/replication/reorderbuffer.h | 16 ++
5 files changed, 222 insertions(+), 36 deletions(-)
diff --git a/contrib/test_decoding/expected/invalidation_distribution.out b/contrib/test_decoding/expected/invalidation_distribution.out
index ad0a944cbf3..ae53b1e61de 100644
--- a/contrib/test_decoding/expected/invalidation_distribution.out
+++ b/contrib/test_decoding/expected/invalidation_distribution.out
@@ -1,4 +1,4 @@
-Parsed test spec with 2 sessions
+Parsed test spec with 3 sessions
starting permutation: s1_insert_tbl1 s1_begin s1_insert_tbl1 s2_alter_pub_add_tbl s1_commit s1_insert_tbl1 s2_get_binary_changes
step s1_insert_tbl1: INSERT INTO tbl1 (val1, val2) VALUES (1, 1);
@@ -18,3 +18,24 @@ count
stop
(1 row)
+
+starting permutation: s1_begin s1_insert_tbl1 s3_begin s3_insert_tbl1 s2_alter_pub_add_tbl s1_insert_tbl1 s1_commit s3_commit s2_get_binary_changes
+step s1_begin: BEGIN;
+step s1_insert_tbl1: INSERT INTO tbl1 (val1, val2) VALUES (1, 1);
+step s3_begin: BEGIN;
+step s3_insert_tbl1: INSERT INTO tbl1 (val1, val2) VALUES (2, 2);
+step s2_alter_pub_add_tbl: ALTER PUBLICATION pub ADD TABLE tbl1;
+step s1_insert_tbl1: INSERT INTO tbl1 (val1, val2) VALUES (1, 1);
+step s1_commit: COMMIT;
+step s3_commit: COMMIT;
+step s2_get_binary_changes: SELECT count(data) FROM pg_logical_slot_get_binary_changes('isolation_slot', NULL, NULL, 'proto_version', '4', 'publication_names', 'pub') WHERE get_byte(data, 0) = 73;
+count
+-----
+ 1
+(1 row)
+
+?column?
+--------
+stop
+(1 row)
+
diff --git a/contrib/test_decoding/specs/invalidation_distribution.spec b/contrib/test_decoding/specs/invalidation_distribution.spec
index decbed627e3..67d41969ac1 100644
--- a/contrib/test_decoding/specs/invalidation_distribution.spec
+++ b/contrib/test_decoding/specs/invalidation_distribution.spec
@@ -28,5 +28,16 @@ setup { SET synchronous_commit=on; }
step "s2_alter_pub_add_tbl" { ALTER PUBLICATION pub ADD TABLE tbl1; }
step "s2_get_binary_changes" { SELECT count(data) FROM pg_logical_slot_get_binary_changes('isolation_slot', NULL, NULL, 'proto_version', '4', 'publication_names', 'pub') WHERE get_byte(data, 0) = 73; }
+session "s3"
+setup { SET synchronous_commit=on; }
+step "s3_begin" { BEGIN; }
+step "s3_insert_tbl1" { INSERT INTO tbl1 (val1, val2) VALUES (2, 2); }
+step "s3_commit" { COMMIT; }
+
# Expect to get one insert change. LOGICAL_REP_MSG_INSERT = 'I'
permutation "s1_insert_tbl1" "s1_begin" "s1_insert_tbl1" "s2_alter_pub_add_tbl" "s1_commit" "s1_insert_tbl1" "s2_get_binary_changes"
+
+# Expect to get one insert change with LOGICAL_REP_MSG_INSERT = 'I' from
+# the second "s1_insert_tbl1" executed after adding the table tbl1 to the
+# publication in "s2_alter_pub_add_tbl".
+permutation "s1_begin" "s1_insert_tbl1" "s3_begin" "s3_insert_tbl1" "s2_alter_pub_add_tbl" "s1_insert_tbl1" "s1_commit" "s3_commit" "s2_get_binary_changes"
diff --git a/src/backend/replication/logical/reorderbuffer.c b/src/backend/replication/logical/reorderbuffer.c
index 67655111875..c4299c76fb1 100644
--- a/src/backend/replication/logical/reorderbuffer.c
+++ b/src/backend/replication/logical/reorderbuffer.c
@@ -109,10 +109,22 @@
#include "storage/procarray.h"
#include "storage/sinval.h"
#include "utils/builtins.h"
+#include "utils/inval.h"
#include "utils/memutils.h"
#include "utils/rel.h"
#include "utils/relfilenumbermap.h"
+/*
+ * Each transaction has an 8MB limit for invalidation messages distributed from
+ * other transactions. This limit is set considering scenarios with many
+ * concurrent logical decoding operations. When the distributed invalidation
+ * messages reach this threshold, the transaction is marked as
+ * RBTXN_DISTR_INVAL_OVERFLOWED to invalidate the complete cache as we have lost
+ * some inval messages and hence don't know what needs to be invalidated.
+ */
+#define MAX_DISTR_INVAL_MSG_PER_TXN \
+ ((8 * 1024 * 1024) / sizeof(SharedInvalidationMessage))
+
/* entry for a hash table we use to map from xid to our transaction state */
typedef struct ReorderBufferTXNByIdEnt
{
@@ -472,6 +484,12 @@ ReorderBufferFreeTXN(ReorderBuffer *rb, ReorderBufferTXN *txn)
txn->invalidations = NULL;
}
+ if (txn->invalidations_distributed)
+ {
+ pfree(txn->invalidations_distributed);
+ txn->invalidations_distributed = NULL;
+ }
+
/* Reset the toast hash */
ReorderBufferToastReset(rb, txn);
@@ -2661,7 +2679,17 @@ ReorderBufferProcessTXN(ReorderBuffer *rb, ReorderBufferTXN *txn,
AbortCurrentTransaction();
/* make sure there's no cache pollution */
- ReorderBufferExecuteInvalidations(txn->ninvalidations, txn->invalidations);
+ if (rbtxn_distr_inval_overflowed(txn))
+ {
+ Assert(txn->ninvalidations_distributed == 0);
+ InvalidateSystemCaches();
+ }
+ else
+ {
+ ReorderBufferExecuteInvalidations(txn->ninvalidations, txn->invalidations);
+ ReorderBufferExecuteInvalidations(txn->ninvalidations_distributed,
+ txn->invalidations_distributed);
+ }
if (using_subtxn)
RollbackAndReleaseCurrentSubTransaction();
@@ -2710,8 +2738,17 @@ ReorderBufferProcessTXN(ReorderBuffer *rb, ReorderBufferTXN *txn,
AbortCurrentTransaction();
/* make sure there's no cache pollution */
- ReorderBufferExecuteInvalidations(txn->ninvalidations,
- txn->invalidations);
+ if (rbtxn_distr_inval_overflowed(txn))
+ {
+ Assert(txn->ninvalidations_distributed == 0);
+ InvalidateSystemCaches();
+ }
+ else
+ {
+ ReorderBufferExecuteInvalidations(txn->ninvalidations, txn->invalidations);
+ ReorderBufferExecuteInvalidations(txn->ninvalidations_distributed,
+ txn->invalidations_distributed);
+ }
if (using_subtxn)
RollbackAndReleaseCurrentSubTransaction();
@@ -3060,7 +3097,8 @@ ReorderBufferAbort(ReorderBuffer *rb, TransactionId xid, XLogRecPtr lsn,
* We might have decoded changes for this transaction that could load
* the cache as per the current transaction's view (consider DDL's
* happened in this transaction). We don't want the decoding of future
- * transactions to use those cache entries so execute invalidations.
+ * transactions to use those cache entries so execute only the inval
+ * messages in this transaction.
*/
if (txn->ninvalidations > 0)
ReorderBufferImmediateInvalidation(rb, txn->ninvalidations,
@@ -3147,9 +3185,10 @@ ReorderBufferForget(ReorderBuffer *rb, TransactionId xid, XLogRecPtr lsn)
txn->final_lsn = lsn;
/*
- * Process cache invalidation messages if there are any. Even if we're not
- * interested in the transaction's contents, it could have manipulated the
- * catalog and we need to update the caches according to that.
+ * Process only cache invalidation messages in this transaction if there
+ * are any. Even if we're not interested in the transaction's contents, it
+ * could have manipulated the catalog and we need to update the caches
+ * according to that.
*/
if (txn->base_snapshot != NULL && txn->ninvalidations > 0)
ReorderBufferImmediateInvalidation(rb, txn->ninvalidations,
@@ -3421,6 +3460,57 @@ ReorderBufferAddNewTupleCids(ReorderBuffer *rb, TransactionId xid,
txn->ntuplecids++;
}
+/*
+ * Add new invalidation messages to the reorder buffer queue.
+ */
+static void
+ReorderBufferQueueInvalidations(ReorderBuffer *rb, TransactionId xid,
+ XLogRecPtr lsn, Size nmsgs,
+ SharedInvalidationMessage *msgs)
+{
+ ReorderBufferChange *change;
+
+ change = ReorderBufferAllocChange(rb);
+ change->action = REORDER_BUFFER_CHANGE_INVALIDATION;
+ change->data.inval.ninvalidations = nmsgs;
+ change->data.inval.invalidations = (SharedInvalidationMessage *)
+ palloc(sizeof(SharedInvalidationMessage) * nmsgs);
+ memcpy(change->data.inval.invalidations, msgs,
+ sizeof(SharedInvalidationMessage) * nmsgs);
+
+ ReorderBufferQueueChange(rb, xid, lsn, change, false);
+}
+
+/*
+ * A helper function for ReorderBufferAddInvalidations() and
+ * ReorderBufferAddDistributedInvalidations() to accumulate the invalidation
+ * messages to the **invals_out.
+ */
+static void
+ReorderBufferAccumulateInvalidations(SharedInvalidationMessage **invals_out,
+ uint32 *ninvals_out,
+ SharedInvalidationMessage *msgs_new,
+ Size nmsgs_new)
+{
+ if (*ninvals_out == 0)
+ {
+ *ninvals_out = nmsgs_new;
+ *invals_out = (SharedInvalidationMessage *)
+ palloc(sizeof(SharedInvalidationMessage) * nmsgs_new);
+ memcpy(*invals_out, msgs_new, sizeof(SharedInvalidationMessage) * nmsgs_new);
+ }
+ else
+ {
+ /* Enlarge the array of inval messages */
+ *invals_out = (SharedInvalidationMessage *)
+ repalloc(*invals_out, sizeof(SharedInvalidationMessage) *
+ (*ninvals_out + nmsgs_new));
+ memcpy(*invals_out + *ninvals_out, msgs_new,
+ nmsgs_new * sizeof(SharedInvalidationMessage));
+ *ninvals_out += nmsgs_new;
+ }
+}
+
/*
* Accumulate the invalidations for executing them later.
*
@@ -3441,7 +3531,6 @@ ReorderBufferAddInvalidations(ReorderBuffer *rb, TransactionId xid,
{
ReorderBufferTXN *txn;
MemoryContext oldcontext;
- ReorderBufferChange *change;
txn = ReorderBufferTXNByXid(rb, xid, true, NULL, lsn, true);
@@ -3456,35 +3545,76 @@ ReorderBufferAddInvalidations(ReorderBuffer *rb, TransactionId xid,
Assert(nmsgs > 0);
- /* Accumulate invalidations. */
- if (txn->ninvalidations == 0)
- {
- txn->ninvalidations = nmsgs;
- txn->invalidations = (SharedInvalidationMessage *)
- palloc(sizeof(SharedInvalidationMessage) * nmsgs);
- memcpy(txn->invalidations, msgs,
- sizeof(SharedInvalidationMessage) * nmsgs);
- }
- else
+ ReorderBufferAccumulateInvalidations(&txn->invalidations,
+ &txn->ninvalidations,
+ msgs, nmsgs);
+
+ ReorderBufferQueueInvalidations(rb, xid, lsn, nmsgs, msgs);
+
+ MemoryContextSwitchTo(oldcontext);
+}
+
+/*
+ * Accumulate the invalidations distributed by other committed transactions
+ * for executing them later.
+ *
+ * This function is similar to ReorderBufferAddInvalidations() but stores
+ * the given inval messages to the txn->invalidations_distributed with the
+ * overflow check.
+ *
+ * This needs to be called by committed transactions to distribute their
+ * inval messages to in-progress transactions.
+ */
+void
+ReorderBufferAddDistributedInvalidations(ReorderBuffer *rb, TransactionId xid,
+ XLogRecPtr lsn, Size nmsgs,
+ SharedInvalidationMessage *msgs)
+{
+ ReorderBufferTXN *txn;
+ MemoryContext oldcontext;
+
+ txn = ReorderBufferTXNByXid(rb, xid, true, NULL, lsn, true);
+
+ oldcontext = MemoryContextSwitchTo(rb->context);
+
+ /*
+ * Collect all the invalidations under the top transaction, if available,
+ * so that we can execute them all together. See comments
+ * ReorderBufferAddInvalidations.
+ */
+ txn = rbtxn_get_toptxn(txn);
+
+ Assert(nmsgs > 0);
+
+ if (!rbtxn_distr_inval_overflowed(txn))
{
- txn->invalidations = (SharedInvalidationMessage *)
- repalloc(txn->invalidations, sizeof(SharedInvalidationMessage) *
- (txn->ninvalidations + nmsgs));
+ /*
+ * Check the transaction has enough space for storing distributed
+ * invalidation messages.
+ */
+ if (txn->ninvalidations_distributed + nmsgs >= MAX_DISTR_INVAL_MSG_PER_TXN)
+ {
+ /*
+ * Mark the invalidation message as overflowed and free up the
+ * messages accumulated so far.
+ */
+ txn->txn_flags |= RBTXN_DISTR_INVAL_OVERFLOWED;
- memcpy(txn->invalidations + txn->ninvalidations, msgs,
- nmsgs * sizeof(SharedInvalidationMessage));
- txn->ninvalidations += nmsgs;
+ if (txn->invalidations_distributed)
+ {
+ pfree(txn->invalidations_distributed);
+ txn->invalidations_distributed = NULL;
+ txn->ninvalidations_distributed = 0;
+ }
+ }
+ else
+ ReorderBufferAccumulateInvalidations(&txn->invalidations_distributed,
+ &txn->ninvalidations_distributed,
+ msgs, nmsgs);
}
- change = ReorderBufferAllocChange(rb);
- change->action = REORDER_BUFFER_CHANGE_INVALIDATION;
- change->data.inval.ninvalidations = nmsgs;
- change->data.inval.invalidations = (SharedInvalidationMessage *)
- palloc(sizeof(SharedInvalidationMessage) * nmsgs);
- memcpy(change->data.inval.invalidations, msgs,
- sizeof(SharedInvalidationMessage) * nmsgs);
-
- ReorderBufferQueueChange(rb, xid, lsn, change, false);
+ /* Queue the invalidation messages into the transaction */
+ ReorderBufferQueueInvalidations(rb, xid, lsn, nmsgs, msgs);
MemoryContextSwitchTo(oldcontext);
}
diff --git a/src/backend/replication/logical/snapbuild.c b/src/backend/replication/logical/snapbuild.c
index 0d7bddbe4ed..adf18c397db 100644
--- a/src/backend/replication/logical/snapbuild.c
+++ b/src/backend/replication/logical/snapbuild.c
@@ -794,6 +794,13 @@ SnapBuildDistributeSnapshotAndInval(SnapBuild *builder, XLogRecPtr lsn, Transact
* contents built by the current transaction even after its decoding,
* which should have been invalidated due to concurrent catalog
* changing transaction.
+ *
+ * Distribute only the invalidation messages generated by the current
+ * committed transaction. Invalidation messages received from other
+ * transactions would have already been propagated to the relevant
+ * in-progress transactions. This transaction would have processed
+ * those invalidations, ensuring that subsequent transactions observe
+ * a consistent cache state.
*/
if (txn->xid != xid)
{
@@ -807,8 +814,9 @@ SnapBuildDistributeSnapshotAndInval(SnapBuild *builder, XLogRecPtr lsn, Transact
{
Assert(msgs != NULL);
- ReorderBufferAddInvalidations(builder->reorder, txn->xid, lsn,
- ninvalidations, msgs);
+ ReorderBufferAddDistributedInvalidations(builder->reorder,
+ txn->xid, lsn,
+ ninvalidations, msgs);
}
}
}
diff --git a/src/include/replication/reorderbuffer.h b/src/include/replication/reorderbuffer.h
index 24e88c409ba..fa0745552f8 100644
--- a/src/include/replication/reorderbuffer.h
+++ b/src/include/replication/reorderbuffer.h
@@ -176,6 +176,7 @@ typedef struct ReorderBufferChange
#define RBTXN_SENT_PREPARE 0x0200
#define RBTXN_IS_COMMITTED 0x0400
#define RBTXN_IS_ABORTED 0x0800
+#define RBTXN_DISTR_INVAL_OVERFLOWED 0x1000
#define RBTXN_PREPARE_STATUS_MASK (RBTXN_IS_PREPARED | RBTXN_SKIPPED_PREPARE | RBTXN_SENT_PREPARE)
@@ -265,6 +266,12 @@ typedef struct ReorderBufferChange
((txn)->txn_flags & RBTXN_SKIPPED_PREPARE) != 0 \
)
+/* Is the array of distributed inval messages overflowed? */
+#define rbtxn_distr_inval_overflowed(txn) \
+( \
+ ((txn)->txn_flags & RBTXN_DISTR_INVAL_OVERFLOWED) != 0 \
+)
+
/* Is this a top-level transaction? */
#define rbtxn_is_toptxn(txn) \
( \
@@ -422,6 +429,12 @@ typedef struct ReorderBufferTXN
uint32 ninvalidations;
SharedInvalidationMessage *invalidations;
+ /*
+ * Stores cache invalidation messages distributed by other transactions.
+ */
+ uint32 ninvalidations_distributed;
+ SharedInvalidationMessage *invalidations_distributed;
+
/* ---
* Position in one of two lists:
* * list of subtransactions if we are *known* to be subxact
@@ -738,6 +751,9 @@ extern void ReorderBufferAddNewTupleCids(ReorderBuffer *rb, TransactionId xid,
CommandId cmin, CommandId cmax, CommandId combocid);
extern void ReorderBufferAddInvalidations(ReorderBuffer *rb, TransactionId xid, XLogRecPtr lsn,
Size nmsgs, SharedInvalidationMessage *msgs);
+extern void ReorderBufferAddDistributedInvalidations(ReorderBuffer *rb, TransactionId xid,
+ XLogRecPtr lsn, Size nmsgs,
+ SharedInvalidationMessage *msgs);
extern void ReorderBufferImmediateInvalidation(ReorderBuffer *rb, uint32 ninvalidations,
SharedInvalidationMessage *invalidations);
extern void ReorderBufferProcessXid(ReorderBuffer *rb, TransactionId xid, XLogRecPtr lsn);
--
2.43.5
[application/octet-stream] REL16_v15-0001-Fix-re-distributing-previously-distributed-inval.patch (19.2K, ../../CAD21AoCHqKXVhUZbxPKBve-uEHJWKxV9SVC5deiB5VUONTH=6w@mail.gmail.com/4-REL16_v15-0001-Fix-re-distributing-previously-distributed-inval.patch)
download | inline diff:
From 19b9b85a61560edc390386e187809bb0d84e8e5a Mon Sep 17 00:00:00 2001
From: Vignesh C <vignesh21@gmail.com>
Date: Thu, 12 Jun 2025 09:51:04 +0530
Subject: [PATCH v15] Fix re-distributing previously distributed invalidation
messages during logical decoding.
Commit 4909b38af0 introduced logic to distribute invalidation messages
from catalog-modifying transactions to all concurrent in-progress
transactions. However, since each transaction distributes not only its
original invalidation messages but also previously distributed
messages to other transactions, this leads to an exponential increase
in allocation request size for invalidation messages, ultimately
causing memory allocation failure.
This commit fixes this issue by tracking distributed invalidation
messages separately per decoded transaction and not redistributing
these messages to other in-progress transactions. The maximum size of
distributed invalidation messages that one transaction can store is
limited to MAX_DISTR_INVAL_MSG_PER_TXN (8MB). Once the size of the
distributed invalidation messages exceeds this threshold, we
invalidate all caches in locations where distributed invalidation
messages need to be executed.
Back-patch to all supported versions where we introduced the fix by
commit 4909b38af0.
Note that this commit adds two new fields to ReorderBufferTXN to store
the distributed transactions. This change breaks ABI compatibility in
back branches, affecting third-party extensions that depend on the
size of the ReorderBufferTXN struct, though this scenario seems
unlikely.
Additionally, it adds a new flag to the txn_flags field of
ReorderBufferTXN to indicate distributed invalidation message
overflow. This should not affect existing implementations, as it is
unlikely that third-party extensions use unused bits in the txn_flags
field.
Bug: #18938 #18942
Author: vignesh C <vignesh21@gmail.com>
Reported-by: Duncan Sands <duncan.sands@deepbluecap.com>
Reported-by: John Hutchins <john.hutchins@wicourts.gov>
Reported-by: Laurence Parry <greenreaper@hotmail.com>
Reported-by: Max Madden <maxmmadden@gmail.com>
Reported-by: Braulio Fdo Gonzalez <brauliofg@gmail.com>
Reviewed-by: Masahiko Sawada <sawada.mshk@gmail.com>
Reviewed-by: Amit Kapila <amit.kapila16@gmail.com>
Reviewed-by: Hayato Kuroda <kuroda.hayato@fujitsu.com>
Discussion: https://postgr.es/m/680bdaf6-f7d1-4536-b580-05c2760c67c6@deepbluecap.com
Discussion: https://postgr.es/m/18942-0ab1e5ae156613ad@postgresql.org
Discussion: https://postgr.es/m/18938-57c9a1c463b68ce0@postgresql.org
Discussion: https://postgr.es/m/CAD1FGCT2sYrP_70RTuo56QTizyc+J3wJdtn2gtO3VttQFpdMZg@mail.gmail.com
Discussion: https://postgr.es/m/CANO2=B=2BT1hSYCE=nuuTnVTnjidMg0+-FfnRnqM6kd23qoygg@mail.gmail.com
Backpatch-through: 13
---
.../expected/invalidation_distribution.out | 23 ++-
.../specs/invalidation_distribution.spec | 11 +
.../replication/logical/reorderbuffer.c | 195 +++++++++++++++---
src/backend/replication/logical/snapbuild.c | 12 +-
src/include/replication/reorderbuffer.h | 34 ++-
5 files changed, 230 insertions(+), 45 deletions(-)
diff --git a/contrib/test_decoding/expected/invalidation_distribution.out b/contrib/test_decoding/expected/invalidation_distribution.out
index ad0a944cbf3..ae53b1e61de 100644
--- a/contrib/test_decoding/expected/invalidation_distribution.out
+++ b/contrib/test_decoding/expected/invalidation_distribution.out
@@ -1,4 +1,4 @@
-Parsed test spec with 2 sessions
+Parsed test spec with 3 sessions
starting permutation: s1_insert_tbl1 s1_begin s1_insert_tbl1 s2_alter_pub_add_tbl s1_commit s1_insert_tbl1 s2_get_binary_changes
step s1_insert_tbl1: INSERT INTO tbl1 (val1, val2) VALUES (1, 1);
@@ -18,3 +18,24 @@ count
stop
(1 row)
+
+starting permutation: s1_begin s1_insert_tbl1 s3_begin s3_insert_tbl1 s2_alter_pub_add_tbl s1_insert_tbl1 s1_commit s3_commit s2_get_binary_changes
+step s1_begin: BEGIN;
+step s1_insert_tbl1: INSERT INTO tbl1 (val1, val2) VALUES (1, 1);
+step s3_begin: BEGIN;
+step s3_insert_tbl1: INSERT INTO tbl1 (val1, val2) VALUES (2, 2);
+step s2_alter_pub_add_tbl: ALTER PUBLICATION pub ADD TABLE tbl1;
+step s1_insert_tbl1: INSERT INTO tbl1 (val1, val2) VALUES (1, 1);
+step s1_commit: COMMIT;
+step s3_commit: COMMIT;
+step s2_get_binary_changes: SELECT count(data) FROM pg_logical_slot_get_binary_changes('isolation_slot', NULL, NULL, 'proto_version', '4', 'publication_names', 'pub') WHERE get_byte(data, 0) = 73;
+count
+-----
+ 1
+(1 row)
+
+?column?
+--------
+stop
+(1 row)
+
diff --git a/contrib/test_decoding/specs/invalidation_distribution.spec b/contrib/test_decoding/specs/invalidation_distribution.spec
index decbed627e3..67d41969ac1 100644
--- a/contrib/test_decoding/specs/invalidation_distribution.spec
+++ b/contrib/test_decoding/specs/invalidation_distribution.spec
@@ -28,5 +28,16 @@ setup { SET synchronous_commit=on; }
step "s2_alter_pub_add_tbl" { ALTER PUBLICATION pub ADD TABLE tbl1; }
step "s2_get_binary_changes" { SELECT count(data) FROM pg_logical_slot_get_binary_changes('isolation_slot', NULL, NULL, 'proto_version', '4', 'publication_names', 'pub') WHERE get_byte(data, 0) = 73; }
+session "s3"
+setup { SET synchronous_commit=on; }
+step "s3_begin" { BEGIN; }
+step "s3_insert_tbl1" { INSERT INTO tbl1 (val1, val2) VALUES (2, 2); }
+step "s3_commit" { COMMIT; }
+
# Expect to get one insert change. LOGICAL_REP_MSG_INSERT = 'I'
permutation "s1_insert_tbl1" "s1_begin" "s1_insert_tbl1" "s2_alter_pub_add_tbl" "s1_commit" "s1_insert_tbl1" "s2_get_binary_changes"
+
+# Expect to get one insert change with LOGICAL_REP_MSG_INSERT = 'I' from
+# the second "s1_insert_tbl1" executed after adding the table tbl1 to the
+# publication in "s2_alter_pub_add_tbl".
+permutation "s1_begin" "s1_insert_tbl1" "s3_begin" "s3_insert_tbl1" "s2_alter_pub_add_tbl" "s1_insert_tbl1" "s1_commit" "s3_commit" "s2_get_binary_changes"
diff --git a/src/backend/replication/logical/reorderbuffer.c b/src/backend/replication/logical/reorderbuffer.c
index fa04e829cc9..6464a926c20 100644
--- a/src/backend/replication/logical/reorderbuffer.c
+++ b/src/backend/replication/logical/reorderbuffer.c
@@ -103,11 +103,22 @@
#include "storage/sinval.h"
#include "utils/builtins.h"
#include "utils/combocid.h"
+#include "utils/inval.h"
#include "utils/memdebug.h"
#include "utils/memutils.h"
#include "utils/rel.h"
#include "utils/relfilenumbermap.h"
+/*
+ * Each transaction has an 8MB limit for invalidation messages distributed from
+ * other transactions. This limit is set considering scenarios with many
+ * concurrent logical decoding operations. When the distributed invalidation
+ * messages reach this threshold, the transaction is marked as
+ * RBTXN_DISTR_INVAL_OVERFLOWED to invalidate the complete cache as we have lost
+ * some inval messages and hence don't know what needs to be invalidated.
+ */
+#define MAX_DISTR_INVAL_MSG_PER_TXN \
+ ((8 * 1024 * 1024) / sizeof(SharedInvalidationMessage))
/* entry for a hash table we use to map from xid to our transaction state */
typedef struct ReorderBufferTXNByIdEnt
@@ -461,6 +472,12 @@ ReorderBufferReturnTXN(ReorderBuffer *rb, ReorderBufferTXN *txn)
txn->invalidations = NULL;
}
+ if (txn->invalidations_distributed)
+ {
+ pfree(txn->invalidations_distributed);
+ txn->invalidations_distributed = NULL;
+ }
+
/* Reset the toast hash */
ReorderBufferToastReset(rb, txn);
@@ -2539,7 +2556,17 @@ ReorderBufferProcessTXN(ReorderBuffer *rb, ReorderBufferTXN *txn,
AbortCurrentTransaction();
/* make sure there's no cache pollution */
- ReorderBufferExecuteInvalidations(txn->ninvalidations, txn->invalidations);
+ if (rbtxn_distr_inval_overflowed(txn))
+ {
+ Assert(txn->ninvalidations_distributed == 0);
+ InvalidateSystemCaches();
+ }
+ else
+ {
+ ReorderBufferExecuteInvalidations(txn->ninvalidations, txn->invalidations);
+ ReorderBufferExecuteInvalidations(txn->ninvalidations_distributed,
+ txn->invalidations_distributed);
+ }
if (using_subtxn)
RollbackAndReleaseCurrentSubTransaction();
@@ -2585,8 +2612,17 @@ ReorderBufferProcessTXN(ReorderBuffer *rb, ReorderBufferTXN *txn,
AbortCurrentTransaction();
/* make sure there's no cache pollution */
- ReorderBufferExecuteInvalidations(txn->ninvalidations,
- txn->invalidations);
+ if (rbtxn_distr_inval_overflowed(txn))
+ {
+ Assert(txn->ninvalidations_distributed == 0);
+ InvalidateSystemCaches();
+ }
+ else
+ {
+ ReorderBufferExecuteInvalidations(txn->ninvalidations, txn->invalidations);
+ ReorderBufferExecuteInvalidations(txn->ninvalidations_distributed,
+ txn->invalidations_distributed);
+ }
if (using_subtxn)
RollbackAndReleaseCurrentSubTransaction();
@@ -2916,7 +2952,8 @@ ReorderBufferAbort(ReorderBuffer *rb, TransactionId xid, XLogRecPtr lsn,
* We might have decoded changes for this transaction that could load
* the cache as per the current transaction's view (consider DDL's
* happened in this transaction). We don't want the decoding of future
- * transactions to use those cache entries so execute invalidations.
+ * transactions to use those cache entries so execute only the inval
+ * messages in this transaction.
*/
if (txn->ninvalidations > 0)
ReorderBufferImmediateInvalidation(rb, txn->ninvalidations,
@@ -3003,9 +3040,10 @@ ReorderBufferForget(ReorderBuffer *rb, TransactionId xid, XLogRecPtr lsn)
txn->final_lsn = lsn;
/*
- * Process cache invalidation messages if there are any. Even if we're not
- * interested in the transaction's contents, it could have manipulated the
- * catalog and we need to update the caches according to that.
+ * Process only cache invalidation messages in this transaction if there
+ * are any. Even if we're not interested in the transaction's contents, it
+ * could have manipulated the catalog and we need to update the caches
+ * according to that.
*/
if (txn->base_snapshot != NULL && txn->ninvalidations > 0)
ReorderBufferImmediateInvalidation(rb, txn->ninvalidations,
@@ -3257,6 +3295,57 @@ ReorderBufferAddNewTupleCids(ReorderBuffer *rb, TransactionId xid,
txn->ntuplecids++;
}
+/*
+ * Add new invalidation messages to the reorder buffer queue.
+ */
+static void
+ReorderBufferQueueInvalidations(ReorderBuffer *rb, TransactionId xid,
+ XLogRecPtr lsn, Size nmsgs,
+ SharedInvalidationMessage *msgs)
+{
+ ReorderBufferChange *change;
+
+ change = ReorderBufferGetChange(rb);
+ change->action = REORDER_BUFFER_CHANGE_INVALIDATION;
+ change->data.inval.ninvalidations = nmsgs;
+ change->data.inval.invalidations = (SharedInvalidationMessage *)
+ palloc(sizeof(SharedInvalidationMessage) * nmsgs);
+ memcpy(change->data.inval.invalidations, msgs,
+ sizeof(SharedInvalidationMessage) * nmsgs);
+
+ ReorderBufferQueueChange(rb, xid, lsn, change, false);
+}
+
+/*
+ * A helper function for ReorderBufferAddInvalidations() and
+ * ReorderBufferAddDistributedInvalidations() to accumulate the invalidation
+ * messages to the **invals_out.
+ */
+static void
+ReorderBufferAccumulateInvalidations(SharedInvalidationMessage **invals_out,
+ uint32 *ninvals_out,
+ SharedInvalidationMessage *msgs_new,
+ Size nmsgs_new)
+{
+ if (*ninvals_out == 0)
+ {
+ *ninvals_out = nmsgs_new;
+ *invals_out = (SharedInvalidationMessage *)
+ palloc(sizeof(SharedInvalidationMessage) * nmsgs_new);
+ memcpy(*invals_out, msgs_new, sizeof(SharedInvalidationMessage) * nmsgs_new);
+ }
+ else
+ {
+ /* Enlarge the array of inval messages */
+ *invals_out = (SharedInvalidationMessage *)
+ repalloc(*invals_out, sizeof(SharedInvalidationMessage) *
+ (*ninvals_out + nmsgs_new));
+ memcpy(*invals_out + *ninvals_out, msgs_new,
+ nmsgs_new * sizeof(SharedInvalidationMessage));
+ *ninvals_out += nmsgs_new;
+ }
+}
+
/*
* Accumulate the invalidations for executing them later.
*
@@ -3277,7 +3366,6 @@ ReorderBufferAddInvalidations(ReorderBuffer *rb, TransactionId xid,
{
ReorderBufferTXN *txn;
MemoryContext oldcontext;
- ReorderBufferChange *change;
txn = ReorderBufferTXNByXid(rb, xid, true, NULL, lsn, true);
@@ -3292,35 +3380,76 @@ ReorderBufferAddInvalidations(ReorderBuffer *rb, TransactionId xid,
Assert(nmsgs > 0);
- /* Accumulate invalidations. */
- if (txn->ninvalidations == 0)
- {
- txn->ninvalidations = nmsgs;
- txn->invalidations = (SharedInvalidationMessage *)
- palloc(sizeof(SharedInvalidationMessage) * nmsgs);
- memcpy(txn->invalidations, msgs,
- sizeof(SharedInvalidationMessage) * nmsgs);
- }
- else
+ ReorderBufferAccumulateInvalidations(&txn->invalidations,
+ &txn->ninvalidations,
+ msgs, nmsgs);
+
+ ReorderBufferQueueInvalidations(rb, xid, lsn, nmsgs, msgs);
+
+ MemoryContextSwitchTo(oldcontext);
+}
+
+/*
+ * Accumulate the invalidations distributed by other committed transactions
+ * for executing them later.
+ *
+ * This function is similar to ReorderBufferAddInvalidations() but stores
+ * the given inval messages to the txn->invalidations_distributed with the
+ * overflow check.
+ *
+ * This needs to be called by committed transactions to distribute their
+ * inval messages to in-progress transactions.
+ */
+void
+ReorderBufferAddDistributedInvalidations(ReorderBuffer *rb, TransactionId xid,
+ XLogRecPtr lsn, Size nmsgs,
+ SharedInvalidationMessage *msgs)
+{
+ ReorderBufferTXN *txn;
+ MemoryContext oldcontext;
+
+ txn = ReorderBufferTXNByXid(rb, xid, true, NULL, lsn, true);
+
+ oldcontext = MemoryContextSwitchTo(rb->context);
+
+ /*
+ * Collect all the invalidations under the top transaction, if available,
+ * so that we can execute them all together. See comments
+ * ReorderBufferAddInvalidations.
+ */
+ txn = rbtxn_get_toptxn(txn);
+
+ Assert(nmsgs > 0);
+
+ if (!rbtxn_distr_inval_overflowed(txn))
{
- txn->invalidations = (SharedInvalidationMessage *)
- repalloc(txn->invalidations, sizeof(SharedInvalidationMessage) *
- (txn->ninvalidations + nmsgs));
+ /*
+ * Check the transaction has enough space for storing distributed
+ * invalidation messages.
+ */
+ if (txn->ninvalidations_distributed + nmsgs >= MAX_DISTR_INVAL_MSG_PER_TXN)
+ {
+ /*
+ * Mark the invalidation message as overflowed and free up the
+ * messages accumulated so far.
+ */
+ txn->txn_flags |= RBTXN_DISTR_INVAL_OVERFLOWED;
- memcpy(txn->invalidations + txn->ninvalidations, msgs,
- nmsgs * sizeof(SharedInvalidationMessage));
- txn->ninvalidations += nmsgs;
+ if (txn->invalidations_distributed)
+ {
+ pfree(txn->invalidations_distributed);
+ txn->invalidations_distributed = NULL;
+ txn->ninvalidations_distributed = 0;
+ }
+ }
+ else
+ ReorderBufferAccumulateInvalidations(&txn->invalidations_distributed,
+ &txn->ninvalidations_distributed,
+ msgs, nmsgs);
}
- change = ReorderBufferGetChange(rb);
- change->action = REORDER_BUFFER_CHANGE_INVALIDATION;
- change->data.inval.ninvalidations = nmsgs;
- change->data.inval.invalidations = (SharedInvalidationMessage *)
- palloc(sizeof(SharedInvalidationMessage) * nmsgs);
- memcpy(change->data.inval.invalidations, msgs,
- sizeof(SharedInvalidationMessage) * nmsgs);
-
- ReorderBufferQueueChange(rb, xid, lsn, change, false);
+ /* Queue the invalidation messages into the transaction */
+ ReorderBufferQueueInvalidations(rb, xid, lsn, nmsgs, msgs);
MemoryContextSwitchTo(oldcontext);
}
diff --git a/src/backend/replication/logical/snapbuild.c b/src/backend/replication/logical/snapbuild.c
index d48ebb8337b..6c49dda75b6 100644
--- a/src/backend/replication/logical/snapbuild.c
+++ b/src/backend/replication/logical/snapbuild.c
@@ -918,6 +918,13 @@ SnapBuildDistributeSnapshotAndInval(SnapBuild *builder, XLogRecPtr lsn, Transact
* contents built by the current transaction even after its decoding,
* which should have been invalidated due to concurrent catalog
* changing transaction.
+ *
+ * Distribute only the invalidation messages generated by the current
+ * committed transaction. Invalidation messages received from other
+ * transactions would have already been propagated to the relevant
+ * in-progress transactions. This transaction would have processed
+ * those invalidations, ensuring that subsequent transactions observe
+ * a consistent cache state.
*/
if (txn->xid != xid)
{
@@ -931,8 +938,9 @@ SnapBuildDistributeSnapshotAndInval(SnapBuild *builder, XLogRecPtr lsn, Transact
{
Assert(msgs != NULL);
- ReorderBufferAddInvalidations(builder->reorder, txn->xid, lsn,
- ninvalidations, msgs);
+ ReorderBufferAddDistributedInvalidations(builder->reorder,
+ txn->xid, lsn,
+ ninvalidations, msgs);
}
}
}
diff --git a/src/include/replication/reorderbuffer.h b/src/include/replication/reorderbuffer.h
index 8a32bbe28df..1606e0ea007 100644
--- a/src/include/replication/reorderbuffer.h
+++ b/src/include/replication/reorderbuffer.h
@@ -177,15 +177,16 @@ typedef struct ReorderBufferChange
} ReorderBufferChange;
/* ReorderBufferTXN txn_flags */
-#define RBTXN_HAS_CATALOG_CHANGES 0x0001
-#define RBTXN_IS_SUBXACT 0x0002
-#define RBTXN_IS_SERIALIZED 0x0004
-#define RBTXN_IS_SERIALIZED_CLEAR 0x0008
-#define RBTXN_IS_STREAMED 0x0010
-#define RBTXN_HAS_PARTIAL_CHANGE 0x0020
-#define RBTXN_PREPARE 0x0040
-#define RBTXN_SKIPPED_PREPARE 0x0080
-#define RBTXN_HAS_STREAMABLE_CHANGE 0x0100
+#define RBTXN_HAS_CATALOG_CHANGES 0x0001
+#define RBTXN_IS_SUBXACT 0x0002
+#define RBTXN_IS_SERIALIZED 0x0004
+#define RBTXN_IS_SERIALIZED_CLEAR 0x0008
+#define RBTXN_IS_STREAMED 0x0010
+#define RBTXN_HAS_PARTIAL_CHANGE 0x0020
+#define RBTXN_PREPARE 0x0040
+#define RBTXN_SKIPPED_PREPARE 0x0080
+#define RBTXN_HAS_STREAMABLE_CHANGE 0x0100
+#define RBTXN_DISTR_INVAL_OVERFLOWED 0x0200
/* Does the transaction have catalog changes? */
#define rbtxn_has_catalog_changes(txn) \
@@ -249,6 +250,12 @@ typedef struct ReorderBufferChange
((txn)->txn_flags & RBTXN_SKIPPED_PREPARE) != 0 \
)
+/* Is the array of distributed inval messages overflowed? */
+#define rbtxn_distr_inval_overflowed(txn) \
+( \
+ ((txn)->txn_flags & RBTXN_DISTR_INVAL_OVERFLOWED) != 0 \
+)
+
/* Is this a top-level transaction? */
#define rbtxn_is_toptxn(txn) \
( \
@@ -435,6 +442,12 @@ typedef struct ReorderBufferTXN
* Private data pointer of the output plugin.
*/
void *output_plugin_private;
+
+ /*
+ * Stores cache invalidation messages distributed by other transactions.
+ */
+ uint32 ninvalidations_distributed;
+ SharedInvalidationMessage *invalidations_distributed;
} ReorderBufferTXN;
/* so we can define the callbacks used inside struct ReorderBuffer itself */
@@ -728,6 +741,9 @@ extern void ReorderBufferAddNewTupleCids(ReorderBuffer *rb, TransactionId xid,
CommandId cmin, CommandId cmax, CommandId combocid);
extern void ReorderBufferAddInvalidations(ReorderBuffer *rb, TransactionId xid, XLogRecPtr lsn,
Size nmsgs, SharedInvalidationMessage *msgs);
+extern void ReorderBufferAddDistributedInvalidations(ReorderBuffer *rb, TransactionId xid,
+ XLogRecPtr lsn, Size nmsgs,
+ SharedInvalidationMessage *msgs);
extern void ReorderBufferImmediateInvalidation(ReorderBuffer *rb, uint32 ninvalidations,
SharedInvalidationMessage *invalidations);
extern void ReorderBufferProcessXid(ReorderBuffer *rb, TransactionId xid, XLogRecPtr lsn);
--
2.43.5
[application/octet-stream] REL14_v15-0001-Fix-re-distributing-previously-distributed-inval.patch (19.0K, ../../CAD21AoCHqKXVhUZbxPKBve-uEHJWKxV9SVC5deiB5VUONTH=6w@mail.gmail.com/5-REL14_v15-0001-Fix-re-distributing-previously-distributed-inval.patch)
download | inline diff:
From 590fbd1463aa49f16d230e65e41bad6318f58018 Mon Sep 17 00:00:00 2001
From: Vignesh C <vignesh21@gmail.com>
Date: Thu, 12 Jun 2025 11:08:32 +0530
Subject: [PATCH v15] Fix re-distributing previously distributed invalidation
messages during logical decoding.
Commit 4909b38af0 introduced logic to distribute invalidation messages
from catalog-modifying transactions to all concurrent in-progress
transactions. However, since each transaction distributes not only its
original invalidation messages but also previously distributed
messages to other transactions, this leads to an exponential increase
in allocation request size for invalidation messages, ultimately
causing memory allocation failure.
This commit fixes this issue by tracking distributed invalidation
messages separately per decoded transaction and not redistributing
these messages to other in-progress transactions. The maximum size of
distributed invalidation messages that one transaction can store is
limited to MAX_DISTR_INVAL_MSG_PER_TXN (8MB). Once the size of the
distributed invalidation messages exceeds this threshold, we
invalidate all caches in locations where distributed invalidation
messages need to be executed.
Back-patch to all supported versions where we introduced the fix by
commit 4909b38af0.
Note that this commit adds two new fields to ReorderBufferTXN to store
the distributed transactions. This change breaks ABI compatibility in
back branches, affecting third-party extensions that depend on the
size of the ReorderBufferTXN struct, though this scenario seems
unlikely.
Additionally, it adds a new flag to the txn_flags field of
ReorderBufferTXN to indicate distributed invalidation message
overflow. This should not affect existing implementations, as it is
unlikely that third-party extensions use unused bits in the txn_flags
field.
Bug: #18938 #18942
Author: vignesh C <vignesh21@gmail.com>
Reported-by: Duncan Sands <duncan.sands@deepbluecap.com>
Reported-by: John Hutchins <john.hutchins@wicourts.gov>
Reported-by: Laurence Parry <greenreaper@hotmail.com>
Reported-by: Max Madden <maxmmadden@gmail.com>
Reported-by: Braulio Fdo Gonzalez <brauliofg@gmail.com>
Reviewed-by: Masahiko Sawada <sawada.mshk@gmail.com>
Reviewed-by: Amit Kapila <amit.kapila16@gmail.com>
Reviewed-by: Hayato Kuroda <kuroda.hayato@fujitsu.com>
Discussion: https://postgr.es/m/680bdaf6-f7d1-4536-b580-05c2760c67c6@deepbluecap.com
Discussion: https://postgr.es/m/18942-0ab1e5ae156613ad@postgresql.org
Discussion: https://postgr.es/m/18938-57c9a1c463b68ce0@postgresql.org
Discussion: https://postgr.es/m/CAD1FGCT2sYrP_70RTuo56QTizyc+J3wJdtn2gtO3VttQFpdMZg@mail.gmail.com
Discussion: https://postgr.es/m/CANO2=B=2BT1hSYCE=nuuTnVTnjidMg0+-FfnRnqM6kd23qoygg@mail.gmail.com
Backpatch-through: 13
---
.../expected/invalidation_distribution.out | 23 +-
.../specs/invalidation_distribution.spec | 11 +
.../replication/logical/reorderbuffer.c | 197 +++++++++++++++---
src/backend/replication/logical/snapbuild.c | 12 +-
src/include/replication/reorderbuffer.h | 32 ++-
5 files changed, 231 insertions(+), 44 deletions(-)
diff --git a/contrib/test_decoding/expected/invalidation_distribution.out b/contrib/test_decoding/expected/invalidation_distribution.out
index c701e290bb9..f2bce5ce75c 100644
--- a/contrib/test_decoding/expected/invalidation_distribution.out
+++ b/contrib/test_decoding/expected/invalidation_distribution.out
@@ -1,4 +1,4 @@
-Parsed test spec with 2 sessions
+Parsed test spec with 3 sessions
starting permutation: s1_insert_tbl1 s1_begin s1_insert_tbl1 s2_alter_pub_add_tbl s1_commit s1_insert_tbl1 s2_get_binary_changes
step s1_insert_tbl1: INSERT INTO tbl1 (val1, val2) VALUES (1, 1);
@@ -18,3 +18,24 @@ count
stop
(1 row)
+
+starting permutation: s1_begin s1_insert_tbl1 s3_begin s3_insert_tbl1 s2_alter_pub_add_tbl s1_insert_tbl1 s1_commit s3_commit s2_get_binary_changes
+step s1_begin: BEGIN;
+step s1_insert_tbl1: INSERT INTO tbl1 (val1, val2) VALUES (1, 1);
+step s3_begin: BEGIN;
+step s3_insert_tbl1: INSERT INTO tbl1 (val1, val2) VALUES (2, 2);
+step s2_alter_pub_add_tbl: ALTER PUBLICATION pub ADD TABLE tbl1;
+step s1_insert_tbl1: INSERT INTO tbl1 (val1, val2) VALUES (1, 1);
+step s1_commit: COMMIT;
+step s3_commit: COMMIT;
+step s2_get_binary_changes: SELECT count(data) FROM pg_logical_slot_get_binary_changes('isolation_slot', NULL, NULL, 'proto_version', '2', 'publication_names', 'pub') WHERE get_byte(data, 0) = 73;
+count
+-----
+ 1
+(1 row)
+
+?column?
+--------
+stop
+(1 row)
+
diff --git a/contrib/test_decoding/specs/invalidation_distribution.spec b/contrib/test_decoding/specs/invalidation_distribution.spec
index b8b14e333a1..d9b5cd75aad 100644
--- a/contrib/test_decoding/specs/invalidation_distribution.spec
+++ b/contrib/test_decoding/specs/invalidation_distribution.spec
@@ -28,5 +28,16 @@ setup { SET synchronous_commit=on; }
step "s2_alter_pub_add_tbl" { ALTER PUBLICATION pub ADD TABLE tbl1; }
step "s2_get_binary_changes" { SELECT count(data) FROM pg_logical_slot_get_binary_changes('isolation_slot', NULL, NULL, 'proto_version', '2', 'publication_names', 'pub') WHERE get_byte(data, 0) = 73; }
+session "s3"
+setup { SET synchronous_commit=on; }
+step "s3_begin" { BEGIN; }
+step "s3_insert_tbl1" { INSERT INTO tbl1 (val1, val2) VALUES (2, 2); }
+step "s3_commit" { COMMIT; }
+
# Expect to get one insert change. LOGICAL_REP_MSG_INSERT = 'I'
permutation "s1_insert_tbl1" "s1_begin" "s1_insert_tbl1" "s2_alter_pub_add_tbl" "s1_commit" "s1_insert_tbl1" "s2_get_binary_changes"
+
+# Expect to get one insert change with LOGICAL_REP_MSG_INSERT = 'I' from
+# the second "s1_insert_tbl1" executed after adding the table tbl1 to the
+# publication in "s2_alter_pub_add_tbl".
+permutation "s1_begin" "s1_insert_tbl1" "s3_begin" "s3_insert_tbl1" "s2_alter_pub_add_tbl" "s1_insert_tbl1" "s1_commit" "s3_commit" "s2_get_binary_changes"
diff --git a/src/backend/replication/logical/reorderbuffer.c b/src/backend/replication/logical/reorderbuffer.c
index a52b51a5e7d..b708877d965 100644
--- a/src/backend/replication/logical/reorderbuffer.c
+++ b/src/backend/replication/logical/reorderbuffer.c
@@ -103,12 +103,24 @@
#include "storage/sinval.h"
#include "utils/builtins.h"
#include "utils/combocid.h"
+#include "utils/inval.h"
#include "utils/memdebug.h"
#include "utils/memutils.h"
#include "utils/rel.h"
#include "utils/relfilenodemap.h"
+/*
+ * Each transaction has an 8MB limit for invalidation messages distributed from
+ * other transactions. This limit is set considering scenarios with many
+ * concurrent logical decoding operations. When the distributed invalidation
+ * messages reach this threshold, the transaction is marked as
+ * RBTXN_DISTR_INVAL_OVERFLOWED to invalidate the complete cache as we have lost
+ * some inval messages and hence don't know what needs to be invalidated.
+ */
+#define MAX_DISTR_INVAL_MSG_PER_TXN \
+ ((8 * 1024 * 1024) / sizeof(SharedInvalidationMessage))
+
/* entry for a hash table we use to map from xid to our transaction state */
typedef struct ReorderBufferTXNByIdEnt
{
@@ -453,6 +465,12 @@ ReorderBufferReturnTXN(ReorderBuffer *rb, ReorderBufferTXN *txn)
txn->invalidations = NULL;
}
+ if (txn->invalidations_distributed)
+ {
+ pfree(txn->invalidations_distributed);
+ txn->invalidations_distributed = NULL;
+ }
+
/* Reset the toast hash */
ReorderBufferToastReset(rb, txn);
@@ -2500,7 +2518,17 @@ ReorderBufferProcessTXN(ReorderBuffer *rb, ReorderBufferTXN *txn,
AbortCurrentTransaction();
/* make sure there's no cache pollution */
- ReorderBufferExecuteInvalidations(txn->ninvalidations, txn->invalidations);
+ if (rbtxn_distr_inval_overflowed(txn))
+ {
+ Assert(txn->ninvalidations_distributed == 0);
+ InvalidateSystemCaches();
+ }
+ else
+ {
+ ReorderBufferExecuteInvalidations(txn->ninvalidations, txn->invalidations);
+ ReorderBufferExecuteInvalidations(txn->ninvalidations_distributed,
+ txn->invalidations_distributed);
+ }
if (using_subtxn)
RollbackAndReleaseCurrentSubTransaction();
@@ -2546,8 +2574,17 @@ ReorderBufferProcessTXN(ReorderBuffer *rb, ReorderBufferTXN *txn,
AbortCurrentTransaction();
/* make sure there's no cache pollution */
- ReorderBufferExecuteInvalidations(txn->ninvalidations,
- txn->invalidations);
+ if (rbtxn_distr_inval_overflowed(txn))
+ {
+ Assert(txn->ninvalidations_distributed == 0);
+ InvalidateSystemCaches();
+ }
+ else
+ {
+ ReorderBufferExecuteInvalidations(txn->ninvalidations, txn->invalidations);
+ ReorderBufferExecuteInvalidations(txn->ninvalidations_distributed,
+ txn->invalidations_distributed);
+ }
if (using_subtxn)
RollbackAndReleaseCurrentSubTransaction();
@@ -2873,7 +2910,8 @@ ReorderBufferAbort(ReorderBuffer *rb, TransactionId xid, XLogRecPtr lsn)
* We might have decoded changes for this transaction that could load
* the cache as per the current transaction's view (consider DDL's
* happened in this transaction). We don't want the decoding of future
- * transactions to use those cache entries so execute invalidations.
+ * transactions to use those cache entries so execute only the inval
+ * messages in this transaction.
*/
if (txn->ninvalidations > 0)
ReorderBufferImmediateInvalidation(rb, txn->ninvalidations,
@@ -2961,9 +2999,10 @@ ReorderBufferForget(ReorderBuffer *rb, TransactionId xid, XLogRecPtr lsn)
txn->final_lsn = lsn;
/*
- * Process cache invalidation messages if there are any. Even if we're not
- * interested in the transaction's contents, it could have manipulated the
- * catalog and we need to update the caches according to that.
+ * Process only cache invalidation messages in this transaction if there
+ * are any. Even if we're not interested in the transaction's contents, it
+ * could have manipulated the catalog and we need to update the caches
+ * according to that.
*/
if (txn->base_snapshot != NULL && txn->ninvalidations > 0)
ReorderBufferImmediateInvalidation(rb, txn->ninvalidations,
@@ -3218,6 +3257,57 @@ ReorderBufferAddNewTupleCids(ReorderBuffer *rb, TransactionId xid,
txn->ntuplecids++;
}
+/*
+ * Add new invalidation messages to the reorder buffer queue.
+ */
+static void
+ReorderBufferQueueInvalidations(ReorderBuffer *rb, TransactionId xid,
+ XLogRecPtr lsn, Size nmsgs,
+ SharedInvalidationMessage *msgs)
+{
+ ReorderBufferChange *change;
+
+ change = ReorderBufferGetChange(rb);
+ change->action = REORDER_BUFFER_CHANGE_INVALIDATION;
+ change->data.inval.ninvalidations = nmsgs;
+ change->data.inval.invalidations = (SharedInvalidationMessage *)
+ palloc(sizeof(SharedInvalidationMessage) * nmsgs);
+ memcpy(change->data.inval.invalidations, msgs,
+ sizeof(SharedInvalidationMessage) * nmsgs);
+
+ ReorderBufferQueueChange(rb, xid, lsn, change, false);
+}
+
+/*
+ * A helper function for ReorderBufferAddInvalidations() and
+ * ReorderBufferAddDistributedInvalidations() to accumulate the invalidation
+ * messages to the **invals_out.
+ */
+static void
+ReorderBufferAccumulateInvalidations(SharedInvalidationMessage **invals_out,
+ uint32 *ninvals_out,
+ SharedInvalidationMessage *msgs_new,
+ Size nmsgs_new)
+{
+ if (*ninvals_out == 0)
+ {
+ *ninvals_out = nmsgs_new;
+ *invals_out = (SharedInvalidationMessage *)
+ palloc(sizeof(SharedInvalidationMessage) * nmsgs_new);
+ memcpy(*invals_out, msgs_new, sizeof(SharedInvalidationMessage) * nmsgs_new);
+ }
+ else
+ {
+ /* Enlarge the array of inval messages */
+ *invals_out = (SharedInvalidationMessage *)
+ repalloc(*invals_out, sizeof(SharedInvalidationMessage) *
+ (*ninvals_out + nmsgs_new));
+ memcpy(*invals_out + *ninvals_out, msgs_new,
+ nmsgs_new * sizeof(SharedInvalidationMessage));
+ *ninvals_out += nmsgs_new;
+ }
+}
+
/*
* Accumulate the invalidations for executing them later.
*
@@ -3238,7 +3328,6 @@ ReorderBufferAddInvalidations(ReorderBuffer *rb, TransactionId xid,
{
ReorderBufferTXN *txn;
MemoryContext oldcontext;
- ReorderBufferChange *change;
txn = ReorderBufferTXNByXid(rb, xid, true, NULL, lsn, true);
@@ -3254,35 +3343,77 @@ ReorderBufferAddInvalidations(ReorderBuffer *rb, TransactionId xid,
Assert(nmsgs > 0);
- /* Accumulate invalidations. */
- if (txn->ninvalidations == 0)
- {
- txn->ninvalidations = nmsgs;
- txn->invalidations = (SharedInvalidationMessage *)
- palloc(sizeof(SharedInvalidationMessage) * nmsgs);
- memcpy(txn->invalidations, msgs,
- sizeof(SharedInvalidationMessage) * nmsgs);
- }
- else
+ ReorderBufferAccumulateInvalidations(&txn->invalidations,
+ &txn->ninvalidations,
+ msgs, nmsgs);
+
+ ReorderBufferQueueInvalidations(rb, xid, lsn, nmsgs, msgs);
+
+ MemoryContextSwitchTo(oldcontext);
+}
+
+/*
+ * Accumulate the invalidations distributed by other committed transactions
+ * for executing them later.
+ *
+ * This function is similar to ReorderBufferAddInvalidations() but stores
+ * the given inval messages to the txn->invalidations_distributed with the
+ * overflow check.
+ *
+ * This needs to be called by committed transactions to distribute their
+ * inval messages to in-progress transactions.
+ */
+void
+ReorderBufferAddDistributedInvalidations(ReorderBuffer *rb, TransactionId xid,
+ XLogRecPtr lsn, Size nmsgs,
+ SharedInvalidationMessage *msgs)
+{
+ ReorderBufferTXN *txn;
+ MemoryContext oldcontext;
+
+ txn = ReorderBufferTXNByXid(rb, xid, true, NULL, lsn, true);
+
+ oldcontext = MemoryContextSwitchTo(rb->context);
+
+ /*
+ * Collect all the invalidations under the top transaction, if available,
+ * so that we can execute them all together. See comments
+ * ReorderBufferAddInvalidations.
+ */
+ if (txn->toptxn)
+ txn = txn->toptxn;
+
+ Assert(nmsgs > 0);
+
+ if (!rbtxn_distr_inval_overflowed(txn))
{
- txn->invalidations = (SharedInvalidationMessage *)
- repalloc(txn->invalidations, sizeof(SharedInvalidationMessage) *
- (txn->ninvalidations + nmsgs));
+ /*
+ * Check the transaction has enough space for storing distributed
+ * invalidation messages.
+ */
+ if (txn->ninvalidations_distributed + nmsgs >= MAX_DISTR_INVAL_MSG_PER_TXN)
+ {
+ /*
+ * Mark the invalidation message as overflowed and free up the
+ * messages accumulated so far.
+ */
+ txn->txn_flags |= RBTXN_DISTR_INVAL_OVERFLOWED;
- memcpy(txn->invalidations + txn->ninvalidations, msgs,
- nmsgs * sizeof(SharedInvalidationMessage));
- txn->ninvalidations += nmsgs;
+ if (txn->invalidations_distributed)
+ {
+ pfree(txn->invalidations_distributed);
+ txn->invalidations_distributed = NULL;
+ txn->ninvalidations_distributed = 0;
+ }
+ }
+ else
+ ReorderBufferAccumulateInvalidations(&txn->invalidations_distributed,
+ &txn->ninvalidations_distributed,
+ msgs, nmsgs);
}
- change = ReorderBufferGetChange(rb);
- change->action = REORDER_BUFFER_CHANGE_INVALIDATION;
- change->data.inval.ninvalidations = nmsgs;
- change->data.inval.invalidations = (SharedInvalidationMessage *)
- palloc(sizeof(SharedInvalidationMessage) * nmsgs);
- memcpy(change->data.inval.invalidations, msgs,
- sizeof(SharedInvalidationMessage) * nmsgs);
-
- ReorderBufferQueueChange(rb, xid, lsn, change, false);
+ /* Queue the invalidation messages into the transaction */
+ ReorderBufferQueueInvalidations(rb, xid, lsn, nmsgs, msgs);
MemoryContextSwitchTo(oldcontext);
}
diff --git a/src/backend/replication/logical/snapbuild.c b/src/backend/replication/logical/snapbuild.c
index 6025f2727ae..67f0b9e089a 100644
--- a/src/backend/replication/logical/snapbuild.c
+++ b/src/backend/replication/logical/snapbuild.c
@@ -916,6 +916,13 @@ SnapBuildDistributeSnapshotAndInval(SnapBuild *builder, XLogRecPtr lsn, Transact
* contents built by the current transaction even after its decoding,
* which should have been invalidated due to concurrent catalog
* changing transaction.
+ *
+ * Distribute only the invalidation messages generated by the current
+ * committed transaction. Invalidation messages received from other
+ * transactions would have already been propagated to the relevant
+ * in-progress transactions. This transaction would have processed
+ * those invalidations, ensuring that subsequent transactions observe
+ * a consistent cache state.
*/
if (txn->xid != xid)
{
@@ -929,8 +936,9 @@ SnapBuildDistributeSnapshotAndInval(SnapBuild *builder, XLogRecPtr lsn, Transact
{
Assert(msgs != NULL);
- ReorderBufferAddInvalidations(builder->reorder, txn->xid, lsn,
- ninvalidations, msgs);
+ ReorderBufferAddDistributedInvalidations(builder->reorder,
+ txn->xid, lsn,
+ ninvalidations, msgs);
}
}
}
diff --git a/src/include/replication/reorderbuffer.h b/src/include/replication/reorderbuffer.h
index d399975e8a5..cb633315d04 100644
--- a/src/include/replication/reorderbuffer.h
+++ b/src/include/replication/reorderbuffer.h
@@ -168,14 +168,15 @@ typedef struct ReorderBufferChange
} ReorderBufferChange;
/* ReorderBufferTXN txn_flags */
-#define RBTXN_HAS_CATALOG_CHANGES 0x0001
-#define RBTXN_IS_SUBXACT 0x0002
-#define RBTXN_IS_SERIALIZED 0x0004
-#define RBTXN_IS_SERIALIZED_CLEAR 0x0008
-#define RBTXN_IS_STREAMED 0x0010
-#define RBTXN_HAS_PARTIAL_CHANGE 0x0020
-#define RBTXN_PREPARE 0x0040
-#define RBTXN_SKIPPED_PREPARE 0x0080
+#define RBTXN_HAS_CATALOG_CHANGES 0x0001
+#define RBTXN_IS_SUBXACT 0x0002
+#define RBTXN_IS_SERIALIZED 0x0004
+#define RBTXN_IS_SERIALIZED_CLEAR 0x0008
+#define RBTXN_IS_STREAMED 0x0010
+#define RBTXN_HAS_PARTIAL_CHANGE 0x0020
+#define RBTXN_PREPARE 0x0040
+#define RBTXN_SKIPPED_PREPARE 0x0080
+#define RBTXN_DISTR_INVAL_OVERFLOWED 0x0100
/* Does the transaction have catalog changes? */
#define rbtxn_has_catalog_changes(txn) \
@@ -233,6 +234,12 @@ typedef struct ReorderBufferChange
((txn)->txn_flags & RBTXN_SKIPPED_PREPARE) != 0 \
)
+/* Is the array of distributed inval messages overflowed? */
+#define rbtxn_distr_inval_overflowed(txn) \
+( \
+ ((txn)->txn_flags & RBTXN_DISTR_INVAL_OVERFLOWED) != 0 \
+)
+
typedef struct ReorderBufferTXN
{
/* See above */
@@ -391,6 +398,12 @@ typedef struct ReorderBufferTXN
* Private data pointer of the output plugin.
*/
void *output_plugin_private;
+
+ /*
+ * Stores cache invalidation messages distributed by other transactions.
+ */
+ uint32 ninvalidations_distributed;
+ SharedInvalidationMessage *invalidations_distributed;
} ReorderBufferTXN;
/* so we can define the callbacks used inside struct ReorderBuffer itself */
@@ -657,6 +670,9 @@ void ReorderBufferAddNewTupleCids(ReorderBuffer *, TransactionId, XLogRecPtr ls
CommandId cmin, CommandId cmax, CommandId combocid);
void ReorderBufferAddInvalidations(ReorderBuffer *, TransactionId, XLogRecPtr lsn,
Size nmsgs, SharedInvalidationMessage *msgs);
+void ReorderBufferAddDistributedInvalidations(ReorderBuffer *rb, TransactionId xid,
+ XLogRecPtr lsn, Size nmsgs,
+ SharedInvalidationMessage *msgs);
void ReorderBufferImmediateInvalidation(ReorderBuffer *, uint32 ninvalidations,
SharedInvalidationMessage *invalidations);
void ReorderBufferProcessXid(ReorderBuffer *, TransactionId xid, XLogRecPtr lsn);
--
2.43.5
[application/octet-stream] REL17_v15-0001-Fix-re-distributing-previously-distributed-inval.patch (19.1K, ../../CAD21AoCHqKXVhUZbxPKBve-uEHJWKxV9SVC5deiB5VUONTH=6w@mail.gmail.com/6-REL17_v15-0001-Fix-re-distributing-previously-distributed-inval.patch)
download | inline diff:
From 82207f0f6b282d0ef687d30fe16e3d977618e22a Mon Sep 17 00:00:00 2001
From: Vignesh C <vignesh21@gmail.com>
Date: Thu, 12 Jun 2025 09:38:58 +0530
Subject: [PATCH v15] Fix re-distributing previously distributed invalidation
messages during logical decoding.
Commit 4909b38af0 introduced logic to distribute invalidation messages
from catalog-modifying transactions to all concurrent in-progress
transactions. However, since each transaction distributes not only its
original invalidation messages but also previously distributed
messages to other transactions, this leads to an exponential increase
in allocation request size for invalidation messages, ultimately
causing memory allocation failure.
This commit fixes this issue by tracking distributed invalidation
messages separately per decoded transaction and not redistributing
these messages to other in-progress transactions. The maximum size of
distributed invalidation messages that one transaction can store is
limited to MAX_DISTR_INVAL_MSG_PER_TXN (8MB). Once the size of the
distributed invalidation messages exceeds this threshold, we
invalidate all caches in locations where distributed invalidation
messages need to be executed.
Back-patch to all supported versions where we introduced the fix by
commit 4909b38af0.
Note that this commit adds two new fields to ReorderBufferTXN to store
the distributed transactions. This change breaks ABI compatibility in
back branches, affecting third-party extensions that depend on the
size of the ReorderBufferTXN struct, though this scenario seems
unlikely.
Additionally, it adds a new flag to the txn_flags field of
ReorderBufferTXN to indicate distributed invalidation message
overflow. This should not affect existing implementations, as it is
unlikely that third-party extensions use unused bits in the txn_flags
field.
Bug: #18938 #18942
Author: vignesh C <vignesh21@gmail.com>
Reported-by: Duncan Sands <duncan.sands@deepbluecap.com>
Reported-by: John Hutchins <john.hutchins@wicourts.gov>
Reported-by: Laurence Parry <greenreaper@hotmail.com>
Reported-by: Max Madden <maxmmadden@gmail.com>
Reported-by: Braulio Fdo Gonzalez <brauliofg@gmail.com>
Reviewed-by: Masahiko Sawada <sawada.mshk@gmail.com>
Reviewed-by: Amit Kapila <amit.kapila16@gmail.com>
Reviewed-by: Hayato Kuroda <kuroda.hayato@fujitsu.com>
Discussion: https://postgr.es/m/680bdaf6-f7d1-4536-b580-05c2760c67c6@deepbluecap.com
Discussion: https://postgr.es/m/18942-0ab1e5ae156613ad@postgresql.org
Discussion: https://postgr.es/m/18938-57c9a1c463b68ce0@postgresql.org
Discussion: https://postgr.es/m/CAD1FGCT2sYrP_70RTuo56QTizyc+J3wJdtn2gtO3VttQFpdMZg@mail.gmail.com
Discussion: https://postgr.es/m/CANO2=B=2BT1hSYCE=nuuTnVTnjidMg0+-FfnRnqM6kd23qoygg@mail.gmail.com
Backpatch-through: 13
---
.../expected/invalidation_distribution.out | 23 +-
.../specs/invalidation_distribution.spec | 11 +
.../replication/logical/reorderbuffer.c | 196 +++++++++++++++---
src/backend/replication/logical/snapbuild.c | 12 +-
src/include/replication/reorderbuffer.h | 34 ++-
5 files changed, 231 insertions(+), 45 deletions(-)
diff --git a/contrib/test_decoding/expected/invalidation_distribution.out b/contrib/test_decoding/expected/invalidation_distribution.out
index ad0a944cbf3..ae53b1e61de 100644
--- a/contrib/test_decoding/expected/invalidation_distribution.out
+++ b/contrib/test_decoding/expected/invalidation_distribution.out
@@ -1,4 +1,4 @@
-Parsed test spec with 2 sessions
+Parsed test spec with 3 sessions
starting permutation: s1_insert_tbl1 s1_begin s1_insert_tbl1 s2_alter_pub_add_tbl s1_commit s1_insert_tbl1 s2_get_binary_changes
step s1_insert_tbl1: INSERT INTO tbl1 (val1, val2) VALUES (1, 1);
@@ -18,3 +18,24 @@ count
stop
(1 row)
+
+starting permutation: s1_begin s1_insert_tbl1 s3_begin s3_insert_tbl1 s2_alter_pub_add_tbl s1_insert_tbl1 s1_commit s3_commit s2_get_binary_changes
+step s1_begin: BEGIN;
+step s1_insert_tbl1: INSERT INTO tbl1 (val1, val2) VALUES (1, 1);
+step s3_begin: BEGIN;
+step s3_insert_tbl1: INSERT INTO tbl1 (val1, val2) VALUES (2, 2);
+step s2_alter_pub_add_tbl: ALTER PUBLICATION pub ADD TABLE tbl1;
+step s1_insert_tbl1: INSERT INTO tbl1 (val1, val2) VALUES (1, 1);
+step s1_commit: COMMIT;
+step s3_commit: COMMIT;
+step s2_get_binary_changes: SELECT count(data) FROM pg_logical_slot_get_binary_changes('isolation_slot', NULL, NULL, 'proto_version', '4', 'publication_names', 'pub') WHERE get_byte(data, 0) = 73;
+count
+-----
+ 1
+(1 row)
+
+?column?
+--------
+stop
+(1 row)
+
diff --git a/contrib/test_decoding/specs/invalidation_distribution.spec b/contrib/test_decoding/specs/invalidation_distribution.spec
index decbed627e3..67d41969ac1 100644
--- a/contrib/test_decoding/specs/invalidation_distribution.spec
+++ b/contrib/test_decoding/specs/invalidation_distribution.spec
@@ -28,5 +28,16 @@ setup { SET synchronous_commit=on; }
step "s2_alter_pub_add_tbl" { ALTER PUBLICATION pub ADD TABLE tbl1; }
step "s2_get_binary_changes" { SELECT count(data) FROM pg_logical_slot_get_binary_changes('isolation_slot', NULL, NULL, 'proto_version', '4', 'publication_names', 'pub') WHERE get_byte(data, 0) = 73; }
+session "s3"
+setup { SET synchronous_commit=on; }
+step "s3_begin" { BEGIN; }
+step "s3_insert_tbl1" { INSERT INTO tbl1 (val1, val2) VALUES (2, 2); }
+step "s3_commit" { COMMIT; }
+
# Expect to get one insert change. LOGICAL_REP_MSG_INSERT = 'I'
permutation "s1_insert_tbl1" "s1_begin" "s1_insert_tbl1" "s2_alter_pub_add_tbl" "s1_commit" "s1_insert_tbl1" "s2_get_binary_changes"
+
+# Expect to get one insert change with LOGICAL_REP_MSG_INSERT = 'I' from
+# the second "s1_insert_tbl1" executed after adding the table tbl1 to the
+# publication in "s2_alter_pub_add_tbl".
+permutation "s1_begin" "s1_insert_tbl1" "s3_begin" "s3_insert_tbl1" "s2_alter_pub_add_tbl" "s1_insert_tbl1" "s1_commit" "s3_commit" "s2_get_binary_changes"
diff --git a/src/backend/replication/logical/reorderbuffer.c b/src/backend/replication/logical/reorderbuffer.c
index 03eb005c39d..d85a9f2bb20 100644
--- a/src/backend/replication/logical/reorderbuffer.c
+++ b/src/backend/replication/logical/reorderbuffer.c
@@ -108,10 +108,22 @@
#include "storage/fd.h"
#include "storage/sinval.h"
#include "utils/builtins.h"
+#include "utils/inval.h"
#include "utils/memutils.h"
#include "utils/rel.h"
#include "utils/relfilenumbermap.h"
+/*
+ * Each transaction has an 8MB limit for invalidation messages distributed from
+ * other transactions. This limit is set considering scenarios with many
+ * concurrent logical decoding operations. When the distributed invalidation
+ * messages reach this threshold, the transaction is marked as
+ * RBTXN_DISTR_INVAL_OVERFLOWED to invalidate the complete cache as we have lost
+ * some inval messages and hence don't know what needs to be invalidated.
+ */
+#define MAX_DISTR_INVAL_MSG_PER_TXN \
+ ((8 * 1024 * 1024) / sizeof(SharedInvalidationMessage))
+
/* entry for a hash table we use to map from xid to our transaction state */
typedef struct ReorderBufferTXNByIdEnt
{
@@ -469,6 +481,12 @@ ReorderBufferReturnTXN(ReorderBuffer *rb, ReorderBufferTXN *txn)
txn->invalidations = NULL;
}
+ if (txn->invalidations_distributed)
+ {
+ pfree(txn->invalidations_distributed);
+ txn->invalidations_distributed = NULL;
+ }
+
/* Reset the toast hash */
ReorderBufferToastReset(rb, txn);
@@ -2574,7 +2592,17 @@ ReorderBufferProcessTXN(ReorderBuffer *rb, ReorderBufferTXN *txn,
AbortCurrentTransaction();
/* make sure there's no cache pollution */
- ReorderBufferExecuteInvalidations(txn->ninvalidations, txn->invalidations);
+ if (rbtxn_distr_inval_overflowed(txn))
+ {
+ Assert(txn->ninvalidations_distributed == 0);
+ InvalidateSystemCaches();
+ }
+ else
+ {
+ ReorderBufferExecuteInvalidations(txn->ninvalidations, txn->invalidations);
+ ReorderBufferExecuteInvalidations(txn->ninvalidations_distributed,
+ txn->invalidations_distributed);
+ }
if (using_subtxn)
RollbackAndReleaseCurrentSubTransaction();
@@ -2620,8 +2648,17 @@ ReorderBufferProcessTXN(ReorderBuffer *rb, ReorderBufferTXN *txn,
AbortCurrentTransaction();
/* make sure there's no cache pollution */
- ReorderBufferExecuteInvalidations(txn->ninvalidations,
- txn->invalidations);
+ if (rbtxn_distr_inval_overflowed(txn))
+ {
+ Assert(txn->ninvalidations_distributed == 0);
+ InvalidateSystemCaches();
+ }
+ else
+ {
+ ReorderBufferExecuteInvalidations(txn->ninvalidations, txn->invalidations);
+ ReorderBufferExecuteInvalidations(txn->ninvalidations_distributed,
+ txn->invalidations_distributed);
+ }
if (using_subtxn)
RollbackAndReleaseCurrentSubTransaction();
@@ -2951,7 +2988,8 @@ ReorderBufferAbort(ReorderBuffer *rb, TransactionId xid, XLogRecPtr lsn,
* We might have decoded changes for this transaction that could load
* the cache as per the current transaction's view (consider DDL's
* happened in this transaction). We don't want the decoding of future
- * transactions to use those cache entries so execute invalidations.
+ * transactions to use those cache entries so execute only the inval
+ * messages in this transaction.
*/
if (txn->ninvalidations > 0)
ReorderBufferImmediateInvalidation(rb, txn->ninvalidations,
@@ -3038,9 +3076,10 @@ ReorderBufferForget(ReorderBuffer *rb, TransactionId xid, XLogRecPtr lsn)
txn->final_lsn = lsn;
/*
- * Process cache invalidation messages if there are any. Even if we're not
- * interested in the transaction's contents, it could have manipulated the
- * catalog and we need to update the caches according to that.
+ * Process only cache invalidation messages in this transaction if there
+ * are any. Even if we're not interested in the transaction's contents, it
+ * could have manipulated the catalog and we need to update the caches
+ * according to that.
*/
if (txn->base_snapshot != NULL && txn->ninvalidations > 0)
ReorderBufferImmediateInvalidation(rb, txn->ninvalidations,
@@ -3312,6 +3351,57 @@ ReorderBufferAddNewTupleCids(ReorderBuffer *rb, TransactionId xid,
txn->ntuplecids++;
}
+/*
+ * Add new invalidation messages to the reorder buffer queue.
+ */
+static void
+ReorderBufferQueueInvalidations(ReorderBuffer *rb, TransactionId xid,
+ XLogRecPtr lsn, Size nmsgs,
+ SharedInvalidationMessage *msgs)
+{
+ ReorderBufferChange *change;
+
+ change = ReorderBufferGetChange(rb);
+ change->action = REORDER_BUFFER_CHANGE_INVALIDATION;
+ change->data.inval.ninvalidations = nmsgs;
+ change->data.inval.invalidations = (SharedInvalidationMessage *)
+ palloc(sizeof(SharedInvalidationMessage) * nmsgs);
+ memcpy(change->data.inval.invalidations, msgs,
+ sizeof(SharedInvalidationMessage) * nmsgs);
+
+ ReorderBufferQueueChange(rb, xid, lsn, change, false);
+}
+
+/*
+ * A helper function for ReorderBufferAddInvalidations() and
+ * ReorderBufferAddDistributedInvalidations() to accumulate the invalidation
+ * messages to the **invals_out.
+ */
+static void
+ReorderBufferAccumulateInvalidations(SharedInvalidationMessage **invals_out,
+ uint32 *ninvals_out,
+ SharedInvalidationMessage *msgs_new,
+ Size nmsgs_new)
+{
+ if (*ninvals_out == 0)
+ {
+ *ninvals_out = nmsgs_new;
+ *invals_out = (SharedInvalidationMessage *)
+ palloc(sizeof(SharedInvalidationMessage) * nmsgs_new);
+ memcpy(*invals_out, msgs_new, sizeof(SharedInvalidationMessage) * nmsgs_new);
+ }
+ else
+ {
+ /* Enlarge the array of inval messages */
+ *invals_out = (SharedInvalidationMessage *)
+ repalloc(*invals_out, sizeof(SharedInvalidationMessage) *
+ (*ninvals_out + nmsgs_new));
+ memcpy(*invals_out + *ninvals_out, msgs_new,
+ nmsgs_new * sizeof(SharedInvalidationMessage));
+ *ninvals_out += nmsgs_new;
+ }
+}
+
/*
* Accumulate the invalidations for executing them later.
*
@@ -3332,7 +3422,6 @@ ReorderBufferAddInvalidations(ReorderBuffer *rb, TransactionId xid,
{
ReorderBufferTXN *txn;
MemoryContext oldcontext;
- ReorderBufferChange *change;
txn = ReorderBufferTXNByXid(rb, xid, true, NULL, lsn, true);
@@ -3347,35 +3436,76 @@ ReorderBufferAddInvalidations(ReorderBuffer *rb, TransactionId xid,
Assert(nmsgs > 0);
- /* Accumulate invalidations. */
- if (txn->ninvalidations == 0)
- {
- txn->ninvalidations = nmsgs;
- txn->invalidations = (SharedInvalidationMessage *)
- palloc(sizeof(SharedInvalidationMessage) * nmsgs);
- memcpy(txn->invalidations, msgs,
- sizeof(SharedInvalidationMessage) * nmsgs);
- }
- else
+ ReorderBufferAccumulateInvalidations(&txn->invalidations,
+ &txn->ninvalidations,
+ msgs, nmsgs);
+
+ ReorderBufferQueueInvalidations(rb, xid, lsn, nmsgs, msgs);
+
+ MemoryContextSwitchTo(oldcontext);
+}
+
+/*
+ * Accumulate the invalidations distributed by other committed transactions
+ * for executing them later.
+ *
+ * This function is similar to ReorderBufferAddInvalidations() but stores
+ * the given inval messages to the txn->invalidations_distributed with the
+ * overflow check.
+ *
+ * This needs to be called by committed transactions to distribute their
+ * inval messages to in-progress transactions.
+ */
+void
+ReorderBufferAddDistributedInvalidations(ReorderBuffer *rb, TransactionId xid,
+ XLogRecPtr lsn, Size nmsgs,
+ SharedInvalidationMessage *msgs)
+{
+ ReorderBufferTXN *txn;
+ MemoryContext oldcontext;
+
+ txn = ReorderBufferTXNByXid(rb, xid, true, NULL, lsn, true);
+
+ oldcontext = MemoryContextSwitchTo(rb->context);
+
+ /*
+ * Collect all the invalidations under the top transaction, if available,
+ * so that we can execute them all together. See comments
+ * ReorderBufferAddInvalidations.
+ */
+ txn = rbtxn_get_toptxn(txn);
+
+ Assert(nmsgs > 0);
+
+ if (!rbtxn_distr_inval_overflowed(txn))
{
- txn->invalidations = (SharedInvalidationMessage *)
- repalloc(txn->invalidations, sizeof(SharedInvalidationMessage) *
- (txn->ninvalidations + nmsgs));
+ /*
+ * Check the transaction has enough space for storing distributed
+ * invalidation messages.
+ */
+ if (txn->ninvalidations_distributed + nmsgs >= MAX_DISTR_INVAL_MSG_PER_TXN)
+ {
+ /*
+ * Mark the invalidation message as overflowed and free up the
+ * messages accumulated so far.
+ */
+ txn->txn_flags |= RBTXN_DISTR_INVAL_OVERFLOWED;
- memcpy(txn->invalidations + txn->ninvalidations, msgs,
- nmsgs * sizeof(SharedInvalidationMessage));
- txn->ninvalidations += nmsgs;
+ if (txn->invalidations_distributed)
+ {
+ pfree(txn->invalidations_distributed);
+ txn->invalidations_distributed = NULL;
+ txn->ninvalidations_distributed = 0;
+ }
+ }
+ else
+ ReorderBufferAccumulateInvalidations(&txn->invalidations_distributed,
+ &txn->ninvalidations_distributed,
+ msgs, nmsgs);
}
- change = ReorderBufferGetChange(rb);
- change->action = REORDER_BUFFER_CHANGE_INVALIDATION;
- change->data.inval.ninvalidations = nmsgs;
- change->data.inval.invalidations = (SharedInvalidationMessage *)
- palloc(sizeof(SharedInvalidationMessage) * nmsgs);
- memcpy(change->data.inval.invalidations, msgs,
- sizeof(SharedInvalidationMessage) * nmsgs);
-
- ReorderBufferQueueChange(rb, xid, lsn, change, false);
+ /* Queue the invalidation messages into the transaction */
+ ReorderBufferQueueInvalidations(rb, xid, lsn, nmsgs, msgs);
MemoryContextSwitchTo(oldcontext);
}
diff --git a/src/backend/replication/logical/snapbuild.c b/src/backend/replication/logical/snapbuild.c
index 110e0b0a044..5c3bbf0e93f 100644
--- a/src/backend/replication/logical/snapbuild.c
+++ b/src/backend/replication/logical/snapbuild.c
@@ -932,6 +932,13 @@ SnapBuildDistributeSnapshotAndInval(SnapBuild *builder, XLogRecPtr lsn, Transact
* contents built by the current transaction even after its decoding,
* which should have been invalidated due to concurrent catalog
* changing transaction.
+ *
+ * Distribute only the invalidation messages generated by the current
+ * committed transaction. Invalidation messages received from other
+ * transactions would have already been propagated to the relevant
+ * in-progress transactions. This transaction would have processed
+ * those invalidations, ensuring that subsequent transactions observe
+ * a consistent cache state.
*/
if (txn->xid != xid)
{
@@ -945,8 +952,9 @@ SnapBuildDistributeSnapshotAndInval(SnapBuild *builder, XLogRecPtr lsn, Transact
{
Assert(msgs != NULL);
- ReorderBufferAddInvalidations(builder->reorder, txn->xid, lsn,
- ninvalidations, msgs);
+ ReorderBufferAddDistributedInvalidations(builder->reorder,
+ txn->xid, lsn,
+ ninvalidations, msgs);
}
}
}
diff --git a/src/include/replication/reorderbuffer.h b/src/include/replication/reorderbuffer.h
index 4c56f219fd8..0cfa9005141 100644
--- a/src/include/replication/reorderbuffer.h
+++ b/src/include/replication/reorderbuffer.h
@@ -159,15 +159,16 @@ typedef struct ReorderBufferChange
} ReorderBufferChange;
/* ReorderBufferTXN txn_flags */
-#define RBTXN_HAS_CATALOG_CHANGES 0x0001
-#define RBTXN_IS_SUBXACT 0x0002
-#define RBTXN_IS_SERIALIZED 0x0004
-#define RBTXN_IS_SERIALIZED_CLEAR 0x0008
-#define RBTXN_IS_STREAMED 0x0010
-#define RBTXN_HAS_PARTIAL_CHANGE 0x0020
-#define RBTXN_PREPARE 0x0040
-#define RBTXN_SKIPPED_PREPARE 0x0080
-#define RBTXN_HAS_STREAMABLE_CHANGE 0x0100
+#define RBTXN_HAS_CATALOG_CHANGES 0x0001
+#define RBTXN_IS_SUBXACT 0x0002
+#define RBTXN_IS_SERIALIZED 0x0004
+#define RBTXN_IS_SERIALIZED_CLEAR 0x0008
+#define RBTXN_IS_STREAMED 0x0010
+#define RBTXN_HAS_PARTIAL_CHANGE 0x0020
+#define RBTXN_PREPARE 0x0040
+#define RBTXN_SKIPPED_PREPARE 0x0080
+#define RBTXN_HAS_STREAMABLE_CHANGE 0x0100
+#define RBTXN_DISTR_INVAL_OVERFLOWED 0x0200
/* Does the transaction have catalog changes? */
#define rbtxn_has_catalog_changes(txn) \
@@ -231,6 +232,12 @@ typedef struct ReorderBufferChange
((txn)->txn_flags & RBTXN_SKIPPED_PREPARE) != 0 \
)
+/* Is the array of distributed inval messages overflowed? */
+#define rbtxn_distr_inval_overflowed(txn) \
+( \
+ ((txn)->txn_flags & RBTXN_DISTR_INVAL_OVERFLOWED) != 0 \
+)
+
/* Is this a top-level transaction? */
#define rbtxn_is_toptxn(txn) \
( \
@@ -422,6 +429,12 @@ typedef struct ReorderBufferTXN
* Private data pointer of the output plugin.
*/
void *output_plugin_private;
+
+ /*
+ * Stores cache invalidation messages distributed by other transactions.
+ */
+ uint32 ninvalidations_distributed;
+ SharedInvalidationMessage *invalidations_distributed;
} ReorderBufferTXN;
/* so we can define the callbacks used inside struct ReorderBuffer itself */
@@ -709,6 +722,9 @@ extern void ReorderBufferAddNewTupleCids(ReorderBuffer *rb, TransactionId xid,
CommandId cmin, CommandId cmax, CommandId combocid);
extern void ReorderBufferAddInvalidations(ReorderBuffer *rb, TransactionId xid, XLogRecPtr lsn,
Size nmsgs, SharedInvalidationMessage *msgs);
+extern void ReorderBufferAddDistributedInvalidations(ReorderBuffer *rb, TransactionId xid,
+ XLogRecPtr lsn, Size nmsgs,
+ SharedInvalidationMessage *msgs);
extern void ReorderBufferImmediateInvalidation(ReorderBuffer *rb, uint32 ninvalidations,
SharedInvalidationMessage *invalidations);
extern void ReorderBufferProcessXid(ReorderBuffer *rb, TransactionId xid, XLogRecPtr lsn);
--
2.43.5
[application/octet-stream] REL13_v15-0001-Fix-re-distributing-previously-distributed-inval.patch (17.8K, ../../CAD21AoCHqKXVhUZbxPKBve-uEHJWKxV9SVC5deiB5VUONTH=6w@mail.gmail.com/7-REL13_v15-0001-Fix-re-distributing-previously-distributed-inval.patch)
download | inline diff:
From 6beccb3345693f4416c3c1d6111655cf42cf146f Mon Sep 17 00:00:00 2001
From: Vignesh C <vignesh21@gmail.com>
Date: Thu, 12 Jun 2025 11:46:35 +0530
Subject: [PATCH v15] Fix re-distributing previously distributed invalidation
messages during logical decoding.
Commit 4909b38af0 introduced logic to distribute invalidation messages
from catalog-modifying transactions to all concurrent in-progress
transactions. However, since each transaction distributes not only its
original invalidation messages but also previously distributed
messages to other transactions, this leads to an exponential increase
in allocation request size for invalidation messages, ultimately
causing memory allocation failure.
This commit fixes this issue by tracking distributed invalidation
messages separately per decoded transaction and not redistributing
these messages to other in-progress transactions. The maximum size of
distributed invalidation messages that one transaction can store is
limited to MAX_DISTR_INVAL_MSG_PER_TXN (8MB). Once the size of the
distributed invalidation messages exceeds this threshold, we
invalidate all caches in locations where distributed invalidation
messages need to be executed.
Back-patch to all supported versions where we introduced the fix by
commit 4909b38af0.
Note that this commit adds two new fields to ReorderBufferTXN to store
the distributed transactions. This change breaks ABI compatibility in
back branches, affecting third-party extensions that depend on the
size of the ReorderBufferTXN struct, though this scenario seems
unlikely.
Additionally, it adds a new flag to the txn_flags field of
ReorderBufferTXN to indicate distributed invalidation message
overflow. This should not affect existing implementations, as it is
unlikely that third-party extensions use unused bits in the txn_flags
field.
Bug: #18938 #18942
Author: vignesh C <vignesh21@gmail.com>
Reported-by: Duncan Sands <duncan.sands@deepbluecap.com>
Reported-by: John Hutchins <john.hutchins@wicourts.gov>
Reported-by: Laurence Parry <greenreaper@hotmail.com>
Reported-by: Max Madden <maxmmadden@gmail.com>
Reported-by: Braulio Fdo Gonzalez <brauliofg@gmail.com>
Reviewed-by: Masahiko Sawada <sawada.mshk@gmail.com>
Reviewed-by: Amit Kapila <amit.kapila16@gmail.com>
Reviewed-by: Hayato Kuroda <kuroda.hayato@fujitsu.com>
Discussion: https://postgr.es/m/680bdaf6-f7d1-4536-b580-05c2760c67c6@deepbluecap.com
Discussion: https://postgr.es/m/18942-0ab1e5ae156613ad@postgresql.org
Discussion: https://postgr.es/m/18938-57c9a1c463b68ce0@postgresql.org
Discussion: https://postgr.es/m/CAD1FGCT2sYrP_70RTuo56QTizyc+J3wJdtn2gtO3VttQFpdMZg@mail.gmail.com
Discussion: https://postgr.es/m/CANO2=B=2BT1hSYCE=nuuTnVTnjidMg0+-FfnRnqM6kd23qoygg@mail.gmail.com
Backpatch-through: 13
---
.../expected/invalidation_distribution.out | 23 ++-
.../specs/invalidation_distribution.spec | 11 ++
.../replication/logical/reorderbuffer.c | 169 +++++++++++++++---
src/backend/replication/logical/snapbuild.c | 12 +-
src/include/replication/reorderbuffer.h | 22 ++-
5 files changed, 207 insertions(+), 30 deletions(-)
diff --git a/contrib/test_decoding/expected/invalidation_distribution.out b/contrib/test_decoding/expected/invalidation_distribution.out
index eb70eda9042..a78ea1a26c5 100644
--- a/contrib/test_decoding/expected/invalidation_distribution.out
+++ b/contrib/test_decoding/expected/invalidation_distribution.out
@@ -1,4 +1,4 @@
-Parsed test spec with 2 sessions
+Parsed test spec with 3 sessions
starting permutation: s1_insert_tbl1 s1_begin s1_insert_tbl1 s2_alter_pub_add_tbl s1_commit s1_insert_tbl1 s2_get_binary_changes
step s1_insert_tbl1: INSERT INTO tbl1 (val1, val2) VALUES (1, 1);
@@ -18,3 +18,24 @@ count
stop
(1 row)
+
+starting permutation: s1_begin s1_insert_tbl1 s3_begin s3_insert_tbl1 s2_alter_pub_add_tbl s1_insert_tbl1 s1_commit s3_commit s2_get_binary_changes
+step s1_begin: BEGIN;
+step s1_insert_tbl1: INSERT INTO tbl1 (val1, val2) VALUES (1, 1);
+step s3_begin: BEGIN;
+step s3_insert_tbl1: INSERT INTO tbl1 (val1, val2) VALUES (2, 2);
+step s2_alter_pub_add_tbl: ALTER PUBLICATION pub ADD TABLE tbl1;
+step s1_insert_tbl1: INSERT INTO tbl1 (val1, val2) VALUES (1, 1);
+step s1_commit: COMMIT;
+step s3_commit: COMMIT;
+step s2_get_binary_changes: SELECT count(data) FROM pg_logical_slot_get_binary_changes('isolation_slot', NULL, NULL, 'proto_version', '1', 'publication_names', 'pub') WHERE get_byte(data, 0) = 73;
+count
+-----
+ 0
+(1 row)
+
+?column?
+--------
+stop
+(1 row)
+
diff --git a/contrib/test_decoding/specs/invalidation_distribution.spec b/contrib/test_decoding/specs/invalidation_distribution.spec
index ca051fc1e85..c5977b8c483 100644
--- a/contrib/test_decoding/specs/invalidation_distribution.spec
+++ b/contrib/test_decoding/specs/invalidation_distribution.spec
@@ -28,5 +28,16 @@ setup { SET synchronous_commit=on; }
step "s2_alter_pub_add_tbl" { ALTER PUBLICATION pub ADD TABLE tbl1; }
step "s2_get_binary_changes" { SELECT count(data) FROM pg_logical_slot_get_binary_changes('isolation_slot', NULL, NULL, 'proto_version', '1', 'publication_names', 'pub') WHERE get_byte(data, 0) = 73; }
+session "s3"
+setup { SET synchronous_commit=on; }
+step "s3_begin" { BEGIN; }
+step "s3_insert_tbl1" { INSERT INTO tbl1 (val1, val2) VALUES (2, 2); }
+step "s3_commit" { COMMIT; }
+
# Expect to get one insert change. LOGICAL_REP_MSG_INSERT = 'I'
permutation "s1_insert_tbl1" "s1_begin" "s1_insert_tbl1" "s2_alter_pub_add_tbl" "s1_commit" "s1_insert_tbl1" "s2_get_binary_changes"
+
+# Expect to get no change because both s1's and s3's transactions
+# use the snapshot from before adding the table tbl1 to the
+# publication by "s2_alter_pub_add_tbl".
+permutation "s1_begin" "s1_insert_tbl1" "s3_begin" "s3_insert_tbl1" "s2_alter_pub_add_tbl" "s1_insert_tbl1" "s1_commit" "s3_commit" "s2_get_binary_changes"
diff --git a/src/backend/replication/logical/reorderbuffer.c b/src/backend/replication/logical/reorderbuffer.c
index fa9413fa2a0..e5fa1d62151 100644
--- a/src/backend/replication/logical/reorderbuffer.c
+++ b/src/backend/replication/logical/reorderbuffer.c
@@ -103,12 +103,24 @@
#include "storage/sinval.h"
#include "utils/builtins.h"
#include "utils/combocid.h"
+#include "utils/inval.h"
#include "utils/memdebug.h"
#include "utils/memutils.h"
#include "utils/rel.h"
#include "utils/relfilenodemap.h"
+/*
+ * Each transaction has an 8MB limit for invalidation messages distributed from
+ * other transactions. This limit is set considering scenarios with many
+ * concurrent logical decoding operations. When the distributed invalidation
+ * messages reach this threshold, the transaction is marked as
+ * RBTXN_DISTR_INVAL_OVERFLOWED to invalidate the complete cache as we have lost
+ * some inval messages and hence don't know what needs to be invalidated.
+ */
+#define MAX_DISTR_INVAL_MSG_PER_TXN \
+ ((8 * 1024 * 1024) / sizeof(SharedInvalidationMessage))
+
/* entry for a hash table we use to map from xid to our transaction state */
typedef struct ReorderBufferTXNByIdEnt
{
@@ -220,7 +232,8 @@ static void ReorderBufferIterTXNInit(ReorderBuffer *rb, ReorderBufferTXN *txn,
static ReorderBufferChange *ReorderBufferIterTXNNext(ReorderBuffer *rb, ReorderBufferIterTXNState *state);
static void ReorderBufferIterTXNFinish(ReorderBuffer *rb,
ReorderBufferIterTXNState *state);
-static void ReorderBufferExecuteInvalidations(ReorderBuffer *rb, ReorderBufferTXN *txn);
+static void ReorderBufferExecuteInvalidations(uint32 nmsgs,
+ SharedInvalidationMessage *msgs);
/*
* ---------------------------------------
@@ -406,6 +419,12 @@ ReorderBufferReturnTXN(ReorderBuffer *rb, ReorderBufferTXN *txn)
txn->invalidations = NULL;
}
+ if (txn->invalidations_distributed)
+ {
+ pfree(txn->invalidations_distributed);
+ txn->invalidations_distributed = NULL;
+ }
+
/* Reset the toast hash */
ReorderBufferToastReset(rb, txn);
@@ -1883,7 +1902,8 @@ ReorderBufferCommit(ReorderBuffer *rb, TransactionId xid,
* see new catalog contents, so execute all
* invalidations.
*/
- ReorderBufferExecuteInvalidations(rb, txn);
+ ReorderBufferExecuteInvalidations(txn->ninvalidations,
+ txn->invalidations);
}
break;
@@ -1921,7 +1941,17 @@ ReorderBufferCommit(ReorderBuffer *rb, TransactionId xid,
AbortCurrentTransaction();
/* make sure there's no cache pollution */
- ReorderBufferExecuteInvalidations(rb, txn);
+ if (rbtxn_distr_inval_overflowed(txn))
+ {
+ Assert(txn->ninvalidations_distributed == 0);
+ InvalidateSystemCaches();
+ }
+ else
+ {
+ ReorderBufferExecuteInvalidations(txn->ninvalidations, txn->invalidations);
+ ReorderBufferExecuteInvalidations(txn->ninvalidations_distributed,
+ txn->invalidations_distributed);
+ }
if (using_subtxn)
RollbackAndReleaseCurrentSubTransaction();
@@ -1947,7 +1977,17 @@ ReorderBufferCommit(ReorderBuffer *rb, TransactionId xid,
AbortCurrentTransaction();
/* make sure there's no cache pollution */
- ReorderBufferExecuteInvalidations(rb, txn);
+ if (rbtxn_distr_inval_overflowed(txn))
+ {
+ Assert(txn->ninvalidations_distributed == 0);
+ InvalidateSystemCaches();
+ }
+ else
+ {
+ ReorderBufferExecuteInvalidations(txn->ninvalidations, txn->invalidations);
+ ReorderBufferExecuteInvalidations(txn->ninvalidations_distributed,
+ txn->invalidations_distributed);
+ }
if (using_subtxn)
RollbackAndReleaseCurrentSubTransaction();
@@ -2060,9 +2100,10 @@ ReorderBufferForget(ReorderBuffer *rb, TransactionId xid, XLogRecPtr lsn)
txn->final_lsn = lsn;
/*
- * Process cache invalidation messages if there are any. Even if we're not
- * interested in the transaction's contents, it could have manipulated the
- * catalog and we need to update the caches according to that.
+ * Process only cache invalidation messages in this transaction if there
+ * are any. Even if we're not interested in the transaction's contents, it
+ * could have manipulated the catalog and we need to update the caches
+ * according to that.
*/
if (txn->base_snapshot != NULL && txn->ninvalidations > 0)
ReorderBufferImmediateInvalidation(rb, txn->ninvalidations,
@@ -2253,6 +2294,36 @@ ReorderBufferAddNewTupleCids(ReorderBuffer *rb, TransactionId xid,
txn->ntuplecids++;
}
+/*
+ * A helper function for ReorderBufferAddInvalidations() and
+ * ReorderBufferAddDistributedInvalidations() to accumulate the invalidation
+ * messages to the **invals_out.
+ */
+static void
+ReorderBufferAccumulateInvalidations(SharedInvalidationMessage **invals_out,
+ uint32 *ninvals_out,
+ SharedInvalidationMessage *msgs_new,
+ Size nmsgs_new)
+{
+ if (*ninvals_out == 0)
+ {
+ *ninvals_out = nmsgs_new;
+ *invals_out = (SharedInvalidationMessage *)
+ palloc(sizeof(SharedInvalidationMessage) * nmsgs_new);
+ memcpy(*invals_out, msgs_new, sizeof(SharedInvalidationMessage) * nmsgs_new);
+ }
+ else
+ {
+ /* Enlarge the array of inval messages */
+ *invals_out = (SharedInvalidationMessage *)
+ repalloc(*invals_out, sizeof(SharedInvalidationMessage) *
+ (*ninvals_out + nmsgs_new));
+ memcpy(*invals_out + *ninvals_out, msgs_new,
+ nmsgs_new * sizeof(SharedInvalidationMessage));
+ *ninvals_out += nmsgs_new;
+ }
+}
+
/*
* Setup the invalidation of the toplevel transaction.
*
@@ -2282,24 +2353,74 @@ ReorderBufferAddInvalidations(ReorderBuffer *rb, TransactionId xid,
Assert(nmsgs > 0);
- /* Accumulate invalidations. */
- if (txn->ninvalidations == 0)
+ ReorderBufferAccumulateInvalidations(&txn->invalidations,
+ &txn->ninvalidations,
+ msgs, nmsgs);
+
+ MemoryContextSwitchTo(oldcontext);
+}
+
+
+/*
+ * Accumulate the invalidations distributed by other committed transactions
+ * for executing them later.
+ *
+ * This function is similar to ReorderBufferAddInvalidations() but stores
+ * the given inval messages to the txn->invalidations_distributed with the
+ * overflow check.
+ *
+ * This needs to be called by committed transactions to distribute their
+ * inval messages to in-progress transactions.
+ */
+void
+ReorderBufferAddDistributedInvalidations(ReorderBuffer *rb, TransactionId xid,
+ XLogRecPtr lsn, Size nmsgs,
+ SharedInvalidationMessage *msgs)
+{
+ ReorderBufferTXN *txn;
+ MemoryContext oldcontext;
+
+ txn = ReorderBufferTXNByXid(rb, xid, true, NULL, lsn, true);
+
+ oldcontext = MemoryContextSwitchTo(rb->context);
+
+ /*
+ * Collect all the invalidations under the top transaction, if available,
+ * so that we can execute them all together.
+ */
+ if (txn->toplevel_xid)
{
- txn->ninvalidations = nmsgs;
- txn->invalidations = (SharedInvalidationMessage *)
- palloc(sizeof(SharedInvalidationMessage) * nmsgs);
- memcpy(txn->invalidations, msgs,
- sizeof(SharedInvalidationMessage) * nmsgs);
+ txn = ReorderBufferTXNByXid(rb, txn->toplevel_xid, true, NULL, lsn,
+ true);
}
- else
+
+ Assert(nmsgs > 0);
+
+ if (!rbtxn_distr_inval_overflowed(txn))
{
- txn->invalidations = (SharedInvalidationMessage *)
- repalloc(txn->invalidations, sizeof(SharedInvalidationMessage) *
- (txn->ninvalidations + nmsgs));
+ /*
+ * Check the transaction has enough space for storing distributed
+ * invalidation messages.
+ */
+ if (txn->ninvalidations_distributed + nmsgs >= MAX_DISTR_INVAL_MSG_PER_TXN)
+ {
+ /*
+ * Mark the invalidation message as overflowed and free up the
+ * messages accumulated so far.
+ */
+ txn->txn_flags |= RBTXN_DISTR_INVAL_OVERFLOWED;
- memcpy(txn->invalidations + txn->ninvalidations, msgs,
- nmsgs * sizeof(SharedInvalidationMessage));
- txn->ninvalidations += nmsgs;
+ if (txn->invalidations_distributed)
+ {
+ pfree(txn->invalidations_distributed);
+ txn->invalidations_distributed = NULL;
+ txn->ninvalidations_distributed = 0;
+ }
+ }
+ else
+ ReorderBufferAccumulateInvalidations(&txn->invalidations_distributed,
+ &txn->ninvalidations_distributed,
+ msgs, nmsgs);
}
MemoryContextSwitchTo(oldcontext);
@@ -2310,12 +2431,12 @@ ReorderBufferAddInvalidations(ReorderBuffer *rb, TransactionId xid,
* in the changestream but we don't know which those are.
*/
static void
-ReorderBufferExecuteInvalidations(ReorderBuffer *rb, ReorderBufferTXN *txn)
+ReorderBufferExecuteInvalidations(uint32 nmsgs, SharedInvalidationMessage *msgs)
{
int i;
- for (i = 0; i < txn->ninvalidations; i++)
- LocalExecuteInvalidationMessage(&txn->invalidations[i]);
+ for (i = 0; i < nmsgs; i++)
+ LocalExecuteInvalidationMessage(&msgs[i]);
}
/*
diff --git a/src/backend/replication/logical/snapbuild.c b/src/backend/replication/logical/snapbuild.c
index 3bda41c5251..69b083a36fe 100644
--- a/src/backend/replication/logical/snapbuild.c
+++ b/src/backend/replication/logical/snapbuild.c
@@ -927,6 +927,13 @@ SnapBuildDistributeSnapshotAndInval(SnapBuild *builder, XLogRecPtr lsn, Transact
* contents built by the current transaction even after its decoding,
* which should have been invalidated due to concurrent catalog
* changing transaction.
+ *
+ * Distribute only the invalidation messages generated by the current
+ * committed transaction. Invalidation messages received from other
+ * transactions would have already been propagated to the relevant
+ * in-progress transactions. This transaction would have processed
+ * those invalidations, ensuring that subsequent transactions observe
+ * a consistent cache state.
*/
if (txn->xid != xid)
{
@@ -940,8 +947,9 @@ SnapBuildDistributeSnapshotAndInval(SnapBuild *builder, XLogRecPtr lsn, Transact
{
Assert(msgs != NULL);
- ReorderBufferAddInvalidations(builder->reorder, txn->xid, lsn,
- ninvalidations, msgs);
+ ReorderBufferAddDistributedInvalidations(builder->reorder,
+ txn->xid, lsn,
+ ninvalidations, msgs);
}
}
}
diff --git a/src/include/replication/reorderbuffer.h b/src/include/replication/reorderbuffer.h
index 545cee891ed..2321bb0b406 100644
--- a/src/include/replication/reorderbuffer.h
+++ b/src/include/replication/reorderbuffer.h
@@ -160,9 +160,10 @@ typedef struct ReorderBufferChange
} ReorderBufferChange;
/* ReorderBufferTXN txn_flags */
-#define RBTXN_HAS_CATALOG_CHANGES 0x0001
-#define RBTXN_IS_SUBXACT 0x0002
-#define RBTXN_IS_SERIALIZED 0x0004
+#define RBTXN_HAS_CATALOG_CHANGES 0x0001
+#define RBTXN_IS_SUBXACT 0x0002
+#define RBTXN_IS_SERIALIZED 0x0004
+#define RBTXN_DISTR_INVAL_OVERFLOWED 0x0008
/* Does the transaction have catalog changes? */
#define rbtxn_has_catalog_changes(txn) \
@@ -182,6 +183,12 @@ typedef struct ReorderBufferChange
((txn)->txn_flags & RBTXN_IS_SERIALIZED) != 0 \
)
+/* Is the array of distributed inval messages overflowed? */
+#define rbtxn_distr_inval_overflowed(txn) \
+( \
+ ((txn)->txn_flags & RBTXN_DISTR_INVAL_OVERFLOWED) != 0 \
+)
+
typedef struct ReorderBufferTXN
{
/* See above */
@@ -311,6 +318,12 @@ typedef struct ReorderBufferTXN
* Size of this transaction (changes currently in memory, in bytes).
*/
Size size;
+
+ /*
+ * Stores cache invalidation messages distributed by other transactions.
+ */
+ uint32 ninvalidations_distributed;
+ SharedInvalidationMessage *invalidations_distributed;
} ReorderBufferTXN;
/* so we can define the callbacks used inside struct ReorderBuffer itself */
@@ -451,6 +464,9 @@ void ReorderBufferAddNewTupleCids(ReorderBuffer *, TransactionId, XLogRecPtr ls
CommandId cmin, CommandId cmax, CommandId combocid);
void ReorderBufferAddInvalidations(ReorderBuffer *, TransactionId, XLogRecPtr lsn,
Size nmsgs, SharedInvalidationMessage *msgs);
+void ReorderBufferAddDistributedInvalidations(ReorderBuffer *rb, TransactionId xid,
+ XLogRecPtr lsn, Size nmsgs,
+ SharedInvalidationMessage *msgs);
void ReorderBufferImmediateInvalidation(ReorderBuffer *, uint32 ninvalidations,
SharedInvalidationMessage *invalidations);
void ReorderBufferProcessXid(ReorderBuffer *, TransactionId xid, XLogRecPtr lsn);
--
2.43.5
^ permalink raw reply [nested|flat] 94+ messages in thread
* Re: Logical replication 'invalid memory alloc request size 1585837200' after upgrading to 17.5
@ 2025-06-16 02:27 vignesh C <vignesh21@gmail.com>
parent: Masahiko Sawada <sawada.mshk@gmail.com>
0 siblings, 1 reply; 94+ messages in thread
From: vignesh C @ 2025-06-16 02:27 UTC (permalink / raw)
To: Masahiko Sawada <sawada.mshk@gmail.com>; +Cc: Amit Kapila <amit.kapila16@gmail.com>; Hayato Kuroda (Fujitsu) <kuroda.hayato@fujitsu.com>; Duncan Sands <duncan.sands@deepbluecap.com>; pgsql-bugs@lists.postgresql.org <pgsql-bugs@lists.postgresql.org>
On Sat, 14 Jun 2025 at 05:22, Masahiko Sawada <sawada.mshk@gmail.com> wrote:
>
> > > Thank you for updating the patch. I have one comment on the newly added test:
> > >
> > > +session "s3"
> > > +step "s3i1" { INSERT INTO tbl1 (val1, val2) VALUES (1, 1);}
> > > +step "s3a" { ALTER PUBLICATION pub ADD TABLE tbl1; }
> > > +step "s3i2" { INSERT INTO tbl1 (val1, val2) VALUES (6, 6); }
> > > +step "s3_get_binary_changes" { SELECT count(data) FROM
> > > pg_logical_slot_get_binary_changes('isolation_slot', NULL, NULL,
> > > 'proto_version', '4', 'publication_names', 'pub') WHERE get_byte(data,
> > > 0) = 73; }
> > > +
> > > +session "s4"
> > > +step "s4b" { BEGIN; }
> > > +step "s4i1" { INSERT INTO tbl1 (val1, val2) VALUES (2, 2);}
> > > +step "s4i2" { INSERT INTO tbl1 (val1, val2) VALUES (4, 4); }
> > > +step "s4c" { COMMIT; }
> > > +step "s4i3" { INSERT INTO tbl1 (val1, val2) VALUES (5, 5); }
> > > +
> > > +session "s5"
> > > +step "s5b" { BEGIN; }
> > > +step "s5i1" { INSERT INTO tbl1 (val1, val2) VALUES (3, 3); }
> > > +step "s5c" { COMMIT; }
> > >
> > > I think we don't necessarily need to add sessions "s4" and "s5". Let's
> > > reuse "s1" and "s2" instead of adding them. I've attached a patch to
> > > change that.
> >
> > Thanks, this is better.
> > In the case of PG13 I have slightly changed the test to do the insert
> > after the commit, because the queuing of invalidation messages into
> > reorder buffer queue is not supported in PG13. This limitation is due
> > to the absence of support for REORDER_BUFFER_CHANGE_INVALIDATION which
> > is already present in >= PG14
> > The attached v14 version patch has the changes for the same.
>
> Hmm, but the modified test is essentially the same as what we already
> have in invalidation_distribution.spec. I think that it's good to have
> the same test case in all branches even though we get a different
> result in v13 as we currently don't have the test to check such
> differences.
>
> I've attached the updated patches accordingly and the commit messages.
Thanks, the changes look good to me.
Regards,
Vignesh
^ permalink raw reply [nested|flat] 94+ messages in thread
* Re: Logical replication 'invalid memory alloc request size 1585837200' after upgrading to 17.5
@ 2025-06-17 17:40 Masahiko Sawada <sawada.mshk@gmail.com>
parent: vignesh C <vignesh21@gmail.com>
0 siblings, 2 replies; 94+ messages in thread
From: Masahiko Sawada @ 2025-06-17 17:40 UTC (permalink / raw)
To: vignesh C <vignesh21@gmail.com>; +Cc: Amit Kapila <amit.kapila16@gmail.com>; Hayato Kuroda (Fujitsu) <kuroda.hayato@fujitsu.com>; Duncan Sands <duncan.sands@deepbluecap.com>; pgsql-bugs@lists.postgresql.org <pgsql-bugs@lists.postgresql.org>
On Sun, Jun 15, 2025 at 7:28 PM vignesh C <vignesh21@gmail.com> wrote:
>
> On Sat, 14 Jun 2025 at 05:22, Masahiko Sawada <sawada.mshk@gmail.com> wrote:
> >
> > > > Thank you for updating the patch. I have one comment on the newly added test:
> > > >
> > > > +session "s3"
> > > > +step "s3i1" { INSERT INTO tbl1 (val1, val2) VALUES (1, 1);}
> > > > +step "s3a" { ALTER PUBLICATION pub ADD TABLE tbl1; }
> > > > +step "s3i2" { INSERT INTO tbl1 (val1, val2) VALUES (6, 6); }
> > > > +step "s3_get_binary_changes" { SELECT count(data) FROM
> > > > pg_logical_slot_get_binary_changes('isolation_slot', NULL, NULL,
> > > > 'proto_version', '4', 'publication_names', 'pub') WHERE get_byte(data,
> > > > 0) = 73; }
> > > > +
> > > > +session "s4"
> > > > +step "s4b" { BEGIN; }
> > > > +step "s4i1" { INSERT INTO tbl1 (val1, val2) VALUES (2, 2);}
> > > > +step "s4i2" { INSERT INTO tbl1 (val1, val2) VALUES (4, 4); }
> > > > +step "s4c" { COMMIT; }
> > > > +step "s4i3" { INSERT INTO tbl1 (val1, val2) VALUES (5, 5); }
> > > > +
> > > > +session "s5"
> > > > +step "s5b" { BEGIN; }
> > > > +step "s5i1" { INSERT INTO tbl1 (val1, val2) VALUES (3, 3); }
> > > > +step "s5c" { COMMIT; }
> > > >
> > > > I think we don't necessarily need to add sessions "s4" and "s5". Let's
> > > > reuse "s1" and "s2" instead of adding them. I've attached a patch to
> > > > change that.
> > >
> > > Thanks, this is better.
> > > In the case of PG13 I have slightly changed the test to do the insert
> > > after the commit, because the queuing of invalidation messages into
> > > reorder buffer queue is not supported in PG13. This limitation is due
> > > to the absence of support for REORDER_BUFFER_CHANGE_INVALIDATION which
> > > is already present in >= PG14
> > > The attached v14 version patch has the changes for the same.
> >
> > Hmm, but the modified test is essentially the same as what we already
> > have in invalidation_distribution.spec. I think that it's good to have
> > the same test case in all branches even though we get a different
> > result in v13 as we currently don't have the test to check such
> > differences.
> >
> > I've attached the updated patches accordingly and the commit messages.
>
> Thanks, the changes look good to me.
Pushed the fix (d87d07b7ad3). Thank you for working on this fix.
Regards,
--
Masahiko Sawada
Amazon Web Services: https://aws.amazon.com
^ permalink raw reply [nested|flat] 94+ messages in thread
* Re: Logical replication 'invalid memory alloc request size 1585837200' after upgrading to 17.5
@ 2025-06-20 05:00 Alexander Lakhin <exclusion@gmail.com>
parent: Masahiko Sawada <sawada.mshk@gmail.com>
1 sibling, 1 reply; 94+ messages in thread
From: Alexander Lakhin @ 2025-06-20 05:00 UTC (permalink / raw)
To: Masahiko Sawada <sawada.mshk@gmail.com>; vignesh C <vignesh21@gmail.com>; +Cc: Amit Kapila <amit.kapila16@gmail.com>; Hayato Kuroda (Fujitsu) <kuroda.hayato@fujitsu.com>; Duncan Sands <duncan.sands@deepbluecap.com>; pgsql-bugs@lists.postgresql.org <pgsql-bugs@lists.postgresql.org>
Hello Sawada-san,
17.06.2025 20:40, Masahiko Sawada wrote:
> Pushed the fix (d87d07b7ad3). Thank you for working on this fix.
As buildfarm [1] shows, the test case added to invalidation_distribution
with 1230be12f fails against -DCLOBBER_CACHE_ALWAYS on REL_13_STABLE:
diff -U3
/home/buildfarm/trilobite/buildroot/REL_13_STABLE/pgsql.build/contrib/test_decoding/expected/invalidation_distribution.out
/home/buildfarm/trilobite/buildroot/REL_13_STABLE/pgsql.build/contrib/test_decoding/output_iso/results/invalidation_distribution.out
---
/home/buildfarm/trilobite/buildroot/REL_13_STABLE/pgsql.build/contrib/test_decoding/expected/invalidation_distribution.out
2025-06-17 10:24:24.382768613 +0200
+++
/home/buildfarm/trilobite/buildroot/REL_13_STABLE/pgsql.build/contrib/test_decoding/output_iso/results/invalidation_distribution.out
2025-06-17 15:01:53.921913314 +0200
@@ -31,7 +31,7 @@
step s2_get_binary_changes: SELECT count(data) FROM pg_logical_slot_get_binary_changes('isolation_slot', NULL, NULL,
'proto_version', '1', 'publication_names', 'pub') WHERE get_byte(data, 0) = 73;
count
-----
- 0
+ 1
(1 row)
?column?
I could reproduce this locally and also checked that the test passes on
REL_14_STABLE and master.
Could you look at this, please?
[1] https://buildfarm.postgresql.org/cgi-bin/show_log.pl?nm=trilobite&dt=2025-06-17%2008%3A24%3A00
Best regards,
Alexander
^ permalink raw reply [nested|flat] 94+ messages in thread
* Re: Logical replication 'invalid memory alloc request size 1585837200' after upgrading to 17.5
@ 2025-06-20 06:04 vignesh C <vignesh21@gmail.com>
parent: Alexander Lakhin <exclusion@gmail.com>
0 siblings, 1 reply; 94+ messages in thread
From: vignesh C @ 2025-06-20 06:04 UTC (permalink / raw)
To: Alexander Lakhin <exclusion@gmail.com>; +Cc: Masahiko Sawada <sawada.mshk@gmail.com>; Amit Kapila <amit.kapila16@gmail.com>; Hayato Kuroda (Fujitsu) <kuroda.hayato@fujitsu.com>; Duncan Sands <duncan.sands@deepbluecap.com>; pgsql-bugs@lists.postgresql.org <pgsql-bugs@lists.postgresql.org>
On Fri, 20 Jun 2025 at 10:30, Alexander Lakhin <exclusion@gmail.com> wrote:
>
> Hello Sawada-san,
>
> 17.06.2025 20:40, Masahiko Sawada wrote:
>
> Pushed the fix (d87d07b7ad3). Thank you for working on this fix.
>
>
> As buildfarm [1] shows, the test case added to invalidation_distribution
> with 1230be12f fails against -DCLOBBER_CACHE_ALWAYS on REL_13_STABLE:
> diff -U3 /home/buildfarm/trilobite/buildroot/REL_13_STABLE/pgsql.build/contrib/test_decoding/expected/invalidation_distribution.out /home/buildfarm/trilobite/buildroot/REL_13_STABLE/pgsql.build/contrib/test_decoding/output_iso/results/invalidation_distribution.out
> --- /home/buildfarm/trilobite/buildroot/REL_13_STABLE/pgsql.build/contrib/test_decoding/expected/invalidation_distribution.out 2025-06-17 10:24:24.382768613 +0200
> +++ /home/buildfarm/trilobite/buildroot/REL_13_STABLE/pgsql.build/contrib/test_decoding/output_iso/results/invalidation_distribution.out 2025-06-17 15:01:53.921913314 +0200
> @@ -31,7 +31,7 @@
> step s2_get_binary_changes: SELECT count(data) FROM pg_logical_slot_get_binary_changes('isolation_slot', NULL, NULL, 'proto_version', '1', 'publication_names', 'pub') WHERE get_byte(data, 0) = 73;
> count
> -----
> - 0
> + 1
> (1 row)
>
> ?column?
>
> I could reproduce this locally and also checked that the test passes on
> REL_14_STABLE and master.
>
> Could you look at this, please?
Thanks for reporting this, we will analyse this and provide a fix for this.
Regards,
Vignesh
^ permalink raw reply [nested|flat] 94+ messages in thread
* Re: Logical replication 'invalid memory alloc request size 1585837200' after upgrading to 17.5
@ 2025-06-20 07:48 Duncan Sands <duncan.sands@deepbluecap.com>
parent: Masahiko Sawada <sawada.mshk@gmail.com>
1 sibling, 0 replies; 94+ messages in thread
From: Duncan Sands @ 2025-06-20 07:48 UTC (permalink / raw)
To: Masahiko Sawada <sawada.mshk@gmail.com>; vignesh C <vignesh21@gmail.com>; +Cc: Amit Kapila <amit.kapila16@gmail.com>; Hayato Kuroda (Fujitsu) <kuroda.hayato@fujitsu.com>; pgsql-bugs@lists.postgresql.org <pgsql-bugs@lists.postgresql.org>
Many thanks to all of you for working on this.
Best wishes, Duncan.
>>> I've attached the updated patches accordingly and the commit messages.
>>
>> Thanks, the changes look good to me.
>
> Pushed the fix (d87d07b7ad3). Thank you for working on this fix.
>
> Regards,
>
^ permalink raw reply [nested|flat] 94+ messages in thread
* RE: Logical replication 'invalid memory alloc request size 1585837200' after upgrading to 17.5
@ 2025-06-20 08:20 Hayato Kuroda (Fujitsu) <kuroda.hayato@fujitsu.com>
parent: vignesh C <vignesh21@gmail.com>
0 siblings, 2 replies; 94+ messages in thread
From: Hayato Kuroda (Fujitsu) @ 2025-06-20 08:20 UTC (permalink / raw)
To: pgsql-bugs@lists.postgresql.org <pgsql-bugs@lists.postgresql.org>; +Cc: Masahiko Sawada <sawada.mshk@gmail.com>; Amit Kapila <amit.kapila16@gmail.com>; Duncan Sands <duncan.sands@deepbluecap.com>; vignesh C <vignesh21@gmail.com>; Alexander Lakhin <exclusion@gmail.com>
Dear hackers,
I've analyzed the issue, and this can always happen when --DCLOBBER_CACHE_ALWAYS is
set on PG13. My suggestion is to remove the latter part of test on this branch.
In the failed workload, we tested the case that one long transaction inserts a
tuple after the concurrent transaction does ALTER PUBLICATION ADD TABLE.
For PG14+ the change can be published, and PG13 it cannot be replicated because
the distributed inval messages cannot be executed during the transaction. That's
why we expected no results were output.
So what if the -DDCLOBBER_CACHE_ALWAYS is set for the workload? relsync cache
can be discarded very frequently and backend process always read the system
catalog when it processes their changes. This means decoder can recognize that the
pg_publication has been updated and it can publish changes based on the altered
publication information. This behavior is opposite from normal.
To fix the test failure, I suggest to just remove the case. Insert-after-commit
case has already been tested by above part of this file, so no need to do others.
Attached file does accordingly.
Best regards,
Hayato Kuroda
FUJITSU LIMITED
Attachments:
[application/octet-stream] 0001-Fix-oversight-by-1230be12.patch (3.3K, ../../OSCPR01MB14966F91DBCAEA80EDEE67842F57CA@OSCPR01MB14966.jpnprd01.prod.outlook.com/2-0001-Fix-oversight-by-1230be12.patch)
download | inline diff:
From b5d5581e6d4bffe969b07978d43e3dda4b082d65 Mon Sep 17 00:00:00 2001
From: Hayato Kuroda <kuroda.hayato@fujitsu.com>
Date: Fri, 20 Jun 2025 15:44:34 +0900
Subject: [PATCH] Fix oversight by 1230be12
1230be12 introduces a test, but it can fail when -DCLOBBER_CACHE_XXX is set.
---
.../expected/invalidation_distribution.out | 23 +------------------
.../specs/invalidation_distribution.spec | 13 +----------
2 files changed, 2 insertions(+), 34 deletions(-)
diff --git a/contrib/test_decoding/expected/invalidation_distribution.out b/contrib/test_decoding/expected/invalidation_distribution.out
index a78ea1a26c5..eb70eda9042 100644
--- a/contrib/test_decoding/expected/invalidation_distribution.out
+++ b/contrib/test_decoding/expected/invalidation_distribution.out
@@ -1,4 +1,4 @@
-Parsed test spec with 3 sessions
+Parsed test spec with 2 sessions
starting permutation: s1_insert_tbl1 s1_begin s1_insert_tbl1 s2_alter_pub_add_tbl s1_commit s1_insert_tbl1 s2_get_binary_changes
step s1_insert_tbl1: INSERT INTO tbl1 (val1, val2) VALUES (1, 1);
@@ -18,24 +18,3 @@ count
stop
(1 row)
-
-starting permutation: s1_begin s1_insert_tbl1 s3_begin s3_insert_tbl1 s2_alter_pub_add_tbl s1_insert_tbl1 s1_commit s3_commit s2_get_binary_changes
-step s1_begin: BEGIN;
-step s1_insert_tbl1: INSERT INTO tbl1 (val1, val2) VALUES (1, 1);
-step s3_begin: BEGIN;
-step s3_insert_tbl1: INSERT INTO tbl1 (val1, val2) VALUES (2, 2);
-step s2_alter_pub_add_tbl: ALTER PUBLICATION pub ADD TABLE tbl1;
-step s1_insert_tbl1: INSERT INTO tbl1 (val1, val2) VALUES (1, 1);
-step s1_commit: COMMIT;
-step s3_commit: COMMIT;
-step s2_get_binary_changes: SELECT count(data) FROM pg_logical_slot_get_binary_changes('isolation_slot', NULL, NULL, 'proto_version', '1', 'publication_names', 'pub') WHERE get_byte(data, 0) = 73;
-count
------
- 0
-(1 row)
-
-?column?
---------
-stop
-(1 row)
-
diff --git a/contrib/test_decoding/specs/invalidation_distribution.spec b/contrib/test_decoding/specs/invalidation_distribution.spec
index c5977b8c483..6e911d45e23 100644
--- a/contrib/test_decoding/specs/invalidation_distribution.spec
+++ b/contrib/test_decoding/specs/invalidation_distribution.spec
@@ -28,16 +28,5 @@ setup { SET synchronous_commit=on; }
step "s2_alter_pub_add_tbl" { ALTER PUBLICATION pub ADD TABLE tbl1; }
step "s2_get_binary_changes" { SELECT count(data) FROM pg_logical_slot_get_binary_changes('isolation_slot', NULL, NULL, 'proto_version', '1', 'publication_names', 'pub') WHERE get_byte(data, 0) = 73; }
-session "s3"
-setup { SET synchronous_commit=on; }
-step "s3_begin" { BEGIN; }
-step "s3_insert_tbl1" { INSERT INTO tbl1 (val1, val2) VALUES (2, 2); }
-step "s3_commit" { COMMIT; }
-
# Expect to get one insert change. LOGICAL_REP_MSG_INSERT = 'I'
-permutation "s1_insert_tbl1" "s1_begin" "s1_insert_tbl1" "s2_alter_pub_add_tbl" "s1_commit" "s1_insert_tbl1" "s2_get_binary_changes"
-
-# Expect to get no change because both s1's and s3's transactions
-# use the snapshot from before adding the table tbl1 to the
-# publication by "s2_alter_pub_add_tbl".
-permutation "s1_begin" "s1_insert_tbl1" "s3_begin" "s3_insert_tbl1" "s2_alter_pub_add_tbl" "s1_insert_tbl1" "s1_commit" "s3_commit" "s2_get_binary_changes"
+permutation "s1_insert_tbl1" "s1_begin" "s1_insert_tbl1" "s2_alter_pub_add_tbl" "s1_commit" "s1_insert_tbl1" "s2_get_binary_changes"
\ No newline at end of file
--
2.47.1
^ permalink raw reply [nested|flat] 94+ messages in thread
* Re: Logical replication 'invalid memory alloc request size 1585837200' after upgrading to 17.5
@ 2025-06-21 00:17 Masahiko Sawada <sawada.mshk@gmail.com>
parent: Hayato Kuroda (Fujitsu) <kuroda.hayato@fujitsu.com>
1 sibling, 0 replies; 94+ messages in thread
From: Masahiko Sawada @ 2025-06-21 00:17 UTC (permalink / raw)
To: Hayato Kuroda (Fujitsu) <kuroda.hayato@fujitsu.com>; +Cc: pgsql-bugs@lists.postgresql.org <pgsql-bugs@lists.postgresql.org>; Amit Kapila <amit.kapila16@gmail.com>; Duncan Sands <duncan.sands@deepbluecap.com>; vignesh C <vignesh21@gmail.com>; Alexander Lakhin <exclusion@gmail.com>
On Fri, Jun 20, 2025 at 5:21 PM Hayato Kuroda (Fujitsu)
<kuroda.hayato@fujitsu.com> wrote:
>
> Dear hackers,
>
> I've analyzed the issue, and this can always happen when --DCLOBBER_CACHE_ALWAYS is
> set on PG13. My suggestion is to remove the latter part of test on this branch.
>
> In the failed workload, we tested the case that one long transaction inserts a
> tuple after the concurrent transaction does ALTER PUBLICATION ADD TABLE.
> For PG14+ the change can be published, and PG13 it cannot be replicated because
> the distributed inval messages cannot be executed during the transaction. That's
> why we expected no results were output.
>
> So what if the -DDCLOBBER_CACHE_ALWAYS is set for the workload? relsync cache
> can be discarded very frequently and backend process always read the system
> catalog when it processes their changes. This means decoder can recognize that the
> pg_publication has been updated and it can publish changes based on the altered
> publication information. This behavior is opposite from normal.
>
> To fix the test failure, I suggest to just remove the case. Insert-after-commit
> case has already been tested by above part of this file, so no need to do others.
>
> Attached file does accordingly.
Thank you for providing the patch. I'll look at the issue and the
patch in depth but it sounds like a reasonable solution.
Regards,
--
Masahiko Sawada
Amazon Web Services: https://aws.amazon.com
^ permalink raw reply [nested|flat] 94+ messages in thread
* Re: Logical replication 'invalid memory alloc request size 1585837200' after upgrading to 17.5
@ 2025-06-22 02:57 vignesh C <vignesh21@gmail.com>
parent: Hayato Kuroda (Fujitsu) <kuroda.hayato@fujitsu.com>
1 sibling, 1 reply; 94+ messages in thread
From: vignesh C @ 2025-06-22 02:57 UTC (permalink / raw)
To: Hayato Kuroda (Fujitsu) <kuroda.hayato@fujitsu.com>; +Cc: pgsql-bugs@lists.postgresql.org <pgsql-bugs@lists.postgresql.org>; Masahiko Sawada <sawada.mshk@gmail.com>; Amit Kapila <amit.kapila16@gmail.com>; Duncan Sands <duncan.sands@deepbluecap.com>; Alexander Lakhin <exclusion@gmail.com>
On Fri, 20 Jun 2025 at 13:51, Hayato Kuroda (Fujitsu)
<kuroda.hayato@fujitsu.com> wrote:
>
> Dear hackers,
>
> I've analyzed the issue, and this can always happen when --DCLOBBER_CACHE_ALWAYS is
> set on PG13. My suggestion is to remove the latter part of test on this branch.
>
> In the failed workload, we tested the case that one long transaction inserts a
> tuple after the concurrent transaction does ALTER PUBLICATION ADD TABLE.
> For PG14+ the change can be published, and PG13 it cannot be replicated because
> the distributed inval messages cannot be executed during the transaction. That's
> why we expected no results were output.
>
> So what if the -DDCLOBBER_CACHE_ALWAYS is set for the workload? relsync cache
> can be discarded very frequently and backend process always read the system
> catalog when it processes their changes. This means decoder can recognize that the
> pg_publication has been updated and it can publish changes based on the altered
> publication information. This behavior is opposite from normal.
I agree with your analysis.
> To fix the test failure, I suggest to just remove the case. Insert-after-commit
> case has already been tested by above part of this file, so no need to do others.
Alternatively I was thinking of a fix if it is possible to run this
test conditionally when CLOBBER_CACHE_ALWAYS is not defined, I was not
sure if it is easy to do that and worth the effort for the PG13
branch. I'm ok with the proposed change.
Regards,
Vignesh
^ permalink raw reply [nested|flat] 94+ messages in thread
* Re: Logical replication 'invalid memory alloc request size 1585837200' after upgrading to 17.5
@ 2025-06-23 02:52 Amit Kapila <amit.kapila16@gmail.com>
parent: vignesh C <vignesh21@gmail.com>
0 siblings, 1 reply; 94+ messages in thread
From: Amit Kapila @ 2025-06-23 02:52 UTC (permalink / raw)
To: vignesh C <vignesh21@gmail.com>; +Cc: Hayato Kuroda (Fujitsu) <kuroda.hayato@fujitsu.com>; pgsql-bugs@lists.postgresql.org <pgsql-bugs@lists.postgresql.org>; Masahiko Sawada <sawada.mshk@gmail.com>; Duncan Sands <duncan.sands@deepbluecap.com>; Alexander Lakhin <exclusion@gmail.com>
On Sun, Jun 22, 2025 at 8:28 AM vignesh C <vignesh21@gmail.com> wrote:
>
> On Fri, 20 Jun 2025 at 13:51, Hayato Kuroda (Fujitsu)
> <kuroda.hayato@fujitsu.com> wrote:
> >
>
> > To fix the test failure, I suggest to just remove the case. Insert-after-commit
> > case has already been tested by above part of this file, so no need to do others.
>
> Alternatively I was thinking of a fix if it is possible to run this
> test conditionally when CLOBBER_CACHE_ALWAYS is not defined, I was not
> sure if it is easy to do that and worth the effort for the PG13
> branch. I'm ok with the proposed change.
>
I prefer to change the test because, if the above analysis is correct,
it indicates that the test has a cache flush hazard. It would be
better to make the test robust instead of working around it.
--
With Regards,
Amit Kapila.
^ permalink raw reply [nested|flat] 94+ messages in thread
* Re: Logical replication 'invalid memory alloc request size 1585837200' after upgrading to 17.5
@ 2025-06-23 07:14 Masahiko Sawada <sawada.mshk@gmail.com>
parent: Amit Kapila <amit.kapila16@gmail.com>
0 siblings, 1 reply; 94+ messages in thread
From: Masahiko Sawada @ 2025-06-23 07:14 UTC (permalink / raw)
To: Amit Kapila <amit.kapila16@gmail.com>; +Cc: vignesh C <vignesh21@gmail.com>; Hayato Kuroda (Fujitsu) <kuroda.hayato@fujitsu.com>; pgsql-bugs@lists.postgresql.org <pgsql-bugs@lists.postgresql.org>; Duncan Sands <duncan.sands@deepbluecap.com>; Alexander Lakhin <exclusion@gmail.com>
On Mon, Jun 23, 2025 at 11:52 AM Amit Kapila <amit.kapila16@gmail.com> wrote:
>
> On Sun, Jun 22, 2025 at 8:28 AM vignesh C <vignesh21@gmail.com> wrote:
> >
> > On Fri, 20 Jun 2025 at 13:51, Hayato Kuroda (Fujitsu)
> > <kuroda.hayato@fujitsu.com> wrote:
> > >
> >
> > > To fix the test failure, I suggest to just remove the case. Insert-after-commit
> > > case has already been tested by above part of this file, so no need to do others.
> >
> > Alternatively I was thinking of a fix if it is possible to run this
> > test conditionally when CLOBBER_CACHE_ALWAYS is not defined, I was not
> > sure if it is easy to do that and worth the effort for the PG13
> > branch. I'm ok with the proposed change.
> >
>
> I prefer to change the test because, if the above analysis is correct,
> it indicates that the test has a cache flush hazard. It would be
> better to make the test robust instead of working around it.
The analysis shared by Kuroda-san matches my understanding. I'd like
to avoid removing this test case just because it doesn't pass with
CLOBBER_CACHE_ALWAYS.
One solution is to have two expected-output files to cover both cases.
We do a similar thing for the plpgsql_cache.sql test case. What do you
think?
Regards,
--
Masahiko Sawada
Amazon Web Services: https://aws.amazon.com
^ permalink raw reply [nested|flat] 94+ messages in thread
* RE: Logical replication 'invalid memory alloc request size 1585837200' after upgrading to 17.5
@ 2025-06-23 12:03 Hayato Kuroda (Fujitsu) <kuroda.hayato@fujitsu.com>
parent: Masahiko Sawada <sawada.mshk@gmail.com>
0 siblings, 1 reply; 94+ messages in thread
From: Hayato Kuroda (Fujitsu) @ 2025-06-23 12:03 UTC (permalink / raw)
To: Masahiko Sawada <sawada.mshk@gmail.com>; Amit Kapila <amit.kapila16@gmail.com>; +Cc: vignesh C <vignesh21@gmail.com>; pgsql-bugs@lists.postgresql.org <pgsql-bugs@lists.postgresql.org>; Duncan Sands <duncan.sands@deepbluecap.com>; Alexander Lakhin <exclusion@gmail.com>
Dear Sawada-san,
> One solution is to have two expected-output files to cover both cases.
> We do a similar thing for the plpgsql_cache.sql test case. What do you
> think?
Personally I don't like the approach because XXX_1.out can be easily missed to be
updated, but it is not a strong opinion.
Best regards,
Hayato Kuroda
FUJITSU LIMITED
^ permalink raw reply [nested|flat] 94+ messages in thread
* Re: Logical replication 'invalid memory alloc request size 1585837200' after upgrading to 17.5
@ 2025-06-23 14:05 Masahiko Sawada <sawada.mshk@gmail.com>
parent: Hayato Kuroda (Fujitsu) <kuroda.hayato@fujitsu.com>
0 siblings, 1 reply; 94+ messages in thread
From: Masahiko Sawada @ 2025-06-23 14:05 UTC (permalink / raw)
To: Hayato Kuroda (Fujitsu) <kuroda.hayato@fujitsu.com>; +Cc: Amit Kapila <amit.kapila16@gmail.com>; vignesh C <vignesh21@gmail.com>; pgsql-bugs@lists.postgresql.org <pgsql-bugs@lists.postgresql.org>; Duncan Sands <duncan.sands@deepbluecap.com>; Alexander Lakhin <exclusion@gmail.com>
On Mon, Jun 23, 2025 at 9:03 PM Hayato Kuroda (Fujitsu)
<kuroda.hayato@fujitsu.com> wrote:
>
> Dear Sawada-san,
>
> > One solution is to have two expected-output files to cover both cases.
> > We do a similar thing for the plpgsql_cache.sql test case. What do you
> > think?
>
> Personally I don't like the approach because XXX_1.out can be easily missed to be
> updated, but it is not a strong opinion.
I think that we have only a few releases for v13 and there would not
likely be many cases where we need to update _1.out file. But I'm open
to other ideas. Do you prefer removing the test from v13? I'm not sure
that just because it's easy to miss updating a _1.out file is a good
reason to remove tests.
Regards,
--
Masahiko Sawada
Amazon Web Services: https://aws.amazon.com
^ permalink raw reply [nested|flat] 94+ messages in thread
* Re: Logical replication 'invalid memory alloc request size 1585837200' after upgrading to 17.5
@ 2025-06-24 01:24 Masahiko Sawada <sawada.mshk@gmail.com>
parent: Masahiko Sawada <sawada.mshk@gmail.com>
0 siblings, 2 replies; 94+ messages in thread
From: Masahiko Sawada @ 2025-06-24 01:24 UTC (permalink / raw)
To: Hayato Kuroda (Fujitsu) <kuroda.hayato@fujitsu.com>; +Cc: Amit Kapila <amit.kapila16@gmail.com>; vignesh C <vignesh21@gmail.com>; pgsql-bugs@lists.postgresql.org <pgsql-bugs@lists.postgresql.org>; Duncan Sands <duncan.sands@deepbluecap.com>; Alexander Lakhin <exclusion@gmail.com>
On Mon, Jun 23, 2025 at 11:05 PM Masahiko Sawada <sawada.mshk@gmail.com> wrote:
>
> On Mon, Jun 23, 2025 at 9:03 PM Hayato Kuroda (Fujitsu)
> <kuroda.hayato@fujitsu.com> wrote:
> >
> > Dear Sawada-san,
> >
> > > One solution is to have two expected-output files to cover both cases.
> > > We do a similar thing for the plpgsql_cache.sql test case. What do you
> > > think?
> >
> > Personally I don't like the approach because XXX_1.out can be easily missed to be
> > updated, but it is not a strong opinion.
>
> I think that we have only a few releases for v13 and there would not
> likely be many cases where we need to update _1.out file. But I'm open
> to other ideas. Do you prefer removing the test from v13? I'm not sure
> that just because it's easy to miss updating a _1.out file is a good
> reason to remove tests.
>
I've added the patch for that idea for discussion. I considered moving
the new cache-behavior-dependent test to another test file to minimize
the maintenance effort but didn't do that at this stage as the test
file has only a few tests.
Regards,
--
Masahiko Sawada
Amazon Web Services: https://aws.amazon.com
Attachments:
[application/octet-stream] v1-0001-Fix-invalidation-distribution-test-failure-in-log.patch (3.9K, ../../CAD21AoACfSCfWv=Jtq2AjDyAKspkhNHN7ePvXyASMphYuDJPuw@mail.gmail.com/2-v1-0001-Fix-invalidation-distribution-test-failure-in-log.patch)
download | inline diff:
From ca4e957e34a685fa30e430dcc0d25a926ca4b849 Mon Sep 17 00:00:00 2001
From: Masahiko Sawada <sawada.mshk@gmail.com>
Date: Mon, 23 Jun 2025 18:18:57 -0700
Subject: [PATCH v1] Fix invalidation distribution test failure in logical
decoding.
Author:
Reviewed-by:
Discussion: https://postgr.es/m/
Backpatch-through:
---
.../expected/invalidation_distribution_1.out | 41 +++++++++++++++++++
.../specs/invalidation_distribution.spec | 10 +++++
2 files changed, 51 insertions(+)
create mode 100644 contrib/test_decoding/expected/invalidation_distribution_1.out
diff --git a/contrib/test_decoding/expected/invalidation_distribution_1.out b/contrib/test_decoding/expected/invalidation_distribution_1.out
new file mode 100644
index 00000000000..e5a19becb68
--- /dev/null
+++ b/contrib/test_decoding/expected/invalidation_distribution_1.out
@@ -0,0 +1,41 @@
+Parsed test spec with 3 sessions
+
+starting permutation: s1_insert_tbl1 s1_begin s1_insert_tbl1 s2_alter_pub_add_tbl s1_commit s1_insert_tbl1 s2_get_binary_changes
+step s1_insert_tbl1: INSERT INTO tbl1 (val1, val2) VALUES (1, 1);
+step s1_begin: BEGIN;
+step s1_insert_tbl1: INSERT INTO tbl1 (val1, val2) VALUES (1, 1);
+step s2_alter_pub_add_tbl: ALTER PUBLICATION pub ADD TABLE tbl1;
+step s1_commit: COMMIT;
+step s1_insert_tbl1: INSERT INTO tbl1 (val1, val2) VALUES (1, 1);
+step s2_get_binary_changes: SELECT count(data) FROM pg_logical_slot_get_binary_changes('isolation_slot', NULL, NULL, 'proto_version', '1', 'publication_names', 'pub') WHERE get_byte(data, 0) = 73;
+count
+-----
+ 1
+(1 row)
+
+?column?
+--------
+stop
+(1 row)
+
+
+starting permutation: s1_begin s1_insert_tbl1 s3_begin s3_insert_tbl1 s2_alter_pub_add_tbl s1_insert_tbl1 s1_commit s3_commit s2_get_binary_changes
+step s1_begin: BEGIN;
+step s1_insert_tbl1: INSERT INTO tbl1 (val1, val2) VALUES (1, 1);
+step s3_begin: BEGIN;
+step s3_insert_tbl1: INSERT INTO tbl1 (val1, val2) VALUES (2, 2);
+step s2_alter_pub_add_tbl: ALTER PUBLICATION pub ADD TABLE tbl1;
+step s1_insert_tbl1: INSERT INTO tbl1 (val1, val2) VALUES (1, 1);
+step s1_commit: COMMIT;
+step s3_commit: COMMIT;
+step s2_get_binary_changes: SELECT count(data) FROM pg_logical_slot_get_binary_changes('isolation_slot', NULL, NULL, 'proto_version', '1', 'publication_names', 'pub') WHERE get_byte(data, 0) = 73;
+count
+-----
+ 1
+(1 row)
+
+?column?
+--------
+stop
+(1 row)
+
diff --git a/contrib/test_decoding/specs/invalidation_distribution.spec b/contrib/test_decoding/specs/invalidation_distribution.spec
index c5977b8c483..24be7a13fbc 100644
--- a/contrib/test_decoding/specs/invalidation_distribution.spec
+++ b/contrib/test_decoding/specs/invalidation_distribution.spec
@@ -1,6 +1,10 @@
# Test that catalog cache invalidation messages are distributed to ongoing
# transactions, ensuring they can access the updated catalog content after
# processing these messages.
+#
+# This file contains cache-behavior-dependent test case. Their reults are
+# different between regular and CLOBBER_CACHE_ALWAYS builds, so we must have
+# two expected-output files to cvoer both cases.
setup
{
SELECT 'init' FROM pg_create_logical_replication_slot('isolation_slot', 'pgoutput');
@@ -40,4 +44,10 @@ permutation "s1_insert_tbl1" "s1_begin" "s1_insert_tbl1" "s2_alter_pub_add_tbl"
# Expect to get no change because both s1's and s3's transactions
# use the snapshot from before adding the table tbl1 to the
# publication by "s2_alter_pub_add_tbl".
+#
+# Note that with CLOBBER_CACHE_ALWAYS, we expect to get one insert
+# change with LOGICAL_REP_MSG_INSERT = "I' from the second
+# "s1_insert_tbl1" executed after adding the table tbl1 to the
+# publication in "s2_alter_pub_add_tbl" as the cache for table tbl
+# is forcibly flushed due to CLOBBER_CACHE_ALWAYS.
permutation "s1_begin" "s1_insert_tbl1" "s3_begin" "s3_insert_tbl1" "s2_alter_pub_add_tbl" "s1_insert_tbl1" "s1_commit" "s3_commit" "s2_get_binary_changes"
--
2.43.5
^ permalink raw reply [nested|flat] 94+ messages in thread
* Re: Logical replication 'invalid memory alloc request size 1585837200' after upgrading to 17.5
@ 2025-06-24 03:51 Amit Kapila <amit.kapila16@gmail.com>
parent: Masahiko Sawada <sawada.mshk@gmail.com>
1 sibling, 0 replies; 94+ messages in thread
From: Amit Kapila @ 2025-06-24 03:51 UTC (permalink / raw)
To: Masahiko Sawada <sawada.mshk@gmail.com>; +Cc: Hayato Kuroda (Fujitsu) <kuroda.hayato@fujitsu.com>; vignesh C <vignesh21@gmail.com>; pgsql-bugs@lists.postgresql.org <pgsql-bugs@lists.postgresql.org>; Duncan Sands <duncan.sands@deepbluecap.com>; Alexander Lakhin <exclusion@gmail.com>
On Tue, Jun 24, 2025 at 6:54 AM Masahiko Sawada <sawada.mshk@gmail.com> wrote:
>
> On Mon, Jun 23, 2025 at 11:05 PM Masahiko Sawada <sawada.mshk@gmail.com> wrote:
> >
> > On Mon, Jun 23, 2025 at 9:03 PM Hayato Kuroda (Fujitsu)
> > <kuroda.hayato@fujitsu.com> wrote:
> > >
> > > Dear Sawada-san,
> > >
> > > > One solution is to have two expected-output files to cover both cases.
> > > > We do a similar thing for the plpgsql_cache.sql test case. What do you
> > > > think?
> > >
> > > Personally I don't like the approach because XXX_1.out can be easily missed to be
> > > updated, but it is not a strong opinion.
> >
> > I think that we have only a few releases for v13 and there would not
> > likely be many cases where we need to update _1.out file. But I'm open
> > to other ideas. Do you prefer removing the test from v13? I'm not sure
> > that just because it's easy to miss updating a _1.out file is a good
> > reason to remove tests.
> >
>
> I've added the patch for that idea for discussion. I considered moving
> the new cache-behavior-dependent test to another test file to minimize
> the maintenance effort but didn't do that at this stage as the test
> file has only a few tests.
>
Your proposal sounds good to me.
--
With Regards,
Amit Kapila.
^ permalink raw reply [nested|flat] 94+ messages in thread
* Re: Logical replication 'invalid memory alloc request size 1585837200' after upgrading to 17.5
@ 2025-06-24 04:18 Tom Lane <tgl@sss.pgh.pa.us>
parent: Masahiko Sawada <sawada.mshk@gmail.com>
1 sibling, 1 reply; 94+ messages in thread
From: Tom Lane @ 2025-06-24 04:18 UTC (permalink / raw)
To: Masahiko Sawada <sawada.mshk@gmail.com>; +Cc: vignesh C <vignesh21@gmail.com>; Amit Kapila <amit.kapila16@gmail.com>; Hayato Kuroda (Fujitsu) <kuroda.hayato@fujitsu.com>; Duncan Sands <duncan.sands@deepbluecap.com>; pgsql-bugs@lists.postgresql.org <pgsql-bugs@lists.postgresql.org>
Masahiko Sawada <sawada.mshk@gmail.com> writes:
> Pushed the fix (d87d07b7ad3). Thank you for working on this fix.
There is something not right about the v13 version of this patch.
BF member trilobite, which builds with -DCLOBBER_CACHE_ALWAYS,
is showing this failure [1]:
diff -U3 /home/buildfarm/trilobite/buildroot/REL_13_STABLE/pgsql.build/contrib/test_decoding/expected/invalidation_distribution.out /home/buildfarm/trilobite/buildroot/REL_13_STABLE/pgsql.build/contrib/test_decoding/output_iso/results/invalidation_distribution.out
--- /home/buildfarm/trilobite/buildroot/REL_13_STABLE/pgsql.build/contrib/test_decoding/expected/invalidation_distribution.out 2025-06-17 10:24:24.382768613 +0200
+++ /home/buildfarm/trilobite/buildroot/REL_13_STABLE/pgsql.build/contrib/test_decoding/output_iso/results/invalidation_distribution.out 2025-06-17 15:01:53.921913314 +0200
@@ -31,7 +31,7 @@
step s2_get_binary_changes: SELECT count(data) FROM pg_logical_slot_get_binary_changes('isolation_slot', NULL, NULL, 'proto_version', '1', 'publication_names', 'pub') WHERE get_byte(data, 0) = 73;
count
-----
- 0
+ 1
(1 row)
I can reproduce that locally if I add -DCLOBBER_CACHE_ALWAYS.
Have not looked for the cause.
regards, tom lane
[1] https://buildfarm.postgresql.org/cgi-bin/show_log.pl?nm=trilobite&dt=2025-06-17%2008%3A24%3A00
^ permalink raw reply [nested|flat] 94+ messages in thread
* Re: Logical replication 'invalid memory alloc request size 1585837200' after upgrading to 17.5
@ 2025-06-24 04:22 Masahiko Sawada <sawada.mshk@gmail.com>
parent: Tom Lane <tgl@sss.pgh.pa.us>
0 siblings, 1 reply; 94+ messages in thread
From: Masahiko Sawada @ 2025-06-24 04:22 UTC (permalink / raw)
To: Tom Lane <tgl@sss.pgh.pa.us>; +Cc: vignesh C <vignesh21@gmail.com>; Amit Kapila <amit.kapila16@gmail.com>; Hayato Kuroda (Fujitsu) <kuroda.hayato@fujitsu.com>; Duncan Sands <duncan.sands@deepbluecap.com>; pgsql-bugs@lists.postgresql.org <pgsql-bugs@lists.postgresql.org>
On Tue, Jun 24, 2025 at 1:18 PM Tom Lane <tgl@sss.pgh.pa.us> wrote:
>
> Masahiko Sawada <sawada.mshk@gmail.com> writes:
> > Pushed the fix (d87d07b7ad3). Thank you for working on this fix.
>
> There is something not right about the v13 version of this patch.
> BF member trilobite, which builds with -DCLOBBER_CACHE_ALWAYS,
> is showing this failure [1]:
Yes. We've discussed how to fix this test failure and I proposed the
patch for that[1]. If there is no comment and objection, I'm going to
push it to v13.
Regards,
[1] https://www.postgresql.org/message-id/CAD21AoACfSCfWv%3DJtq2AjDyAKspkhNHN7ePvXyASMphYuDJPuw%40mail.g...
--
Masahiko Sawada
Amazon Web Services: https://aws.amazon.com
^ permalink raw reply [nested|flat] 94+ messages in thread
* Re: Logical replication 'invalid memory alloc request size 1585837200' after upgrading to 17.5
@ 2025-06-24 04:34 Tom Lane <tgl@sss.pgh.pa.us>
parent: Masahiko Sawada <sawada.mshk@gmail.com>
0 siblings, 0 replies; 94+ messages in thread
From: Tom Lane @ 2025-06-24 04:34 UTC (permalink / raw)
To: Masahiko Sawada <sawada.mshk@gmail.com>; +Cc: vignesh C <vignesh21@gmail.com>; Amit Kapila <amit.kapila16@gmail.com>; Hayato Kuroda (Fujitsu) <kuroda.hayato@fujitsu.com>; Duncan Sands <duncan.sands@deepbluecap.com>; pgsql-bugs@lists.postgresql.org <pgsql-bugs@lists.postgresql.org>
Masahiko Sawada <sawada.mshk@gmail.com> writes:
> On Tue, Jun 24, 2025 at 1:18 PM Tom Lane <tgl@sss.pgh.pa.us> wrote:
>> There is something not right about the v13 version of this patch.
> Yes. We've discussed how to fix this test failure and I proposed the
> patch for that[1]. If there is no comment and objection, I'm going to
> push it to v13.
Oh, my apologies for not having kept up on that thread!
Both of the proposed patches seem like band-aids, but I'm not
sure that it's worth working harder on a branch that has only
a few months to live. I'm okay with either solution.
regards, tom lane
^ permalink raw reply [nested|flat] 94+ messages in thread
* Re: Logical replication 'invalid memory alloc request size 1585837200' after upgrading to 17.5
@ 2025-06-24 04:58 Michael Paquier <michael@paquier.xyz>
parent: Masahiko Sawada <sawada.mshk@gmail.com>
1 sibling, 1 reply; 94+ messages in thread
From: Michael Paquier @ 2025-06-24 04:58 UTC (permalink / raw)
To: Masahiko Sawada <sawada.mshk@gmail.com>; +Cc: Hayato Kuroda (Fujitsu) <kuroda.hayato@fujitsu.com>; Amit Kapila <amit.kapila16@gmail.com>; vignesh C <vignesh21@gmail.com>; pgsql-bugs@lists.postgresql.org <pgsql-bugs@lists.postgresql.org>; Duncan Sands <duncan.sands@deepbluecap.com>; Alexander Lakhin <exclusion@gmail.com>
On Tue, Jun 24, 2025 at 10:24:03AM +0900, Masahiko Sawada wrote:
> I've added the patch for that idea for discussion. I considered moving
> the new cache-behavior-dependent test to another test file to minimize
> the maintenance effort but didn't do that at this stage as the test
> file has only a few tests.
The spec test file only includes two short permutations, making the
generated output really short. A secondary output file sounds fine to
me as long as you document in the spec file the reason why the file is
around, and your patch does that.
+# This file contains cache-behavior-dependent test case. Their reults are
[..]
+# two expected-output files to cvoer both cases.
Two typos in three lines of comments: s/reults/results/ and
s/cvoer/cover/.
--
Michael
Attachments:
[application/pgp-signature] signature.asc (832B, ../../aFowg2mxXDrrh8CG@paquier.xyz/2-signature.asc)
download
^ permalink raw reply [nested|flat] 94+ messages in thread
* Re: Logical replication 'invalid memory alloc request size 1585837200' after upgrading to 17.5
@ 2025-06-24 05:14 Masahiko Sawada <sawada.mshk@gmail.com>
parent: Michael Paquier <michael@paquier.xyz>
0 siblings, 1 reply; 94+ messages in thread
From: Masahiko Sawada @ 2025-06-24 05:14 UTC (permalink / raw)
To: Michael Paquier <michael@paquier.xyz>; +Cc: Hayato Kuroda (Fujitsu) <kuroda.hayato@fujitsu.com>; Amit Kapila <amit.kapila16@gmail.com>; vignesh C <vignesh21@gmail.com>; pgsql-bugs@lists.postgresql.org <pgsql-bugs@lists.postgresql.org>; Duncan Sands <duncan.sands@deepbluecap.com>; Alexander Lakhin <exclusion@gmail.com>
On Tue, Jun 24, 2025 at 1:59 PM Michael Paquier <michael@paquier.xyz> wrote:
>
> On Tue, Jun 24, 2025 at 10:24:03AM +0900, Masahiko Sawada wrote:
> > I've added the patch for that idea for discussion. I considered moving
> > the new cache-behavior-dependent test to another test file to minimize
> > the maintenance effort but didn't do that at this stage as the test
> > file has only a few tests.
>
> The spec test file only includes two short permutations, making the
> generated output really short. A secondary output file sounds fine to
> me as long as you document in the spec file the reason why the file is
> around, and your patch does that.
>
> +# This file contains cache-behavior-dependent test case. Their reults are
> [..]
> +# two expected-output files to cvoer both cases.
>
> Two typos in three lines of comments: s/reults/results/ and
> s/cvoer/cover/.
Thank you for reviewing the patch!
I've attached the updated patch. I'm going to push it barring further comments.
Regards,
--
Masahiko Sawada
Amazon Web Services: https://aws.amazon.com
Attachments:
[application/octet-stream] v2-0001-Fix-cache-dependent-test-failures-in-logical-deco.patch (4.5K, ../../CAD21AoBfOJMMA_Xw6BKyWLGuhhZ6NNm+qBt-6P4gnmX3Xj_i5w@mail.gmail.com/2-v2-0001-Fix-cache-dependent-test-failures-in-logical-deco.patch)
download | inline diff:
From d877f19e986409c2937d3f13be1c0b702be8aca3 Mon Sep 17 00:00:00 2001
From: Masahiko Sawada <sawada.mshk@gmail.com>
Date: Mon, 23 Jun 2025 18:18:57 -0700
Subject: [PATCH v2] Fix cache-dependent test failures in logical decoding.
The regression test added in commit 1230be12f failed with
CLOBBER_CACHE_ALWAYS builds as it depends on cache behavior. This test
failure occurred only on v13 because the original data loss problem
was fixed differently in v13 compared to v14 and later versions,
resulting in different expected-output files.
This commit adds an extra expected-output file to cover both regular
and CLOBBER_CACHE_ALWAYS build cases.
Oversight in 1230be12f.
Per buildfarm member trilobite.
Reported-by: Alexander Lakhin <exclusion@gmail.com>
Reviewed-by: Amit Kapila <amit.kapila16@gmail.com>
Reviewed-by: Hayato Kuroda <kuroda.hayato@fujitsu.com>
Reviewed-by: Michael Paquier <michael@paquier.xyz>
Discussion: https://postgr.es/m/dbf561f7-465e-4086-adfa-733b9b9a34b3@gmail.com
---
.../expected/invalidation_distribution_1.out | 41 +++++++++++++++++++
.../specs/invalidation_distribution.spec | 10 +++++
2 files changed, 51 insertions(+)
create mode 100644 contrib/test_decoding/expected/invalidation_distribution_1.out
diff --git a/contrib/test_decoding/expected/invalidation_distribution_1.out b/contrib/test_decoding/expected/invalidation_distribution_1.out
new file mode 100644
index 00000000000..e5a19becb68
--- /dev/null
+++ b/contrib/test_decoding/expected/invalidation_distribution_1.out
@@ -0,0 +1,41 @@
+Parsed test spec with 3 sessions
+
+starting permutation: s1_insert_tbl1 s1_begin s1_insert_tbl1 s2_alter_pub_add_tbl s1_commit s1_insert_tbl1 s2_get_binary_changes
+step s1_insert_tbl1: INSERT INTO tbl1 (val1, val2) VALUES (1, 1);
+step s1_begin: BEGIN;
+step s1_insert_tbl1: INSERT INTO tbl1 (val1, val2) VALUES (1, 1);
+step s2_alter_pub_add_tbl: ALTER PUBLICATION pub ADD TABLE tbl1;
+step s1_commit: COMMIT;
+step s1_insert_tbl1: INSERT INTO tbl1 (val1, val2) VALUES (1, 1);
+step s2_get_binary_changes: SELECT count(data) FROM pg_logical_slot_get_binary_changes('isolation_slot', NULL, NULL, 'proto_version', '1', 'publication_names', 'pub') WHERE get_byte(data, 0) = 73;
+count
+-----
+ 1
+(1 row)
+
+?column?
+--------
+stop
+(1 row)
+
+
+starting permutation: s1_begin s1_insert_tbl1 s3_begin s3_insert_tbl1 s2_alter_pub_add_tbl s1_insert_tbl1 s1_commit s3_commit s2_get_binary_changes
+step s1_begin: BEGIN;
+step s1_insert_tbl1: INSERT INTO tbl1 (val1, val2) VALUES (1, 1);
+step s3_begin: BEGIN;
+step s3_insert_tbl1: INSERT INTO tbl1 (val1, val2) VALUES (2, 2);
+step s2_alter_pub_add_tbl: ALTER PUBLICATION pub ADD TABLE tbl1;
+step s1_insert_tbl1: INSERT INTO tbl1 (val1, val2) VALUES (1, 1);
+step s1_commit: COMMIT;
+step s3_commit: COMMIT;
+step s2_get_binary_changes: SELECT count(data) FROM pg_logical_slot_get_binary_changes('isolation_slot', NULL, NULL, 'proto_version', '1', 'publication_names', 'pub') WHERE get_byte(data, 0) = 73;
+count
+-----
+ 1
+(1 row)
+
+?column?
+--------
+stop
+(1 row)
+
diff --git a/contrib/test_decoding/specs/invalidation_distribution.spec b/contrib/test_decoding/specs/invalidation_distribution.spec
index c5977b8c483..c36ee2d59e5 100644
--- a/contrib/test_decoding/specs/invalidation_distribution.spec
+++ b/contrib/test_decoding/specs/invalidation_distribution.spec
@@ -1,6 +1,10 @@
# Test that catalog cache invalidation messages are distributed to ongoing
# transactions, ensuring they can access the updated catalog content after
# processing these messages.
+#
+# This file contains a cache-behavior-dependent test case. Its result is
+# different between regular and CLOBBER_CACHE_ALWAYS builds, so we must have
+# two expected-output files to cover both cases.
setup
{
SELECT 'init' FROM pg_create_logical_replication_slot('isolation_slot', 'pgoutput');
@@ -40,4 +44,10 @@ permutation "s1_insert_tbl1" "s1_begin" "s1_insert_tbl1" "s2_alter_pub_add_tbl"
# Expect to get no change because both s1's and s3's transactions
# use the snapshot from before adding the table tbl1 to the
# publication by "s2_alter_pub_add_tbl".
+#
+# Note that with CLOBBER_CACHE_ALWAYS, we expect to get one insert
+# change with LOGICAL_REP_MSG_INSERT = 'I' from the second
+# "s1_insert_tbl1" executed after adding the table tbl1 to the
+# publication in "s2_alter_pub_add_tbl" as the cache for table tbl1
+# is forcibly flushed due to CLOBBER_CACHE_ALWAYS.
permutation "s1_begin" "s1_insert_tbl1" "s3_begin" "s3_insert_tbl1" "s2_alter_pub_add_tbl" "s1_insert_tbl1" "s1_commit" "s3_commit" "s2_get_binary_changes"
--
2.43.5
^ permalink raw reply [nested|flat] 94+ messages in thread
* Re: Logical replication 'invalid memory alloc request size 1585837200' after upgrading to 17.5
@ 2025-07-02 06:32 Masahiko Sawada <sawada.mshk@gmail.com>
parent: Masahiko Sawada <sawada.mshk@gmail.com>
0 siblings, 0 replies; 94+ messages in thread
From: Masahiko Sawada @ 2025-07-02 06:32 UTC (permalink / raw)
To: Michael Paquier <michael@paquier.xyz>; +Cc: Hayato Kuroda (Fujitsu) <kuroda.hayato@fujitsu.com>; Amit Kapila <amit.kapila16@gmail.com>; vignesh C <vignesh21@gmail.com>; pgsql-bugs@lists.postgresql.org <pgsql-bugs@lists.postgresql.org>; Duncan Sands <duncan.sands@deepbluecap.com>; Alexander Lakhin <exclusion@gmail.com>
On Tue, Jun 24, 2025 at 2:14 PM Masahiko Sawada <sawada.mshk@gmail.com> wrote:
>
> On Tue, Jun 24, 2025 at 1:59 PM Michael Paquier <michael@paquier.xyz> wrote:
> >
> > On Tue, Jun 24, 2025 at 10:24:03AM +0900, Masahiko Sawada wrote:
> > > I've added the patch for that idea for discussion. I considered moving
> > > the new cache-behavior-dependent test to another test file to minimize
> > > the maintenance effort but didn't do that at this stage as the test
> > > file has only a few tests.
> >
> > The spec test file only includes two short permutations, making the
> > generated output really short. A secondary output file sounds fine to
> > me as long as you document in the spec file the reason why the file is
> > around, and your patch does that.
> >
> > +# This file contains cache-behavior-dependent test case. Their reults are
> > [..]
> > +# two expected-output files to cvoer both cases.
> >
> > Two typos in three lines of comments: s/reults/results/ and
> > s/cvoer/cover/.
>
> Thank you for reviewing the patch!
>
> I've attached the updated patch. I'm going to push it barring further comments.
>
Forgot to record. I pushed the fix[1] and confirmed that trilobite is
back to green.
Regards,
[1] https://git.postgresql.org/gitweb/?p=postgresql.git;a=commit;h=d87d07b7ad3b782cb74566cd771ecdb2823ad...
--
Masahiko Sawada
Amazon Web Services: https://aws.amazon.com
^ permalink raw reply [nested|flat] 94+ messages in thread
end of thread, other threads:[~2025-07-02 06:32 UTC | newest]
Thread overview: 94+ messages (download: mbox mbox.gz follow: Atom feed)
-- links below jump to the message on this page --
2025-05-19 11:46 Logical replication 'invalid memory alloc request size 1585837200' after upgrading to 17.5 Duncan Sands <duncan.sands@deepbluecap.com>
2025-05-21 05:46 ` Shlok Kyal <shlok.kyal.oss@gmail.com>
2025-05-21 05:48 ` Amit Kapila <amit.kapila16@gmail.com>
2025-05-21 11:12 ` Amit Kapila <amit.kapila16@gmail.com>
2025-05-21 11:48 ` Hayato Kuroda (Fujitsu) <kuroda.hayato@fujitsu.com>
2025-05-21 13:27 ` Duncan Sands <duncan.sands@deepbluecap.com>
2025-05-22 12:23 ` Shlok Kyal <shlok.kyal.oss@gmail.com>
2025-05-24 15:42 ` Duncan Sands <duncan.sands@deepbluecap.com>
2025-05-21 18:24 ` Masahiko Sawada <sawada.mshk@gmail.com>
2025-05-22 10:56 ` Amit Kapila <amit.kapila16@gmail.com>
2025-05-22 12:59 ` Hayato Kuroda (Fujitsu) <kuroda.hayato@fujitsu.com>
2025-05-23 03:00 ` Amit Kapila <amit.kapila16@gmail.com>
2025-05-23 17:31 ` Masahiko Sawada <sawada.mshk@gmail.com>
2025-05-25 04:55 ` Hayato Kuroda (Fujitsu) <kuroda.hayato@fujitsu.com>
2025-05-26 09:22 ` Hayato Kuroda (Fujitsu) <kuroda.hayato@fujitsu.com>
2025-05-26 11:18 ` Amit Kapila <amit.kapila16@gmail.com>
2025-05-27 21:05 ` Masahiko Sawada <sawada.mshk@gmail.com>
2025-05-28 12:27 ` Hayato Kuroda (Fujitsu) <kuroda.hayato@fujitsu.com>
2025-06-02 07:12 ` Duncan Sands <duncan.sands@deepbluecap.com>
2025-05-28 18:51 ` Masahiko Sawada <sawada.mshk@gmail.com>
2025-05-29 05:20 ` Amit Kapila <amit.kapila16@gmail.com>
2025-05-29 06:07 ` Masahiko Sawada <sawada.mshk@gmail.com>
2025-05-29 08:22 ` Amit Kapila <amit.kapila16@gmail.com>
2025-05-29 17:26 ` Masahiko Sawada <sawada.mshk@gmail.com>
2025-05-30 04:01 ` vignesh C <vignesh21@gmail.com>
2025-05-30 04:42 ` Amit Kapila <amit.kapila16@gmail.com>
2025-05-30 12:15 ` vignesh C <vignesh21@gmail.com>
2025-05-30 17:29 ` Masahiko Sawada <sawada.mshk@gmail.com>
2025-05-31 04:50 ` Amit Kapila <amit.kapila16@gmail.com>
2025-06-02 03:13 ` vignesh C <vignesh21@gmail.com>
2025-06-02 16:51 ` Masahiko Sawada <sawada.mshk@gmail.com>
2025-06-03 03:38 ` Amit Kapila <amit.kapila16@gmail.com>
2025-05-31 07:57 ` vignesh C <vignesh21@gmail.com>
2025-06-02 10:22 ` vignesh C <vignesh21@gmail.com>
2025-06-02 11:35 ` Hayato Kuroda (Fujitsu) <kuroda.hayato@fujitsu.com>
2025-06-02 18:08 ` Masahiko Sawada <sawada.mshk@gmail.com>
2025-06-03 03:09 ` Amit Kapila <amit.kapila16@gmail.com>
2025-06-02 17:19 ` Masahiko Sawada <sawada.mshk@gmail.com>
2025-06-03 07:07 ` vignesh C <vignesh21@gmail.com>
2025-06-03 12:24 ` vignesh C <vignesh21@gmail.com>
2025-06-03 19:43 ` Masahiko Sawada <sawada.mshk@gmail.com>
2025-06-04 06:47 ` vignesh C <vignesh21@gmail.com>
2025-06-04 21:48 ` Masahiko Sawada <sawada.mshk@gmail.com>
2025-06-05 06:20 ` Amit Kapila <amit.kapila16@gmail.com>
2025-06-05 11:07 ` Hayato Kuroda (Fujitsu) <kuroda.hayato@fujitsu.com>
2025-06-05 12:14 ` Hayato Kuroda (Fujitsu) <kuroda.hayato@fujitsu.com>
2025-06-05 19:21 ` Masahiko Sawada <sawada.mshk@gmail.com>
2025-06-06 03:21 ` Amit Kapila <amit.kapila16@gmail.com>
2025-06-06 17:23 ` Masahiko Sawada <sawada.mshk@gmail.com>
2025-06-05 18:49 ` Masahiko Sawada <sawada.mshk@gmail.com>
2025-06-05 10:28 ` vignesh C <vignesh21@gmail.com>
2025-06-05 21:59 ` Masahiko Sawada <sawada.mshk@gmail.com>
2025-06-04 09:20 ` Hayato Kuroda (Fujitsu) <kuroda.hayato@fujitsu.com>
2025-06-04 22:06 ` Masahiko Sawada <sawada.mshk@gmail.com>
2025-06-05 06:36 ` Hayato Kuroda (Fujitsu) <kuroda.hayato@fujitsu.com>
2025-06-05 17:43 ` Masahiko Sawada <sawada.mshk@gmail.com>
2025-06-06 03:43 ` Amit Kapila <amit.kapila16@gmail.com>
2025-06-06 17:20 ` Masahiko Sawada <sawada.mshk@gmail.com>
2025-06-07 06:03 ` Amit Kapila <amit.kapila16@gmail.com>
2025-06-07 07:17 ` vignesh C <vignesh21@gmail.com>
2025-06-08 13:43 ` vignesh C <vignesh21@gmail.com>
2025-06-10 19:19 ` Masahiko Sawada <sawada.mshk@gmail.com>
2025-06-11 22:03 ` Masahiko Sawada <sawada.mshk@gmail.com>
2025-06-12 06:47 ` vignesh C <vignesh21@gmail.com>
2025-06-13 23:52 ` Masahiko Sawada <sawada.mshk@gmail.com>
2025-06-16 02:27 ` vignesh C <vignesh21@gmail.com>
2025-06-17 17:40 ` Masahiko Sawada <sawada.mshk@gmail.com>
2025-06-20 05:00 ` Alexander Lakhin <exclusion@gmail.com>
2025-06-20 06:04 ` vignesh C <vignesh21@gmail.com>
2025-06-20 08:20 ` Hayato Kuroda (Fujitsu) <kuroda.hayato@fujitsu.com>
2025-06-21 00:17 ` Masahiko Sawada <sawada.mshk@gmail.com>
2025-06-22 02:57 ` vignesh C <vignesh21@gmail.com>
2025-06-23 02:52 ` Amit Kapila <amit.kapila16@gmail.com>
2025-06-23 07:14 ` Masahiko Sawada <sawada.mshk@gmail.com>
2025-06-23 12:03 ` Hayato Kuroda (Fujitsu) <kuroda.hayato@fujitsu.com>
2025-06-23 14:05 ` Masahiko Sawada <sawada.mshk@gmail.com>
2025-06-24 01:24 ` Masahiko Sawada <sawada.mshk@gmail.com>
2025-06-24 03:51 ` Amit Kapila <amit.kapila16@gmail.com>
2025-06-24 04:58 ` Michael Paquier <michael@paquier.xyz>
2025-06-24 05:14 ` Masahiko Sawada <sawada.mshk@gmail.com>
2025-07-02 06:32 ` Masahiko Sawada <sawada.mshk@gmail.com>
2025-06-20 07:48 ` Duncan Sands <duncan.sands@deepbluecap.com>
2025-06-24 04:18 ` Tom Lane <tgl@sss.pgh.pa.us>
2025-06-24 04:22 ` Masahiko Sawada <sawada.mshk@gmail.com>
2025-06-24 04:34 ` Tom Lane <tgl@sss.pgh.pa.us>
2025-06-11 01:45 ` Hayato Kuroda (Fujitsu) <kuroda.hayato@fujitsu.com>
2025-06-11 03:19 ` Amit Kapila <amit.kapila16@gmail.com>
2025-06-06 06:22 ` Hayato Kuroda (Fujitsu) <kuroda.hayato@fujitsu.com>
2025-06-06 06:54 ` Hayato Kuroda (Fujitsu) <kuroda.hayato@fujitsu.com>
2025-05-30 16:45 ` Masahiko Sawada <sawada.mshk@gmail.com>
2025-05-29 12:09 ` Hayato Kuroda (Fujitsu) <kuroda.hayato@fujitsu.com>
2025-05-29 06:10 ` Hayato Kuroda (Fujitsu) <kuroda.hayato@fujitsu.com>
2025-05-21 11:30 ` Duncan Sands <duncan.sands@deepbluecap.com>
2025-05-26 11:47 ` Christoph Berg <myon@debian.org>
This inbox is served by DDX for PostgreSQL; see mirroring instructions
for how to clone and mirror all data and code used for this inbox