pg.ddx.io  pgsql-bugs@postgresql.org mailing list archive  
help / color / mirror / Atom feed
From: Zhijie Hou <houzhijie22@gmail.com>
To: Bertrand Drouvot <bertranddrouvot.pg@gmail.com>
Cc: Atisss hh <125atishprak@gmail.com>
Cc: pgsql-bugs@lists.postgresql.org
Subject: Re: BUG #19839: Assert in ReorderBufferRestoreCleanup for aborted, spilled transaction
Date: Mon, 9 Nov 2026 19:46:59 +0800
Message-ID: <ba2ec30c-e381-4422-913b-24afb4263fc8@gmail.com> (raw)
In-Reply-To: <avGmDSdgyIZB73Sl@bdtpg>
References: <19839-ead02c2208a3f8bd@postgresql.org>
	<CAGqfTtvQB4J0dJFnNMAF9LCAJhGBe5THd2R_=MxTHP5C5vwD=w@mail.gmail.com>
	<1da45744-968f-4f40-addf-dc085e928ec5@gmail.com>
	<avGmDSdgyIZB73Sl@bdtpg>

Hi,

On Mon, Nov 9, 2026 at 06:53 PM Bertrand Drouvot wrote:
>
> On Sun, Nov 08, 2026 at 11:11:09PM +0800, Zhijie Hou wrote:
> > Thanks for the report and patch.
>
> +1.
>
> > See the attachment for a test to reproduce this issue. (Feel free to merge
> > if it's worth adding a test to cover this path)
>
> > I think in a rare case, this can also affect the stats collection
> > for spilled txns. For example, if an empty subtransaction is marked
> > as
> > serialized, we skip incrementing spillTxns because no changes were written;
> > if real changes are later serialized in this subtransaction, we still do not
> > increment spillTxns because the subtransaction is already marked as
> > serialized and we avoid counting it "twice".
>
> Agreed.
>
> That's why I think the test is worth adding (as the bug could also
> produce real user visible impact as it can cause spill_txns to be
> undercounted on non assert enabled builds). The test only detects the
> regression on assert enabled builds, but I think that is enough. A
> test for the spill_txns case would be more complicated and is probably
> not worth it.
>
> >
> > +	if (spilled)
> > +		txn->txn_flags |= RBTXN_IS_SERIALIZED;
> >
> > ReorderBufferSerializeTXN() already has an "if (spilled)" block
> > right above this, so we shall directly move the assignment into that
> > block.
>
> +1.

Thanks for the review. It looks like Atisss's update patches didn't 
reach the list, probably a missing Reply-All, so I'm helping to attach 
the same patches (provided by Atisss) here.


-- 
Best Regards,
Zhijie Hou

Attachments: download all (3 files, .tar.gz)

v2-0001-fix-empty-subxact-serialized-flag.patch (text/x-patch, 4.7K) download From 092dc34678cd151de4262322bda1b2d31b1cec11 Mon Sep 17 00:00:00 2001 From: Atish Prakash <125atishprak@gmail.com> Date: Sun, 8 Nov 2026 21:06:11 +0530 Subject: [PATCH v2] Don't mark a transaction serialized when nothing was spilled ReorderBufferSerializeTXN() set RBTXN_IS_SERIALIZED on every transaction it visited, including subtransactions with no changes to write. Such a transaction keeps an invalid final_lsn, because final_lsn is only advanced as changes are written, so ReorderBufferRestoreCleanup() can later fail its assertion on it. This happens when ReorderBufferCheckAndTruncateAbortedTXN() truncates an aborted transaction, and when a transaction is truncated after being streamed once a partial change completes. It also made spill_txns undercount: when such a subtransaction later spilled real changes, it was not counted, as it was already marked serialized. Set the flag only when changes were written, after the statistics update that checks it. Test case by Zhijie Hou. Bug: #19839 Reported-by: Atish Prakash <125atishprak@gmail.com> Author: Atish Prakash <125atishprak@gmail.com> Reviewed-by: Zhijie Hou <houzhijie22@gmail.com> Backpatch-through: 14 --- contrib/test_decoding/expected/stream.out | 24 +++++++++++++++++++ contrib/test_decoding/sql/stream.sql | 15 ++++++++++++ .../replication/logical/reorderbuffer.c | 11 ++++++++- 3 files changed, 49 insertions(+), 1 deletion(-) diff --git a/contrib/test_decoding/expected/stream.out b/contrib/test_decoding/expected/stream.out index 0ec5c93361..9c164847b0 100644 --- a/contrib/test_decoding/expected/stream.out +++ b/contrib/test_decoding/expected/stream.out @@ -157,6 +157,30 @@ SELECT data FROM pg_logical_slot_get_changes('regression_slot', NULL, NULL, 'inc COMMIT (3 rows) +-- bug #19839 +-- +-- A subtransaction without decoded changes must not be marked as serialized +-- when its parent transaction spills to disk during a partial TOAST change +-- (stream_test.data has EXTERNAL storage set above). Otherwise, truncating +-- the transaction after streaming fails on the subtransaction's invalid +-- final_lsn. +BEGIN; +SAVEPOINT s; +SELECT 1 FROM stream_test LIMIT 1 FOR UPDATE; + ?column? +---------- + 1 +(1 row) + +RELEASE SAVEPOINT s; +INSERT INTO stream_test VALUES (repeat('a', 100000)); +COMMIT; +SELECT count(*) FROM pg_logical_slot_get_changes('regression_slot', NULL, NULL, 'include-xids', '0', 'skip-empty-xacts', '1', 'stream-changes', '1'); + count +------- + 4 +(1 row) + DROP TABLE stream_test; SELECT pg_drop_replication_slot('regression_slot'); pg_drop_replication_slot diff --git a/contrib/test_decoding/sql/stream.sql b/contrib/test_decoding/sql/stream.sql index 5e45a8e9b6..f315933556 100644 --- a/contrib/test_decoding/sql/stream.sql +++ b/contrib/test_decoding/sql/stream.sql @@ -82,5 +82,20 @@ ROLLBACK; INSERT INTO stream_test VALUES ('after-abort'); SELECT data FROM pg_logical_slot_get_changes('regression_slot', NULL, NULL, 'include-xids', '0', 'skip-empty-xacts', '1'); +-- bug #19839 +-- +-- A subtransaction without decoded changes must not be marked as serialized +-- when its parent transaction spills to disk during a partial TOAST change +-- (stream_test.data has EXTERNAL storage set above). Otherwise, truncating +-- the transaction after streaming fails on the subtransaction's invalid +-- final_lsn. +BEGIN; +SAVEPOINT s; +SELECT 1 FROM stream_test LIMIT 1 FOR UPDATE; +RELEASE SAVEPOINT s; +INSERT INTO stream_test VALUES (repeat('a', 100000)); +COMMIT; +SELECT count(*) FROM pg_logical_slot_get_changes('regression_slot', NULL, NULL, 'include-xids', '0', 'skip-empty-xacts', '1', 'stream-changes', '1'); + DROP TABLE stream_test; SELECT pg_drop_replication_slot('regression_slot'); diff --git a/src/backend/replication/logical/reorderbuffer.c b/src/backend/replication/logical/reorderbuffer.c index 4684398531..acbd2c1344 100644 --- a/src/backend/replication/logical/reorderbuffer.c +++ b/src/backend/replication/logical/reorderbuffer.c @@ -4211,12 +4211,21 @@ ReorderBufferSerializeTXN(ReorderBuffer *rb, ReorderBufferTXN *txn) /* update the decoding stats */ UpdateDecodingStats((LogicalDecodingContext *) rb->private_data); + + /* + * Mark the transaction as serialized only if something was written, + * and only after the check above. final_lsn is advanced for each + * change written, so a transaction that wrote nothing, such as a + * subtransaction without decoded changes, would be marked with an + * invalid final_lsn, and ReorderBufferRestoreCleanup() would later + * fail on it. + */ + txn->txn_flags |= RBTXN_IS_SERIALIZED; } Assert(spilled == txn->nentries_mem); Assert(dlist_is_empty(&txn->changes)); txn->nentries_mem = 0; - txn->txn_flags |= RBTXN_IS_SERIALIZED; if (fd != -1) CloseTransientFile(fd); -- 2.53.0
v2-REL_14_STABLE.patch (text/x-patch, 3.3K) download diff --git a/contrib/test_decoding/expected/stream.out b/contrib/test_decoding/expected/stream.out index 0f21dcb8e0..0b6288fb14 100644 --- a/contrib/test_decoding/expected/stream.out +++ b/contrib/test_decoding/expected/stream.out @@ -106,6 +106,29 @@ SELECT data FROM pg_logical_slot_get_changes('regression_slot', NULL,NULL, 'incl committing streamed transaction (17 rows) +-- bug #19839 +-- +-- A subtransaction without decoded changes must not be marked as serialized +-- when its parent transaction spills to disk during a partial TOAST change. +-- Otherwise, truncating the transaction after streaming fails on the +-- subtransaction's invalid final_lsn. +BEGIN; +SAVEPOINT s; +SELECT 1 FROM stream_test LIMIT 1 FOR UPDATE; + ?column? +---------- + 1 +(1 row) + +RELEASE SAVEPOINT s; +INSERT INTO stream_test VALUES (repeat('a', 100000)); +COMMIT; +SELECT count(*) FROM pg_logical_slot_get_changes('regression_slot', NULL, NULL, 'include-xids', '0', 'skip-empty-xacts', '1', 'stream-changes', '1'); + count +------- + 4 +(1 row) + DROP TABLE stream_test; SELECT pg_drop_replication_slot('regression_slot'); pg_drop_replication_slot diff --git a/contrib/test_decoding/sql/stream.sql b/contrib/test_decoding/sql/stream.sql index 4feec62972..1541e5dca3 100644 --- a/contrib/test_decoding/sql/stream.sql +++ b/contrib/test_decoding/sql/stream.sql @@ -44,5 +44,19 @@ toasted-123456789012345678901234567890123456789012345678901234567890123456789012 SELECT data FROM pg_logical_slot_get_changes('regression_slot', NULL,NULL, 'include-xids', '0', 'skip-empty-xacts', '1', 'stream-changes', '1'); +-- bug #19839 +-- +-- A subtransaction without decoded changes must not be marked as serialized +-- when its parent transaction spills to disk during a partial TOAST change. +-- Otherwise, truncating the transaction after streaming fails on the +-- subtransaction's invalid final_lsn. +BEGIN; +SAVEPOINT s; +SELECT 1 FROM stream_test LIMIT 1 FOR UPDATE; +RELEASE SAVEPOINT s; +INSERT INTO stream_test VALUES (repeat('a', 100000)); +COMMIT; +SELECT count(*) FROM pg_logical_slot_get_changes('regression_slot', NULL, NULL, 'include-xids', '0', 'skip-empty-xacts', '1', 'stream-changes', '1'); + DROP TABLE stream_test; SELECT pg_drop_replication_slot('regression_slot'); diff --git a/src/backend/replication/logical/reorderbuffer.c b/src/backend/replication/logical/reorderbuffer.c index 48a19ed4f6..71bf1e8e3d 100644 --- a/src/backend/replication/logical/reorderbuffer.c +++ b/src/backend/replication/logical/reorderbuffer.c @@ -3896,12 +3896,21 @@ ReorderBufferSerializeTXN(ReorderBuffer *rb, ReorderBufferTXN *txn) /* update the decoding stats */ UpdateDecodingStats((LogicalDecodingContext *) rb->private_data); + + /* + * Mark the transaction as serialized only if something was written, + * and only after the check above. final_lsn is advanced for each + * change written, so a transaction that wrote nothing, such as a + * subtransaction without decoded changes, would be marked with an + * invalid final_lsn, and ReorderBufferRestoreCleanup() would later + * fail on it. + */ + txn->txn_flags |= RBTXN_IS_SERIALIZED; } Assert(spilled == txn->nentries_mem); Assert(dlist_is_empty(&txn->changes)); txn->nentries_mem = 0; - txn->txn_flags |= RBTXN_IS_SERIALIZED; if (fd != -1) CloseTransientFile(fd);
v2-REL_17_STABLE.patch (text/x-patch, 3.2K) download diff --git a/contrib/test_decoding/expected/stream.out b/contrib/test_decoding/expected/stream.out index a76f77601e..a4b51c0051 100644 --- a/contrib/test_decoding/expected/stream.out +++ b/contrib/test_decoding/expected/stream.out @@ -128,6 +128,29 @@ SELECT count(*) FROM pg_logical_slot_get_changes('regression_slot', NULL, NULL, 5 (1 row) +-- bug #19839 +-- +-- A subtransaction without decoded changes must not be marked as serialized +-- when its parent transaction spills to disk during a partial TOAST change. +-- Otherwise, truncating the transaction after streaming fails on the +-- subtransaction's invalid final_lsn. +BEGIN; +SAVEPOINT s; +SELECT 1 FROM stream_test LIMIT 1 FOR UPDATE; + ?column? +---------- + 1 +(1 row) + +RELEASE SAVEPOINT s; +INSERT INTO stream_test VALUES (repeat('a', 100000)); +COMMIT; +SELECT count(*) FROM pg_logical_slot_get_changes('regression_slot', NULL, NULL, 'include-xids', '0', 'skip-empty-xacts', '1', 'stream-changes', '1'); + count +------- + 4 +(1 row) + DROP TABLE stream_test; SELECT pg_drop_replication_slot('regression_slot'); pg_drop_replication_slot diff --git a/contrib/test_decoding/sql/stream.sql b/contrib/test_decoding/sql/stream.sql index 7f43f0c2ab..22a4e1255b 100644 --- a/contrib/test_decoding/sql/stream.sql +++ b/contrib/test_decoding/sql/stream.sql @@ -59,5 +59,19 @@ ROLLBACK TO s1; COMMIT; SELECT count(*) FROM pg_logical_slot_get_changes('regression_slot', NULL, NULL, 'include-xids', '0', 'skip-empty-xacts', '1', 'stream-changes', '1'); +-- bug #19839 +-- +-- A subtransaction without decoded changes must not be marked as serialized +-- when its parent transaction spills to disk during a partial TOAST change. +-- Otherwise, truncating the transaction after streaming fails on the +-- subtransaction's invalid final_lsn. +BEGIN; +SAVEPOINT s; +SELECT 1 FROM stream_test LIMIT 1 FOR UPDATE; +RELEASE SAVEPOINT s; +INSERT INTO stream_test VALUES (repeat('a', 100000)); +COMMIT; +SELECT count(*) FROM pg_logical_slot_get_changes('regression_slot', NULL, NULL, 'include-xids', '0', 'skip-empty-xacts', '1', 'stream-changes', '1'); + DROP TABLE stream_test; SELECT pg_drop_replication_slot('regression_slot'); diff --git a/src/backend/replication/logical/reorderbuffer.c b/src/backend/replication/logical/reorderbuffer.c index 50989db6c4..1eb42b1396 100644 --- a/src/backend/replication/logical/reorderbuffer.c +++ b/src/backend/replication/logical/reorderbuffer.c @@ -4049,12 +4049,21 @@ ReorderBufferSerializeTXN(ReorderBuffer *rb, ReorderBufferTXN *txn) /* update the decoding stats */ UpdateDecodingStats((LogicalDecodingContext *) rb->private_data); + + /* + * Mark the transaction as serialized only if something was written, + * and only after the check above. final_lsn is advanced for each + * change written, so a transaction that wrote nothing, such as a + * subtransaction without decoded changes, would be marked with an + * invalid final_lsn, and ReorderBufferRestoreCleanup() would later + * fail on it. + */ + txn->txn_flags |= RBTXN_IS_SERIALIZED; } Assert(spilled == txn->nentries_mem); Assert(dlist_is_empty(&txn->changes)); txn->nentries_mem = 0; - txn->txn_flags |= RBTXN_IS_SERIALIZED; if (fd != -1) CloseTransientFile(fd);

view thread (7+ messages)  latest in thread

Message-ID: <ba2ec30c-e381-4422-913b-24afb4263fc8@gmail.com>
Permalink:  ../ba2ec30c-e381-4422-913b-24afb4263fc8@gmail.com/
Also on:    postgresql.org/message-id/ba2ec30c-e381-4422-913b-24afb4263fc8@gmail.com

 ·  · 

reply

Reply instructions:

You may reply publicly to this message via plain-text email
using any one of the following methods:

* Reply to all the recipients from your mail client; the link fills in
  the headers below: reply via email
  Some clients, webmail especially, drop In-Reply-To and References from
  a link, and the reply then starts a new thread: use the mbox below.

  To: pgsql-bugs@postgresql.org
  Cc: houzhijie22@gmail.com, bertranddrouvot.pg@gmail.com, 125atishprak@gmail.com
  Subject: Re: BUG #19839: Assert in ReorderBufferRestoreCleanup for aborted, spilled transaction
  In-Reply-To: <ba2ec30c-e381-4422-913b-24afb4263fc8@gmail.com>
  References: <19839-ead02c2208a3f8bd@postgresql.org> <CAGqfTtvQB4J0dJFnNMAF9LCAJhGBe5THd2R_=MxTHP5C5vwD=w@mail.gmail.com> <1da45744-968f-4f40-addf-dc085e928ec5@gmail.com> <avGmDSdgyIZB73Sl@bdtpg> <ba2ec30c-e381-4422-913b-24afb4263fc8@gmail.com>

* Save the following mbox file, import it into your mail client,
  and reply-to-all from there: mbox

This inbox is served by DDX for PostgreSQL; see mirroring instructions
for how to clone and mirror all data and code used for this inbox