pg.ddx.io  pgsql-hackers@postgresql.org mailing list archive  
help / color / mirror / Atom feed
From: Zhijie Hou <houzhijie22@gmail.com>
To: Masahiko Sawada <sawada.mshk@gmail.com>
To: Bharath Rupireddy <bharath.rupireddyforpostgres@gmail.com>
Cc: Amit Kapila <amit.kapila16@gmail.com>
Cc: Chao Li <li.evan.chao@gmail.com>
Cc: Fujii Masao <masao.fujii@gmail.com>
Cc: PostgreSQL-development <pgsql-hackers@postgresql.org>
Subject: Re: Add a hook for handling logical decoding messages on subscribers.
Date: Sat, 31 Oct 2026 20:35:15 +0800
Message-ID: <b9b5bd27-6416-48b4-8f73-c5a60e2c97e2@gmail.com> (raw)
In-Reply-To: <CAD21AoD5rwBPQv-_dGZEdsgzL7H1KywpsAd5HD3-DWKGQ5oTeQ@mail.gmail.com>
References: <CAD21AoCTNGiddikkUcDKj5QLnsg-51bpr-o6L-GTHWZL4ZFYtQ@mail.gmail.com>
	<CAD21AoBqVO_fwLkvxwzibohpMG-7n+7MqgkD9qRMGTVN6e8x7g@mail.gmail.com>
	<875ACE0E-20ED-4ECC-AEBF-733BFA46D97D@gmail.com>
	<CAD21AoDq0Z1Z+bZQxpTeXhu+TnTqunWycSq-8-b_BFQkLqTYAQ@mail.gmail.com>
	<CAD21AoBVgHk99_-AmE4-i4gR2-Gt8KENoAJ0rutQp91ko3yCgg@mail.gmail.com>
	<CALj2ACXY4Ht_sujNW75AMoLYVW1UDyA1WYYgh1VeDss6teAX=w@mail.gmail.com>
	<CAD21AoBUHTR72eqRdY9TEJem-aC4B5QFXvuUCJupqxxvQ9D_uw@mail.gmail.com>
	<CALj2ACUyBPL3t=0ECTO9W=OauRN+3qd--pmZC-KkKaENd0QdGw@mail.gmail.com>
	<CAD21AoDrvttNr0sOywJ7+1C1pivOgj8L3QuMqpziPDRopw+UDQ@mail.gmail.com>
	<CAA4eK1Lr2sKVarK4oqe_1ad9QsP0AEeh5cu0+i6DG0E=Jkdp-Q@mail.gmail.com>
	<CALj2ACVj+khUL=8Y3ukqJ6uF8cr0kmY6Q++fxJKYZ4P4bww=AA@mail.gmail.com>
	<CAD21AoAP9iYrJh4HkPYBfiQAncyw-ch7uiyXyeQ=yyyxU-V8xQ@mail.gmail.com>
	<CALj2ACVuHCZJ2vdrbB22LG77gCjwsw2DYcqNEy8GuHuKe_1Agg@mail.gmail.com>
	<CAD21AoD5rwBPQv-_dGZEdsgzL7H1KywpsAd5HD3-DWKGQ5oTeQ@mail.gmail.com>

Hi,

On Sat, Oct 31, 2026 at 06:34 AM Masahiko Sawada wrote:
>
> It seems to me worth adding a check for tablesync workers since both
> the apply workers and the tablesync workers call
> walrcv_startstreaming() and it would prevent mistakes by future
> changes.
> 
> I've attached the updated patch.

While reviewing how transactional logical messages interact with 
subtransaction aborts, I noticed an issue with streamed transactions. In 
non-streaming mode, it works as expected (i.e., the message is not 
applied if the subtransaction is aborted). However, in streaming mode, 
if a transactional message is emitted in a subtransaction that is later 
aborted, the STREAM ABORT message does not discard or roll back that 
logical message, so it still gets applied in both streaming=on and 
streaming=parallel modes.

I think the reason is that ReorderBufferApplyMessage() passes the 
top-level txn rather than change->txn to rb->stream_message(), so 
logical decoding always sends the top-level XID instead of the 
subtransaction XID when outputting a streamed message. Thus the apply 
worker cannot associate the message with the subtransaction and cannot 
roll it back properly.

Strictly speaking, this is an existing issue, but it becomes more 
visible once subscribers can handle logical messages via this hook. So 
it may be worth considering whether to fix it along with this patch or 
separately.

Attached 0002 adds a test diff in 001_basic.pl to show the issue when 
streaming a message in an aborted subtransaction. 0001 is a draft patch 
showing that passing the subtransaction of the message when streaming 
allows the message to be rolled back properly (we may not necessarily 
fix it in this exact style, just sharing it to show the behavior change).


-- 
Best Regards,
Zhijie Hou

Attachments: download all (2 files, .tar.gz)

nocfbot-0001-Fix-subtransaction-XID-in-streamed-logical-messag.patch (text/plain, 5.8K) download From 01d4606950996fd8d5d7e2af2b5a19a752b561c2 Mon Sep 17 00:00:00 2001 From: Zhijie Hou <houzhijie22@gmail.com> Date: Sat, 31 Oct 2026 19:55:25 +0800 Subject: [PATCH] Fix subtransaction XID in streamed logical messages. When streaming a transactional logical decoding message emitted inside a subtransaction, ReorderBufferApplyMessage() passed the top-level ReorderBufferTXN to stream_message_cb instead of the subtransaction's ReorderBufferTXN (change->txn). As a result, stream_message_cb_wrapper() and pgoutput_message() recorded the top-level XID instead of the subtransaction XID for the streamed message, unlike pgoutput_change() and pgoutput_truncate(). Any downstream consumer of streamed logical messages therefore could not associate the message with its subtransaction or roll it back upon receiving STREAM ABORT for a rolled-back savepoint. Fix this by passing change->txn to rb->stream_message() in ReorderBufferApplyMessage(). Also update pg_decode_stream_message() in test_decoding to look up output_plugin_private on the top-level transaction via rbtxn_get_toptxn(txn), matching pg_decode_stream_abort(). --- contrib/test_decoding/expected/stream.out | 28 +++++++++++++++++++ contrib/test_decoding/sql/stream.sql | 20 +++++++++++++ contrib/test_decoding/test_decoding.c | 11 ++++++-- .../replication/logical/reorderbuffer.c | 10 ++++++- 4 files changed, 66 insertions(+), 3 deletions(-) diff --git a/contrib/test_decoding/expected/stream.out b/contrib/test_decoding/expected/stream.out index 0ec5c933610..f34a07ba761 100644 --- a/contrib/test_decoding/expected/stream.out +++ b/contrib/test_decoding/expected/stream.out @@ -157,6 +157,34 @@ SELECT data FROM pg_logical_slot_get_changes('regression_slot', NULL, NULL, 'inc COMMIT (3 rows) +-- Verify that a transactional message emitted in a streamed subtransaction +-- is associated with the subtransaction's XID. +SET debug_logical_replication_streaming = immediate; +BEGIN; +SAVEPOINT sp; +SELECT 'msg_subxact' FROM pg_logical_emit_message(true, 'test', 'msg'); + ?column? +------------- + msg_subxact +(1 row) + +ROLLBACK TO SAVEPOINT sp; +COMMIT; +WITH changes AS ( + SELECT xid, data + FROM pg_logical_slot_get_changes('regression_slot', NULL, NULL, + 'include-xids', '0', 'skip-empty-xacts', '1', 'stream-changes', '1') +) +SELECT m.xid = a.xid AS subxid_matches +FROM changes m, changes a +WHERE m.data LIKE 'streaming message:%' + AND a.data = 'aborting streamed (sub)transaction'; + subxid_matches +---------------- + t +(1 row) + +RESET debug_logical_replication_streaming; DROP TABLE stream_test; SELECT pg_drop_replication_slot('regression_slot'); pg_drop_replication_slot diff --git a/contrib/test_decoding/sql/stream.sql b/contrib/test_decoding/sql/stream.sql index 5e45a8e9b64..ec68efed421 100644 --- a/contrib/test_decoding/sql/stream.sql +++ b/contrib/test_decoding/sql/stream.sql @@ -82,5 +82,25 @@ ROLLBACK; INSERT INTO stream_test VALUES ('after-abort'); SELECT data FROM pg_logical_slot_get_changes('regression_slot', NULL, NULL, 'include-xids', '0', 'skip-empty-xacts', '1'); +-- Verify that a transactional message emitted in a streamed subtransaction +-- is associated with the subtransaction's XID. +SET debug_logical_replication_streaming = immediate; +BEGIN; +SAVEPOINT sp; +SELECT 'msg_subxact' FROM pg_logical_emit_message(true, 'test', 'msg'); +ROLLBACK TO SAVEPOINT sp; +COMMIT; + +WITH changes AS ( + SELECT xid, data + FROM pg_logical_slot_get_changes('regression_slot', NULL, NULL, + 'include-xids', '0', 'skip-empty-xacts', '1', 'stream-changes', '1') +) +SELECT m.xid = a.xid AS subxid_matches +FROM changes m, changes a +WHERE m.data LIKE 'streaming message:%' + AND a.data = 'aborting streamed (sub)transaction'; +RESET debug_logical_replication_streaming; + DROP TABLE stream_test; SELECT pg_drop_replication_slot('regression_slot'); diff --git a/contrib/test_decoding/test_decoding.c b/contrib/test_decoding/test_decoding.c index 78185837aef..961dc475a12 100644 --- a/contrib/test_decoding/test_decoding.c +++ b/contrib/test_decoding/test_decoding.c @@ -961,11 +961,18 @@ pg_decode_stream_message(LogicalDecodingContext *ctx, if (transactional) { TestDecodingData *data = ctx->output_plugin_private; - TestDecodingTxnData *txndata = txn->output_plugin_private; + + /* + * stream_message_cb can be called for an individual subtransaction + * but we maintain the output_plugin_private only under the toptxn so + * if this is not the toptxn then fetch the toptxn. + */ + ReorderBufferTXN *toptxn = rbtxn_get_toptxn(txn); + TestDecodingTxnData *txndata = toptxn->output_plugin_private; if (data->skip_empty_xacts && !txndata->stream_wrote_changes) { - pg_output_stream_start(ctx, data, txn, false); + pg_output_stream_start(ctx, data, toptxn, false); } txndata->xact_wrote_changes = txndata->stream_wrote_changes = true; } diff --git a/src/backend/replication/logical/reorderbuffer.c b/src/backend/replication/logical/reorderbuffer.c index 46843985317..7d53879d18e 100644 --- a/src/backend/replication/logical/reorderbuffer.c +++ b/src/backend/replication/logical/reorderbuffer.c @@ -2104,10 +2104,18 @@ ReorderBufferApplyMessage(ReorderBuffer *rb, ReorderBufferTXN *txn, ReorderBufferChange *change, bool streaming) { if (streaming) - rb->stream_message(rb, txn, change->lsn, true, + { + /* + * Pass the change's transaction entry (which could be a + * subtransaction) so that the output plugin can associate the message + * with it and discard the message if the subtransaction is aborted + * later. + */ + rb->stream_message(rb, change->txn, change->lsn, true, change->data.msg.prefix, change->data.msg.message_size, change->data.msg.message); + } else rb->message(rb, txn, change->lsn, true, change->data.msg.prefix, -- 2.54.0 (Apple Git-157)
nocfbot-0002-Top-up-for-v6-0001-Test-streamed-subtransaction-.patch (text/plain, 1.6K) download From 4b377025db31174f8f2a43123f0b2c4205b240ed Mon Sep 17 00:00:00 2001 From: Zhijie Hou <houzhijie22@gmail.com> Date: Sat, 31 Oct 2026 16:28:45 +0800 Subject: [PATCH] Top-up for v6-0001: Test streamed subtransaction rollback with logical messages. Verify in 001_basic.pl that a transactional logical message emitted in a streamed subtransaction that later rolls back to a savepoint is not applied under both streaming = on and streaming = parallel. --- src/test/modules/test_logicalmsg_hooks/t/001_basic.pl | 7 +++++++ 1 file changed, 7 insertions(+) diff --git a/src/test/modules/test_logicalmsg_hooks/t/001_basic.pl b/src/test/modules/test_logicalmsg_hooks/t/001_basic.pl index a9ff6171ad6..59d38769c03 100644 --- a/src/test/modules/test_logicalmsg_hooks/t/001_basic.pl +++ b/src/test/modules/test_logicalmsg_hooks/t/001_basic.pl @@ -124,7 +124,10 @@ foreach my $mode ('on', 'parallel') $pub->safe_psql( 'postgres', qq{ BEGIN; +SAVEPOINT sp; +SELECT pg_logical_emit_message(true, 'streaming_subxact_aborted_$mode', 'not delivered'); INSERT INTO test SELECT generate_series(1000, 6000); +ROLLBACK TO SAVEPOINT sp; SELECT pg_logical_emit_message(true, 'streaming_$mode', 'streamed message'); COMMIT; }); @@ -136,6 +139,10 @@ COMMIT; q{SELECT stream_txns > 0 FROM pg_stat_replication_slots WHERE slot_name = 'sub'} ) or die "transaction was not streamed"; + is(logged_count("streaming_subxact_aborted_$mode"), + 0, + "message of an aborted streamed subtransaction is not applied (streaming = $mode)" + ); is(logged_count("streaming_$mode"), 1, "message in a streamed transaction is applied (streaming = $mode)"); -- 2.54.0 (Apple Git-157)

view thread (44+ messages)  latest in thread

Message-ID: <b9b5bd27-6416-48b4-8f73-c5a60e2c97e2@gmail.com>
Permalink:  ../b9b5bd27-6416-48b4-8f73-c5a60e2c97e2@gmail.com/
Also on:    postgresql.org/message-id/b9b5bd27-6416-48b4-8f73-c5a60e2c97e2@gmail.com

 ·  · 

reply

Reply instructions:

You may reply publicly to this message via plain-text email
using any one of the following methods:

* Reply to all the recipients from your mail client; the link fills in
  the headers below: reply via email
  Some clients, webmail especially, drop In-Reply-To and References from
  a link, and the reply then starts a new thread: use the mbox below.

  To: pgsql-hackers@postgresql.org
  Cc: houzhijie22@gmail.com, sawada.mshk@gmail.com, bharath.rupireddyforpostgres@gmail.com, amit.kapila16@gmail.com, li.evan.chao@gmail.com, masao.fujii@gmail.com
  Subject: Re: Add a hook for handling logical decoding messages on subscribers.
  In-Reply-To: <b9b5bd27-6416-48b4-8f73-c5a60e2c97e2@gmail.com>
  References: <CAD21AoCTNGiddikkUcDKj5QLnsg-51bpr-o6L-GTHWZL4ZFYtQ@mail.gmail.com> <875ACE0E-20ED-4ECC-AEBF-733BFA46D97D@gmail.com> <CAD21AoDq0Z1Z+bZQxpTeXhu+TnTqunWycSq-8-b_BFQkLqTYAQ@mail.gmail.com> <CAD21AoBVgHk99_-AmE4-i4gR2-Gt8KENoAJ0rutQp91ko3yCgg@mail.gmail.com> <CALj2ACXY4Ht_sujNW75AMoLYVW1UDyA1WYYgh1VeDss6teAX=w@mail.gmail.com> <CAD21AoBUHTR72eqRdY9TEJem-aC4B5QFXvuUCJupqxxvQ9D_uw@mail.gmail.com> <CALj2ACUyBPL3t=0ECTO9W=OauRN+3qd--pmZC-KkKaENd0QdGw@mail.gmail.com> <CAD21AoDrvttNr0sOywJ7+1C1pivOgj8L3QuMqpziPDRopw+UDQ@mail.gmail.com> <CAA4eK1Lr2sKVarK4oqe_1ad9QsP0AEeh5cu0+i6DG0E=Jkdp-Q@mail.gmail.com> <CALj2ACVj+khUL=8Y3ukqJ6uF8cr0kmY6Q++fxJKYZ4P4bww=AA@mail.gmail.com> <CAD21AoAP9iYrJh4HkPYBfiQAncyw-ch7uiyXyeQ=yyyxU-V8xQ@mail.gmail.com> <CALj2ACVuHCZJ2vdrbB22LG77gCjwsw2DYcqNEy8GuHuKe_1Agg@mail.gmail.com> <CAD21AoD5rwBPQv-_dGZEdsgzL7H1KywpsAd5HD3-DWKGQ5oTeQ@mail.gmail.com> <b9b5bd27-6416-48b4-8f73-c5a60e2c97e2@gmail.com>

* Save the following mbox file, import it into your mail client,
  and reply-to-all from there: mbox

This inbox is served by DDX for PostgreSQL; see mirroring instructions
for how to clone and mirror all data and code used for this inbox