pg.ddx.io  pgsql-hackers@postgresql.org mailing list archive  
help / color / mirror / Atom feed
Add a hook for handling logical decoding messages on subscribers.
44+ messages / 8 participants
[nested] [flat]

* Add a hook for handling logical decoding messages on subscribers.
@ 2026-06-19 22:33  Masahiko Sawada <sawada.mshk@gmail.com>
  0 siblings, 1 reply; 44+ messages in thread

From: Masahiko Sawada @ 2026-06-19 22:33 UTC (permalink / raw)
  To: pgsql-hackers

Hi all,

Commit ac4645c015 allows pgoutput to send logical decoding messages,
but it's limited to applications that use the pgoutput plugin -- the
built-in logical replication doesn't use it. I'd like to propose
introducing a hook to the logical replication message handling so that
extensions can plug in their own handling routine. This feature can be
used for extensions to implement DDL replication, function
replication, or trigger user-specific routines on the subscriber side.

I've attached the PoC patch; it adds a hook function, and adds a new
'message' subscription option that allows the user to request the
publisher to send logical decoding messages. Therefore, users need to
enable the 'message' option and set up the hook function at server
startup in order to receive the messages and trigger the hook
function.

I I went with a hook function in the patch. While it lets you chain
the multiple hook functions, providing the registration API might be
better, or other types of registry can also be considered.

Feedback is very welcome.

Regards,

--
Masahiko Sawada
Amazon Web Services: https://aws.amazon.com

Attachments:

v1-0001-Add-a-hook-for-handling-logical-messages-on-subsc.patch (text/x-patch, 91.8K) download From 546809304986e5cdcdee4ee41387c5edeb435fb2 Mon Sep 17 00:00:00 2001 From: Masahiko Sawada <sawada.mshk@gmail.com> Date: Mon, 8 Jun 2026 11:25:47 -0700 Subject: [PATCH v1] Add a hook for handling logical messages on subscribers. Add a "message" subscription option (default off) that asks the publisher's output plugin to stream messages, and dispatch each received message to a new LogicalMessageHandle_hook. When the option is off or no hook is installed, behavior is unchanged. This lets an extension act on logical messages on the subscriber -- for instance to carry deparsed DDL to a custom apply worker -- without modifying core apply code. Message handling honors transaction streaming and ALTER SUBSCRIPTION ... SKIP, as exercised by the new test_logicalmsg_hooks module. XXX Bump catalog version for the new pg_subscription.submessage column. --- src/backend/catalog/pg_subscription.c | 1 + src/backend/catalog/system_views.sql | 2 +- src/backend/commands/subscriptioncmds.c | 24 ++- .../libpqwalreceiver/libpqwalreceiver.c | 4 + src/backend/replication/logical/proto.c | 25 +++ src/backend/replication/logical/worker.c | 42 +++- src/bin/pg_dump/pg_dump.c | 16 +- src/bin/pg_dump/pg_dump.h | 1 + src/bin/psql/describe.c | 5 +- src/include/catalog/pg_subscription.h | 5 + src/include/replication/logicalproto.h | 13 ++ src/include/replication/walreceiver.h | 1 + src/include/replication/worker_internal.h | 4 + src/test/modules/Makefile | 1 + src/test/modules/meson.build | 1 + .../modules/test_logicalmsg_hooks/.gitignore | 4 + .../modules/test_logicalmsg_hooks/Makefile | 20 ++ .../modules/test_logicalmsg_hooks/meson.build | 28 +++ .../test_logicalmsg_hooks/t/001_basic.pl | 113 ++++++++++ .../test_logicalmsg_hooks.c | 40 ++++ src/test/regress/expected/subscription.out | 202 ++++++++++-------- src/test/regress/sql/subscription.sql | 17 ++ src/tools/pgindent/typedefs.list | 1 + 23 files changed, 469 insertions(+), 101 deletions(-) create mode 100644 src/test/modules/test_logicalmsg_hooks/.gitignore create mode 100644 src/test/modules/test_logicalmsg_hooks/Makefile create mode 100644 src/test/modules/test_logicalmsg_hooks/meson.build create mode 100644 src/test/modules/test_logicalmsg_hooks/t/001_basic.pl create mode 100644 src/test/modules/test_logicalmsg_hooks/test_logicalmsg_hooks.c diff --git a/src/backend/catalog/pg_subscription.c b/src/backend/catalog/pg_subscription.c index b5cb301db88..f14649a245b 100644 --- a/src/backend/catalog/pg_subscription.c +++ b/src/backend/catalog/pg_subscription.c @@ -132,6 +132,7 @@ GetSubscription(Oid subid, bool missing_ok, bool conninfo_needed, sub->retaindeadtuples = subform->subretaindeadtuples; sub->maxretention = subform->submaxretention; sub->retentionactive = subform->subretentionactive; + sub->message = subform->submessage; if (conninfo_needed) { diff --git a/src/backend/catalog/system_views.sql b/src/backend/catalog/system_views.sql index 8f129baec90..6ebaa93b1ff 100644 --- a/src/backend/catalog/system_views.sql +++ b/src/backend/catalog/system_views.sql @@ -1528,7 +1528,7 @@ GRANT SELECT (oid, subdbid, subskiplsn, subname, subowner, subenabled, subpasswordrequired, subrunasowner, subfailover, subretaindeadtuples, submaxretention, subretentionactive, subserver, subslotname, subsynccommit, subwalrcvtimeout, - subpublications, suborigin) + subpublications, suborigin, submessage) ON pg_subscription TO public; CREATE VIEW pg_stat_subscription_stats AS diff --git a/src/backend/commands/subscriptioncmds.c b/src/backend/commands/subscriptioncmds.c index ee06a726f42..5183ab01156 100644 --- a/src/backend/commands/subscriptioncmds.c +++ b/src/backend/commands/subscriptioncmds.c @@ -79,6 +79,7 @@ #define SUBOPT_WAL_RECEIVER_TIMEOUT 0x00010000 #define SUBOPT_LSN 0x00020000 #define SUBOPT_ORIGIN 0x00040000 +#define SUBOPT_MESSAGE 0x00080000 /* check if the 'val' has 'bits' set */ #define IsSet(val, bits) (((val) & (bits)) == (bits)) @@ -105,6 +106,7 @@ typedef struct SubOpts bool runasowner; bool failover; bool retaindeadtuples; + bool message; int32 maxretention; char *origin; XLogRecPtr lsn; @@ -192,6 +194,8 @@ parse_subscription_options(ParseState *pstate, List *stmt_options, opts->failover = false; if (IsSet(supported_opts, SUBOPT_RETAIN_DEAD_TUPLES)) opts->retaindeadtuples = false; + if (IsSet(supported_opts, SUBOPT_MESSAGE)) + opts->message = false; if (IsSet(supported_opts, SUBOPT_MAX_RETENTION_DURATION)) opts->maxretention = 0; if (IsSet(supported_opts, SUBOPT_ORIGIN)) @@ -348,6 +352,15 @@ parse_subscription_options(ParseState *pstate, List *stmt_options, opts->specified_opts |= SUBOPT_RETAIN_DEAD_TUPLES; opts->retaindeadtuples = defGetBoolean(defel); } + else if (IsSet(supported_opts, SUBOPT_MESSAGE) && + strcmp(defel->defname, "message") == 0) + { + if (IsSet(opts->specified_opts, SUBOPT_MESSAGE)) + errorConflictingDefElem(defel, pstate); + + opts->specified_opts |= SUBOPT_MESSAGE; + opts->message = defGetBoolean(defel); + } else if (IsSet(supported_opts, SUBOPT_MAX_RETENTION_DURATION) && strcmp(defel->defname, "max_retention_duration") == 0) { @@ -674,7 +687,7 @@ CreateSubscription(ParseState *pstate, CreateSubscriptionStmt *stmt, SUBOPT_RUN_AS_OWNER | SUBOPT_FAILOVER | SUBOPT_RETAIN_DEAD_TUPLES | SUBOPT_MAX_RETENTION_DURATION | - SUBOPT_WAL_RECEIVER_TIMEOUT | SUBOPT_ORIGIN); + SUBOPT_WAL_RECEIVER_TIMEOUT | SUBOPT_ORIGIN | SUBOPT_MESSAGE); parse_subscription_options(pstate, stmt->options, supported_opts, &opts); /* @@ -824,6 +837,7 @@ CreateSubscription(ParseState *pstate, CreateSubscriptionStmt *stmt, values[Anum_pg_subscription_subfailover - 1] = BoolGetDatum(opts.failover); values[Anum_pg_subscription_subretaindeadtuples - 1] = BoolGetDatum(opts.retaindeadtuples); + values[Anum_pg_subscription_submessage - 1] = BoolGetDatum(opts.message); values[Anum_pg_subscription_submaxretention - 1] = Int32GetDatum(opts.maxretention); values[Anum_pg_subscription_subretentionactive - 1] = @@ -1499,7 +1513,7 @@ AlterSubscription(ParseState *pstate, AlterSubscriptionStmt *stmt, SUBOPT_RETAIN_DEAD_TUPLES | SUBOPT_MAX_RETENTION_DURATION | SUBOPT_WAL_RECEIVER_TIMEOUT | - SUBOPT_ORIGIN); + SUBOPT_ORIGIN | SUBOPT_MESSAGE); break; case ALTER_SUBSCRIPTION_ENABLED: @@ -1858,6 +1872,12 @@ AlterSubscription(ParseState *pstate, AlterSubscriptionStmt *stmt, replaces[Anum_pg_subscription_subwalrcvtimeout - 1] = true; } + if (IsSet(opts.specified_opts, SUBOPT_MESSAGE)) + { + values[Anum_pg_subscription_submessage - 1] = BoolGetDatum(opts.message); + replaces[Anum_pg_subscription_submessage - 1] = true; + } + update_tuple = true; break; } diff --git a/src/backend/replication/libpqwalreceiver/libpqwalreceiver.c b/src/backend/replication/libpqwalreceiver/libpqwalreceiver.c index 5376519fea5..490e0590970 100644 --- a/src/backend/replication/libpqwalreceiver/libpqwalreceiver.c +++ b/src/backend/replication/libpqwalreceiver/libpqwalreceiver.c @@ -614,6 +614,10 @@ libpqrcv_startstreaming(WalReceiverConn *conn, appendQuotedLiteral(&cmd, options->proto.logical.origin); } + if (options->proto.logical.messages && + PQserverVersion(conn->streamConn) >= 140000) + appendStringInfo(&cmd, ", messages 'on'"); + pubnames = options->proto.logical.publication_names; pubnames_str = stringlist_to_identifierstr(pubnames); appendStringInfoString(&cmd, ", publication_names "); diff --git a/src/backend/replication/logical/proto.c b/src/backend/replication/logical/proto.c index 86ad97cd937..af8ee26ed0c 100644 --- a/src/backend/replication/logical/proto.c +++ b/src/backend/replication/logical/proto.c @@ -660,6 +660,31 @@ logicalrep_write_message(StringInfo out, TransactionId xid, XLogRecPtr lsn, pq_sendbytes(out, message, sz); } +/* + * Read MESSAGE from stream. + */ +void +logicalrep_read_message(StringInfo in, LogicalRepMessageData *msg_data) +{ + Size len; + char *msg; + + msg_data->flags = pq_getmsgint(in, 1); + msg_data->lsn = pq_getmsgint64(in); + msg_data->prefix = pstrdup(pq_getmsgstring(in)); + + /* read message length */ + len = pq_getmsgint(in, 4); + msg_data->message_size = len; + + /* and data */ + msg = palloc(len + 1); + pq_copymsgbytes(in, msg, len); + + msg[len] = '\0'; + msg_data->message = msg; +} + /* * Write relation description to the output stream. */ diff --git a/src/backend/replication/logical/worker.c b/src/backend/replication/logical/worker.c index 7799266c614..3457ed7b379 100644 --- a/src/backend/replication/logical/worker.c +++ b/src/backend/replication/logical/worker.c @@ -549,6 +549,9 @@ typedef struct ApplySubXactData static ApplySubXactData subxact_data = {0, 0, InvalidTransactionId, NULL}; +/* Hook for plugins to get control in handling logical decoding messages */ +LogicalRepMessageHandle_hook_type LogicalRepMessageHandle_hook = NULL; + static inline void subxact_filename(char *path, Oid subid, TransactionId xid); static inline void changes_filename(char *path, Oid subid, TransactionId xid); @@ -1694,6 +1697,34 @@ apply_handle_origin(StringInfo s) errmsg_internal("ORIGIN message sent out of order"))); } +/* + * Handle MESSAGE message. + * + * Invoke a hook function if set. + */ +static void +apply_handle_message(StringInfo s) +{ + LogicalRepMessageData msg; + + if (!LogicalRepMessageHandle_hook) + return; + + /* + * Logical messages are handled only the (parallel) apply workers + */ + if (am_tablesync_worker() || am_sequencesync_worker()) + return; + + if (is_skipping_changes() || + handle_streamed_transaction(LOGICAL_REP_MSG_MESSAGE, s)) + return; + + logicalrep_read_message(s, &msg); + + (*LogicalRepMessageHandle_hook) (&msg); +} + /* * Initialize fileset (if not already done). * @@ -3846,12 +3877,7 @@ apply_dispatch(StringInfo s) break; case LOGICAL_REP_MSG_MESSAGE: - - /* - * Logical replication does not use generic logical messages yet. - * Although, it could be used by other applications that use this - * output plugin. - */ + apply_handle_message(s); break; case LOGICAL_REP_MSG_STREAM_START: @@ -5128,7 +5154,8 @@ maybe_reread_subscription(void) newsub->passwordrequired != MySubscription->passwordrequired || strcmp(newsub->origin, MySubscription->origin) != 0 || newsub->owner != MySubscription->owner || - !equal(newsub->publications, MySubscription->publications)) + !equal(newsub->publications, MySubscription->publications) || + newsub->message != MySubscription->message) { if (am_parallel_apply_worker()) ereport(LOG, @@ -5614,6 +5641,7 @@ set_stream_options(WalRcvStreamOptions *options, options->proto.logical.twophase = false; options->proto.logical.origin = pstrdup(MySubscription->origin); + options->proto.logical.messages = MySubscription->message; } /* diff --git a/src/bin/pg_dump/pg_dump.c b/src/bin/pg_dump/pg_dump.c index c56437d6057..4441b14082c 100644 --- a/src/bin/pg_dump/pg_dump.c +++ b/src/bin/pg_dump/pg_dump.c @@ -5184,6 +5184,7 @@ getSubscriptions(Archive *fout) int i_subfailover; int i_subretaindeadtuples; int i_submaxretention; + int i_submessage; int i, ntups; @@ -5282,9 +5283,14 @@ getSubscriptions(Archive *fout) " '-1' AS subwalrcvtimeout,\n"); if (fout->remoteVersion >= 190000) - appendPQExpBufferStr(query, " fs.srvname AS subservername\n"); + appendPQExpBufferStr(query, " fs.srvname AS subservername,\n"); else - appendPQExpBufferStr(query, " NULL AS subservername\n"); + appendPQExpBufferStr(query, " NULL AS subservername,\n"); + + if (fout->remoteVersion >= 190000) + appendPQExpBufferStr(query, " s.submessage\n"); + else + appendPQExpBufferStr(query, "false AS submessage\n"); appendPQExpBufferStr(query, "FROM pg_subscription s\n"); @@ -5333,6 +5339,7 @@ getSubscriptions(Archive *fout) i_subpublications = PQfnumber(res, "subpublications"); i_suborigin = PQfnumber(res, "suborigin"); i_suboriginremotelsn = PQfnumber(res, "suboriginremotelsn"); + i_submessage = PQfnumber(res, "submessage"); subinfo = pg_malloc_array(SubscriptionInfo, ntups); @@ -5390,6 +5397,8 @@ getSubscriptions(Archive *fout) else subinfo[i].suboriginremotelsn = pg_strdup(PQgetvalue(res, i, i_suboriginremotelsn)); + subinfo[i].submessage = + (strcmp(PQgetvalue(res, i, i_submessage), "t") == 0); /* Decide whether we want to dump it */ selectDumpableObject(&(subinfo[i].dobj), fout); @@ -5642,6 +5651,9 @@ dumpSubscription(Archive *fout, const SubscriptionInfo *subinfo) if (subinfo->subretaindeadtuples) appendPQExpBufferStr(query, ", retain_dead_tuples = true"); + if (subinfo->submessage) + appendPQExpBufferStr(query, ", message = true"); + if (subinfo->submaxretention) appendPQExpBuffer(query, ", max_retention_duration = %d", subinfo->submaxretention); diff --git a/src/bin/pg_dump/pg_dump.h b/src/bin/pg_dump/pg_dump.h index 5a6726d8b12..30bd1575a26 100644 --- a/src/bin/pg_dump/pg_dump.h +++ b/src/bin/pg_dump/pg_dump.h @@ -721,6 +721,7 @@ typedef struct _SubscriptionInfo bool subrunasowner; bool subfailover; bool subretaindeadtuples; + bool submessage; int submaxretention; char *subservername; char *subconninfo; diff --git a/src/bin/psql/describe.c b/src/bin/psql/describe.c index af3935b0078..ac5a2c1835b 100644 --- a/src/bin/psql/describe.c +++ b/src/bin/psql/describe.c @@ -7093,7 +7093,7 @@ describeSubscriptions(const char *pattern, bool verbose) printQueryOpt myopt = pset.popt; static const bool translate_columns[] = {false, false, false, false, false, false, false, false, false, false, false, false, false, false, - false, false, false, false, false, false, false}; + false, false, false, false, false, false, false, false}; if (pset.sversion < 100000) { @@ -7179,6 +7179,9 @@ describeSubscriptions(const char *pattern, bool verbose) appendPQExpBuffer(&buf, ", subretentionactive AS \"%s\"\n", gettext_noop("Retention active")); + appendPQExpBuffer(&buf, + ", submessage AS \"%s\"\n", + gettext_noop("Message")); } appendPQExpBuffer(&buf, diff --git a/src/include/catalog/pg_subscription.h b/src/include/catalog/pg_subscription.h index 48944201889..cdda8265573 100644 --- a/src/include/catalog/pg_subscription.h +++ b/src/include/catalog/pg_subscription.h @@ -83,6 +83,9 @@ CATALOG(pg_subscription,6100,SubscriptionRelationId) BKI_SHARED_RELATION BKI_ROW bool subretaindeadtuples; /* True if dead tuples useful for * conflict detection are retained */ + bool submessage; /* True if the subscription wants to receive + * logical messages. */ + int32 submaxretention; /* The maximum duration (in milliseconds) * for which information useful for * conflict detection can be retained */ @@ -171,6 +174,8 @@ typedef struct Subscription List *publications; /* List of publication names to subscribe to */ char *origin; /* Only publish data originating from the * specified origin */ + bool message; /* True if the subscription wants to receive + * logical messages */ } Subscription; #ifdef EXPOSE_TO_CLIENT_CODE diff --git a/src/include/replication/logicalproto.h b/src/include/replication/logicalproto.h index 058a955e20c..85add073202 100644 --- a/src/include/replication/logicalproto.h +++ b/src/include/replication/logicalproto.h @@ -191,6 +191,18 @@ typedef struct LogicalRepStreamAbortData TimestampTz abort_time; } LogicalRepStreamAbortData; +/* + * Logical decoding message information + */ +typedef struct LogicalRepMessageData +{ + uint8 flags; + XLogRecPtr lsn; + const char *prefix; + Size message_size; /* length of message in bytes */ + const char *message; /* payload; may contain embedded nulls */ +} LogicalRepMessageData; + extern void logicalrep_write_begin(StringInfo out, ReorderBufferTXN *txn); extern void logicalrep_read_begin(StringInfo in, LogicalRepBeginData *begin_data); @@ -248,6 +260,7 @@ extern List *logicalrep_read_truncate(StringInfo in, bool *cascade, bool *restart_seqs); extern void logicalrep_write_message(StringInfo out, TransactionId xid, XLogRecPtr lsn, bool transactional, const char *prefix, Size sz, const char *message); +extern void logicalrep_read_message(StringInfo in, LogicalRepMessageData *msg_data); extern void logicalrep_write_rel(StringInfo out, TransactionId xid, Relation rel, Bitmapset *columns, PublishGencolsType include_gencols_type); diff --git a/src/include/replication/walreceiver.h b/src/include/replication/walreceiver.h index 47c07574d4d..44a08cc2170 100644 --- a/src/include/replication/walreceiver.h +++ b/src/include/replication/walreceiver.h @@ -188,6 +188,7 @@ typedef struct * prepare time */ char *origin; /* Only publish data originating from the * specified origin */ + bool messages; /* Logical message */ } logical; } proto; } WalRcvStreamOptions; diff --git a/src/include/replication/worker_internal.h b/src/include/replication/worker_internal.h index 745b7d9e969..b07042846cb 100644 --- a/src/include/replication/worker_internal.h +++ b/src/include/replication/worker_internal.h @@ -18,6 +18,7 @@ #include "miscadmin.h" #include "replication/logicalrelation.h" #include "replication/walreceiver.h" +#include "replication/logicalproto.h" #include "storage/buffile.h" #include "storage/fileset.h" #include "storage/shm_mq.h" @@ -255,6 +256,9 @@ extern PGDLLIMPORT bool InitializingApplyWorker; extern PGDLLIMPORT List *table_states_not_ready; +typedef void (*LogicalRepMessageHandle_hook_type) (LogicalRepMessageData *msg); +extern PGDLLIMPORT LogicalRepMessageHandle_hook_type LogicalRepMessageHandle_hook; + extern void logicalrep_worker_attach(int slot); extern LogicalRepWorker *logicalrep_worker_find(LogicalRepWorkerType wtype, Oid subid, Oid relid, diff --git a/src/test/modules/Makefile b/src/test/modules/Makefile index 0a74ab5c86f..603fe1ff907 100644 --- a/src/test/modules/Makefile +++ b/src/test/modules/Makefile @@ -36,6 +36,7 @@ SUBDIRS = \ test_integerset \ test_json_parser \ test_lfind \ + test_logicalmsg_hooks \ test_lwlock_tranches \ test_misc \ test_oat_hooks \ diff --git a/src/test/modules/meson.build b/src/test/modules/meson.build index 4bca42bb370..5f6c0cae77a 100644 --- a/src/test/modules/meson.build +++ b/src/test/modules/meson.build @@ -37,6 +37,7 @@ subdir('test_int128') subdir('test_integerset') subdir('test_json_parser') subdir('test_lfind') +subdir('test_logicalmsg_hooks') subdir('test_lwlock_tranches') subdir('test_misc') subdir('test_oat_hooks') diff --git a/src/test/modules/test_logicalmsg_hooks/.gitignore b/src/test/modules/test_logicalmsg_hooks/.gitignore new file mode 100644 index 00000000000..5dcb3ff9723 --- /dev/null +++ b/src/test/modules/test_logicalmsg_hooks/.gitignore @@ -0,0 +1,4 @@ +# Generated subdirectories +/log/ +/results/ +/tmp_check/ diff --git a/src/test/modules/test_logicalmsg_hooks/Makefile b/src/test/modules/test_logicalmsg_hooks/Makefile new file mode 100644 index 00000000000..af129009b8a --- /dev/null +++ b/src/test/modules/test_logicalmsg_hooks/Makefile @@ -0,0 +1,20 @@ +# src/test/modules/test_logicalmsg_hooks/Makefile + +MODULE_big = test_logicalmsg_hooks +OBJS = \ + $(WIN32RES) \ + test_logicalmsg_hooks.o +PGFILEDESC = "test_logicalmsg_hooks - test logical replication message hooks" + +TAP_TESTS = 1 + +ifdef USE_PGXS +PG_CONFIG = pg_config +PGXS := $(shell $(PG_CONFIG) --pgxs) +include $(PGXS) +else +subdir = src/test/modules/test_logicalmsg_hooks +top_builddir = ../../../.. +include $(top_builddir)/src/Makefile.global +include $(top_srcdir)/contrib/contrib-global.mk +endif diff --git a/src/test/modules/test_logicalmsg_hooks/meson.build b/src/test/modules/test_logicalmsg_hooks/meson.build new file mode 100644 index 00000000000..8084d82e47b --- /dev/null +++ b/src/test/modules/test_logicalmsg_hooks/meson.build @@ -0,0 +1,28 @@ +# Copyright (c) 2022-2026, PostgreSQL Global Development Group + +test_logicalmsg_hooks_sources = files( + 'test_logicalmsg_hooks.c', +) + +if host_system == 'windows' + test_logicalmsg_hooks_sources += rc_lib_gen.process(win32ver_rc, extra_args: [ + '--NAME', 'test_logicalmsg_hooks', + '--FILEDESC', 'test_logicalmsg_hooks - test logical message hooks',]) +endif + +test_logicalmsg_hooks = shared_module('test_logicalmsg_hooks', + test_logicalmsg_hooks_sources, + kwargs: pg_test_mod_args, +) +test_install_libs += test_logicalmsg_hooks + +tests += { + 'name': 'test_logicalmsg_hooks', + 'sd': meson.current_source_dir(), + 'bd': meson.current_build_dir(), + 'tap': { + 'tests': [ + 't/001_basic.pl', + ], + }, +} diff --git a/src/test/modules/test_logicalmsg_hooks/t/001_basic.pl b/src/test/modules/test_logicalmsg_hooks/t/001_basic.pl new file mode 100644 index 00000000000..64ca51d1786 --- /dev/null +++ b/src/test/modules/test_logicalmsg_hooks/t/001_basic.pl @@ -0,0 +1,113 @@ +# Copyright (c) 2021-2026, PostgreSQL Global Development Group + +use strict; +use warnings FATAL => 'all'; + +use PostgreSQL::Test::Cluster; +use PostgreSQL::Test::Utils; +use Test::More; + +my $pub = PostgreSQL::Test::Cluster->new('publisher'); +$pub->init(allows_streaming => 'logical'); +$pub->start; + +my $sub = PostgreSQL::Test::Cluster->new('subscriber'); +$sub->init(allows_streaming => 'logical'); +$sub->append_conf( + 'postgresql.conf', q{ +shared_preload_libraries = 'test_logicalmsg_hooks' +}); +$sub->start; + +$pub->safe_psql( + 'postgres', + qq{ +CREATE TABLE test (a int); +INSERT INTO test VALUES (1); +CREATE PUBLICATION pub FOR ALL TABLES; +}); + +my $pub_connstr = $pub->connstr . ' dbname=postgres'; + +$sub->safe_psql( + 'postgres', + qq{ +CREATE TABLE test (a int primary key); +CREATE SUBSCRIPTION sub CONNECTION '$pub_connstr' PUBLICATION pub WITH (message = true, disable_on_error = true) +}); + +$sub->wait_for_subscription_sync($pub, 'sub'); + +my $log_location = -s $sub->logfile; + +# Emit a logical message and verify that the subscriber processed it in the +# hook function. +$pub->safe_psql( + 'postgres', + qq{ +SELECT pg_logical_emit_message(true, 'test_prefix', 'test_mesasge'); +}); +$pub->wait_for_catchup('sub'); +$sub->wait_for_log( + qr/LOG[^\n]+received message: LSN [[:xdigit:]]+\/[[:xdigit:]]+, prefix: test_prefix, message: test_mesasge/, + $log_location); +ok(1, "logical message is correctly handled on the subscriber"); + +$log_location = -s $sub->logfile; + +# INSERT a tuple with a=1 conflicts on the subscriber. Test that ALTER SUBSCRIPTION ... SKIP +# the transacitonal message as well. +$pub->safe_psql( + 'postgres', + q{ +BEGIN; +INSERT INTO test VALUES (1); +SELECT pg_logical_emit_message(true, 'conflict', 'should be skipped'); +COMMIT; +}); + +# Wait until a conflict occurs on the subscriber. +$sub->poll_query_until('postgres', + q{SELECT subenabled = FALSE FROM pg_subscription WHERE subname = 'sub'}); + +# Get the finish LSN of the error transaction. +my $contents = slurp_file($sub->logfile, $log_location); +$contents =~ + qr/conflict detected on relation "public.test".*\n.*DETAIL:.*Could not apply remote change.*\n.*Key already exists in unique index "test_pkey", modified in transaction \d+: key .*, local row .*\n.*CONTEXT:.* for replication target relation "public.test" in transaction \d+, finished at ([[:xdigit:]]+\/[[:xdigit:]]+)/m + or die "could not get error-LSN"; +my $lsn = $1; + +$log_location = -s $sub->logfile; + +# Set skip LSN and re-enable the subscription. +$sub->safe_psql('postgres', qq{ALTER SUBSCRIPTION sub SKIP (lsn = '$lsn');}); +$sub->safe_psql('postgres', "ALTER SUBSCRIPTION sub ENABLE"); + +# Wait for the failed transaction to be skipped +$sub->poll_query_until('postgres', + q{SELECT subskiplsn = '0/0' FROM pg_subscription WHERE subname = 'sub'}); + +ok( !$sub->log_contains( + qr/LOG[^\n]+received message: LSN .*, prefix: conflict, message: should be skipped/ + ), + "logical message is skipped by ALTER SUBSCRIPTION SKIP"); + +# Test that a non-transactional message is applied even if the transaction rolls back. +$log_location = -s $sub->logfile; +$pub->safe_psql( + 'postgres', + q{ +BEGIN; +SELECT pg_logical_emit_message(false, 'test_prefix', 'non-transactional message'); +ROLLBACK; +}); +$pub->wait_for_catchup('sub'); +$sub->wait_for_log( + qr/LOG[^\n]+received message: LSN [[:xdigit:]]+\/[[:xdigit:]]+, prefix: test_prefix, message: non-transactional message/, + $log_location); +ok(1, "logical message is correctly handled on the subscriber"); + + +$pub->stop; +$sub->stop; +done_testing(); diff --git a/src/test/modules/test_logicalmsg_hooks/test_logicalmsg_hooks.c b/src/test/modules/test_logicalmsg_hooks/test_logicalmsg_hooks.c new file mode 100644 index 00000000000..9167043508e --- /dev/null +++ b/src/test/modules/test_logicalmsg_hooks/test_logicalmsg_hooks.c @@ -0,0 +1,40 @@ +/*-------------------------------------------------------------------------- + * + * test_logicalmsg_hooks.c + * Code for testing LogicalMessageHandle_hook + * + * Portions Copyright (c) 1996-2026, PostgreSQL Global Development Group + * Portions Copyright (c) 1994, Regents of the University of California + * + * IDENTIFICATION + * src/test/modules/test_logicalmsg_hooks/test_logicalmsg_hooks.c + * + * ------------------------------------------------------------------------- + */ + +#include "postgres.h" + +#include "replication/worker_internal.h" + +PG_MODULE_MAGIC; + +static void test_logical_message_handler(LogicalRepMessageData *msg); + +/* + * Module load callback + */ +void +_PG_init(void) +{ + LogicalRepMessageHandle_hook = &test_logical_message_handler; +} + +void +test_logical_message_handler(LogicalRepMessageData *msg) +{ + ereport(LOG, + (errmsg("received message: LSN %X/%08X, prefix: %s, message: %s", + LSN_FORMAT_ARGS(msg->lsn), + msg->prefix, + msg->message))); +} diff --git a/src/test/regress/expected/subscription.out b/src/test/regress/expected/subscription.out index 8dbfac66326..29c2fac6504 100644 --- a/src/test/regress/expected/subscription.out +++ b/src/test/regress/expected/subscription.out @@ -139,18 +139,18 @@ CREATE SUBSCRIPTION regress_testsub4 CONNECTION 'dbname=regress_doesnotexist' PU WARNING: subscription was created, but is not connected HINT: To initiate replication, you must manually create the replication slot, enable the subscription, and alter the subscription to refresh publications. \dRs+ regress_testsub4 - List of subscriptions - Name | Owner | Enabled | Publication | Binary | Streaming | Two-phase commit | Disable on error | Origin | Password required | Run as owner? | Failover | Server | Retain dead tuples | Max retention duration | Retention active | Synchronous commit | Conninfo | Receiver timeout | Skip LSN | Description -------------------+---------------------------+---------+-------------+--------+-----------+------------------+------------------+--------+-------------------+---------------+----------+--------+--------------------+------------------------+------------------+--------------------+-----------------------------+------------------+------------+------------- - regress_testsub4 | regress_subscription_user | f | {testpub} | f | parallel | d | f | none | t | f | f | | f | 0 | f | off | dbname=regress_doesnotexist | -1 | 0/00000000 | + List of subscriptions + Name | Owner | Enabled | Publication | Binary | Streaming | Two-phase commit | Disable on error | Origin | Password required | Run as owner? | Failover | Server | Retain dead tuples | Max retention duration | Retention active | Message | Synchronous commit | Conninfo | Receiver timeout | Skip LSN | Description +------------------+---------------------------+---------+-------------+--------+-----------+------------------+------------------+--------+-------------------+---------------+----------+--------+--------------------+------------------------+------------------+---------+--------------------+-----------------------------+------------------+------------+------------- + regress_testsub4 | regress_subscription_user | f | {testpub} | f | parallel | d | f | none | t | f | f | | f | 0 | f | f | off | dbname=regress_doesnotexist | -1 | 0/00000000 | (1 row) ALTER SUBSCRIPTION regress_testsub4 SET (origin = any); \dRs+ regress_testsub4 - List of subscriptions - Name | Owner | Enabled | Publication | Binary | Streaming | Two-phase commit | Disable on error | Origin | Password required | Run as owner? | Failover | Server | Retain dead tuples | Max retention duration | Retention active | Synchronous commit | Conninfo | Receiver timeout | Skip LSN | Description -------------------+---------------------------+---------+-------------+--------+-----------+------------------+------------------+--------+-------------------+---------------+----------+--------+--------------------+------------------------+------------------+--------------------+-----------------------------+------------------+------------+------------- - regress_testsub4 | regress_subscription_user | f | {testpub} | f | parallel | d | f | any | t | f | f | | f | 0 | f | off | dbname=regress_doesnotexist | -1 | 0/00000000 | + List of subscriptions + Name | Owner | Enabled | Publication | Binary | Streaming | Two-phase commit | Disable on error | Origin | Password required | Run as owner? | Failover | Server | Retain dead tuples | Max retention duration | Retention active | Message | Synchronous commit | Conninfo | Receiver timeout | Skip LSN | Description +------------------+---------------------------+---------+-------------+--------+-----------+------------------+------------------+--------+-------------------+---------------+----------+--------+--------------------+------------------------+------------------+---------+--------------------+-----------------------------+------------------+------------+------------- + regress_testsub4 | regress_subscription_user | f | {testpub} | f | parallel | d | f | any | t | f | f | | f | 0 | f | f | off | dbname=regress_doesnotexist | -1 | 0/00000000 | (1 row) DROP SUBSCRIPTION regress_testsub3; @@ -237,10 +237,10 @@ ALTER SUBSCRIPTION regress_testsub CONNECTION 'foobar'; ERROR: invalid connection string syntax: missing "=" after "foobar" in connection info string \dRs+ - List of subscriptions - Name | Owner | Enabled | Publication | Binary | Streaming | Two-phase commit | Disable on error | Origin | Password required | Run as owner? | Failover | Server | Retain dead tuples | Max retention duration | Retention active | Synchronous commit | Conninfo | Receiver timeout | Skip LSN | Description ------------------+---------------------------+---------+-------------+--------+-----------+------------------+------------------+--------+-------------------+---------------+----------+--------+--------------------+------------------------+------------------+--------------------+-----------------------------+------------------+------------+------------------- - regress_testsub | regress_subscription_user | f | {testpub} | f | parallel | d | f | any | t | f | f | | f | 0 | f | off | dbname=regress_doesnotexist | -1 | 0/00000000 | test subscription + List of subscriptions + Name | Owner | Enabled | Publication | Binary | Streaming | Two-phase commit | Disable on error | Origin | Password required | Run as owner? | Failover | Server | Retain dead tuples | Max retention duration | Retention active | Message | Synchronous commit | Conninfo | Receiver timeout | Skip LSN | Description +-----------------+---------------------------+---------+-------------+--------+-----------+------------------+------------------+--------+-------------------+---------------+----------+--------+--------------------+------------------------+------------------+---------+--------------------+-----------------------------+------------------+------------+------------------- + regress_testsub | regress_subscription_user | f | {testpub} | f | parallel | d | f | any | t | f | f | | f | 0 | f | f | off | dbname=regress_doesnotexist | -1 | 0/00000000 | test subscription (1 row) ALTER SUBSCRIPTION regress_testsub SET PUBLICATION testpub2, testpub3 WITH (refresh = false); @@ -249,10 +249,10 @@ ALTER SUBSCRIPTION regress_testsub SET (slot_name = 'newname'); ALTER SUBSCRIPTION regress_testsub SET (password_required = false); ALTER SUBSCRIPTION regress_testsub SET (run_as_owner = true); \dRs+ - List of subscriptions - Name | Owner | Enabled | Publication | Binary | Streaming | Two-phase commit | Disable on error | Origin | Password required | Run as owner? | Failover | Server | Retain dead tuples | Max retention duration | Retention active | Synchronous commit | Conninfo | Receiver timeout | Skip LSN | Description ------------------+---------------------------+---------+---------------------+--------+-----------+------------------+------------------+--------+-------------------+---------------+----------+--------+--------------------+------------------------+------------------+--------------------+------------------------------+------------------+------------+------------------- - regress_testsub | regress_subscription_user | f | {testpub2,testpub3} | f | parallel | d | f | any | f | t | f | | f | 0 | f | off | dbname=regress_doesnotexist2 | -1 | 0/00000000 | test subscription + List of subscriptions + Name | Owner | Enabled | Publication | Binary | Streaming | Two-phase commit | Disable on error | Origin | Password required | Run as owner? | Failover | Server | Retain dead tuples | Max retention duration | Retention active | Message | Synchronous commit | Conninfo | Receiver timeout | Skip LSN | Description +-----------------+---------------------------+---------+---------------------+--------+-----------+------------------+------------------+--------+-------------------+---------------+----------+--------+--------------------+------------------------+------------------+---------+--------------------+------------------------------+------------------+------------+------------------- + regress_testsub | regress_subscription_user | f | {testpub2,testpub3} | f | parallel | d | f | any | f | t | f | | f | 0 | f | f | off | dbname=regress_doesnotexist2 | -1 | 0/00000000 | test subscription (1 row) ALTER SUBSCRIPTION regress_testsub SET (password_required = true); @@ -268,10 +268,10 @@ ERROR: unrecognized subscription parameter: "create_slot" -- ok ALTER SUBSCRIPTION regress_testsub SKIP (lsn = '0/12345'); \dRs+ - List of subscriptions - Name | Owner | Enabled | Publication | Binary | Streaming | Two-phase commit | Disable on error | Origin | Password required | Run as owner? | Failover | Server | Retain dead tuples | Max retention duration | Retention active | Synchronous commit | Conninfo | Receiver timeout | Skip LSN | Description ------------------+---------------------------+---------+---------------------+--------+-----------+------------------+------------------+--------+-------------------+---------------+----------+--------+--------------------+------------------------+------------------+--------------------+------------------------------+------------------+------------+------------------- - regress_testsub | regress_subscription_user | f | {testpub2,testpub3} | f | parallel | d | f | any | t | f | f | | f | 0 | f | off | dbname=regress_doesnotexist2 | -1 | 0/00012345 | test subscription + List of subscriptions + Name | Owner | Enabled | Publication | Binary | Streaming | Two-phase commit | Disable on error | Origin | Password required | Run as owner? | Failover | Server | Retain dead tuples | Max retention duration | Retention active | Message | Synchronous commit | Conninfo | Receiver timeout | Skip LSN | Description +-----------------+---------------------------+---------+---------------------+--------+-----------+------------------+------------------+--------+-------------------+---------------+----------+--------+--------------------+------------------------+------------------+---------+--------------------+------------------------------+------------------+------------+------------------- + regress_testsub | regress_subscription_user | f | {testpub2,testpub3} | f | parallel | d | f | any | t | f | f | | f | 0 | f | f | off | dbname=regress_doesnotexist2 | -1 | 0/00012345 | test subscription (1 row) -- ok - with lsn = NONE @@ -280,10 +280,10 @@ ALTER SUBSCRIPTION regress_testsub SKIP (lsn = NONE); ALTER SUBSCRIPTION regress_testsub SKIP (lsn = '0/0'); ERROR: invalid WAL location (LSN): 0/0 \dRs+ - List of subscriptions - Name | Owner | Enabled | Publication | Binary | Streaming | Two-phase commit | Disable on error | Origin | Password required | Run as owner? | Failover | Server | Retain dead tuples | Max retention duration | Retention active | Synchronous commit | Conninfo | Receiver timeout | Skip LSN | Description ------------------+---------------------------+---------+---------------------+--------+-----------+------------------+------------------+--------+-------------------+---------------+----------+--------+--------------------+------------------------+------------------+--------------------+------------------------------+------------------+------------+------------------- - regress_testsub | regress_subscription_user | f | {testpub2,testpub3} | f | parallel | d | f | any | t | f | f | | f | 0 | f | off | dbname=regress_doesnotexist2 | -1 | 0/00000000 | test subscription + List of subscriptions + Name | Owner | Enabled | Publication | Binary | Streaming | Two-phase commit | Disable on error | Origin | Password required | Run as owner? | Failover | Server | Retain dead tuples | Max retention duration | Retention active | Message | Synchronous commit | Conninfo | Receiver timeout | Skip LSN | Description +-----------------+---------------------------+---------+---------------------+--------+-----------+------------------+------------------+--------+-------------------+---------------+----------+--------+--------------------+------------------------+------------------+---------+--------------------+------------------------------+------------------+------------+------------------- + regress_testsub | regress_subscription_user | f | {testpub2,testpub3} | f | parallel | d | f | any | t | f | f | | f | 0 | f | f | off | dbname=regress_doesnotexist2 | -1 | 0/00000000 | test subscription (1 row) BEGIN; @@ -319,10 +319,10 @@ ALTER SUBSCRIPTION regress_testsub_foo SET (wal_receiver_timeout = '80s'); ALTER SUBSCRIPTION regress_testsub_foo SET (wal_receiver_timeout = 'foobar'); ERROR: invalid value for parameter "wal_receiver_timeout": "foobar" \dRs+ - List of subscriptions - Name | Owner | Enabled | Publication | Binary | Streaming | Two-phase commit | Disable on error | Origin | Password required | Run as owner? | Failover | Server | Retain dead tuples | Max retention duration | Retention active | Synchronous commit | Conninfo | Receiver timeout | Skip LSN | Description ----------------------+---------------------------+---------+---------------------+--------+-----------+------------------+------------------+--------+-------------------+---------------+----------+--------+--------------------+------------------------+------------------+--------------------+------------------------------+------------------+------------+------------------- - regress_testsub_foo | regress_subscription_user | f | {testpub2,testpub3} | f | parallel | d | f | any | t | f | f | | f | 0 | f | local | dbname=regress_doesnotexist2 | 80s | 0/00000000 | test subscription + List of subscriptions + Name | Owner | Enabled | Publication | Binary | Streaming | Two-phase commit | Disable on error | Origin | Password required | Run as owner? | Failover | Server | Retain dead tuples | Max retention duration | Retention active | Message | Synchronous commit | Conninfo | Receiver timeout | Skip LSN | Description +---------------------+---------------------------+---------+---------------------+--------+-----------+------------------+------------------+--------+-------------------+---------------+----------+--------+--------------------+------------------------+------------------+---------+--------------------+------------------------------+------------------+------------+------------------- + regress_testsub_foo | regress_subscription_user | f | {testpub2,testpub3} | f | parallel | d | f | any | t | f | f | | f | 0 | f | f | local | dbname=regress_doesnotexist2 | 80s | 0/00000000 | test subscription (1 row) -- rename back to keep the rest simple @@ -351,19 +351,19 @@ CREATE SUBSCRIPTION regress_testsub CONNECTION 'dbname=regress_doesnotexist' PUB WARNING: subscription was created, but is not connected HINT: To initiate replication, you must manually create the replication slot, enable the subscription, and alter the subscription to refresh publications. \dRs+ - List of subscriptions - Name | Owner | Enabled | Publication | Binary | Streaming | Two-phase commit | Disable on error | Origin | Password required | Run as owner? | Failover | Server | Retain dead tuples | Max retention duration | Retention active | Synchronous commit | Conninfo | Receiver timeout | Skip LSN | Description ------------------+---------------------------+---------+-------------+--------+-----------+------------------+------------------+--------+-------------------+---------------+----------+--------+--------------------+------------------------+------------------+--------------------+-----------------------------+------------------+------------+------------- - regress_testsub | regress_subscription_user | f | {testpub} | t | parallel | d | f | any | t | f | f | | f | 0 | f | off | dbname=regress_doesnotexist | -1 | 0/00000000 | + List of subscriptions + Name | Owner | Enabled | Publication | Binary | Streaming | Two-phase commit | Disable on error | Origin | Password required | Run as owner? | Failover | Server | Retain dead tuples | Max retention duration | Retention active | Message | Synchronous commit | Conninfo | Receiver timeout | Skip LSN | Description +-----------------+---------------------------+---------+-------------+--------+-----------+------------------+------------------+--------+-------------------+---------------+----------+--------+--------------------+------------------------+------------------+---------+--------------------+-----------------------------+------------------+------------+------------- + regress_testsub | regress_subscription_user | f | {testpub} | t | parallel | d | f | any | t | f | f | | f | 0 | f | f | off | dbname=regress_doesnotexist | -1 | 0/00000000 | (1 row) ALTER SUBSCRIPTION regress_testsub SET (binary = false); ALTER SUBSCRIPTION regress_testsub SET (slot_name = NONE); \dRs+ - List of subscriptions - Name | Owner | Enabled | Publication | Binary | Streaming | Two-phase commit | Disable on error | Origin | Password required | Run as owner? | Failover | Server | Retain dead tuples | Max retention duration | Retention active | Synchronous commit | Conninfo | Receiver timeout | Skip LSN | Description ------------------+---------------------------+---------+-------------+--------+-----------+------------------+------------------+--------+-------------------+---------------+----------+--------+--------------------+------------------------+------------------+--------------------+-----------------------------+------------------+------------+------------- - regress_testsub | regress_subscription_user | f | {testpub} | f | parallel | d | f | any | t | f | f | | f | 0 | f | off | dbname=regress_doesnotexist | -1 | 0/00000000 | + List of subscriptions + Name | Owner | Enabled | Publication | Binary | Streaming | Two-phase commit | Disable on error | Origin | Password required | Run as owner? | Failover | Server | Retain dead tuples | Max retention duration | Retention active | Message | Synchronous commit | Conninfo | Receiver timeout | Skip LSN | Description +-----------------+---------------------------+---------+-------------+--------+-----------+------------------+------------------+--------+-------------------+---------------+----------+--------+--------------------+------------------------+------------------+---------+--------------------+-----------------------------+------------------+------------+------------- + regress_testsub | regress_subscription_user | f | {testpub} | f | parallel | d | f | any | t | f | f | | f | 0 | f | f | off | dbname=regress_doesnotexist | -1 | 0/00000000 | (1 row) DROP SUBSCRIPTION regress_testsub; @@ -375,27 +375,27 @@ CREATE SUBSCRIPTION regress_testsub CONNECTION 'dbname=regress_doesnotexist' PUB WARNING: subscription was created, but is not connected HINT: To initiate replication, you must manually create the replication slot, enable the subscription, and alter the subscription to refresh publications. \dRs+ - List of subscriptions - Name | Owner | Enabled | Publication | Binary | Streaming | Two-phase commit | Disable on error | Origin | Password required | Run as owner? | Failover | Server | Retain dead tuples | Max retention duration | Retention active | Synchronous commit | Conninfo | Receiver timeout | Skip LSN | Description ------------------+---------------------------+---------+-------------+--------+-----------+------------------+------------------+--------+-------------------+---------------+----------+--------+--------------------+------------------------+------------------+--------------------+-----------------------------+------------------+------------+------------- - regress_testsub | regress_subscription_user | f | {testpub} | f | on | d | f | any | t | f | f | | f | 0 | f | off | dbname=regress_doesnotexist | -1 | 0/00000000 | + List of subscriptions + Name | Owner | Enabled | Publication | Binary | Streaming | Two-phase commit | Disable on error | Origin | Password required | Run as owner? | Failover | Server | Retain dead tuples | Max retention duration | Retention active | Message | Synchronous commit | Conninfo | Receiver timeout | Skip LSN | Description +-----------------+---------------------------+---------+-------------+--------+-----------+------------------+------------------+--------+-------------------+---------------+----------+--------+--------------------+------------------------+------------------+---------+--------------------+-----------------------------+------------------+------------+------------- + regress_testsub | regress_subscription_user | f | {testpub} | f | on | d | f | any | t | f | f | | f | 0 | f | f | off | dbname=regress_doesnotexist | -1 | 0/00000000 | (1 row) ALTER SUBSCRIPTION regress_testsub SET (streaming = parallel); \dRs+ - List of subscriptions - Name | Owner | Enabled | Publication | Binary | Streaming | Two-phase commit | Disable on error | Origin | Password required | Run as owner? | Failover | Server | Retain dead tuples | Max retention duration | Retention active | Synchronous commit | Conninfo | Receiver timeout | Skip LSN | Description ------------------+---------------------------+---------+-------------+--------+-----------+------------------+------------------+--------+-------------------+---------------+----------+--------+--------------------+------------------------+------------------+--------------------+-----------------------------+------------------+------------+------------- - regress_testsub | regress_subscription_user | f | {testpub} | f | parallel | d | f | any | t | f | f | | f | 0 | f | off | dbname=regress_doesnotexist | -1 | 0/00000000 | + List of subscriptions + Name | Owner | Enabled | Publication | Binary | Streaming | Two-phase commit | Disable on error | Origin | Password required | Run as owner? | Failover | Server | Retain dead tuples | Max retention duration | Retention active | Message | Synchronous commit | Conninfo | Receiver timeout | Skip LSN | Description +-----------------+---------------------------+---------+-------------+--------+-----------+------------------+------------------+--------+-------------------+---------------+----------+--------+--------------------+------------------------+------------------+---------+--------------------+-----------------------------+------------------+------------+------------- + regress_testsub | regress_subscription_user | f | {testpub} | f | parallel | d | f | any | t | f | f | | f | 0 | f | f | off | dbname=regress_doesnotexist | -1 | 0/00000000 | (1 row) ALTER SUBSCRIPTION regress_testsub SET (streaming = false); ALTER SUBSCRIPTION regress_testsub SET (slot_name = NONE); \dRs+ - List of subscriptions - Name | Owner | Enabled | Publication | Binary | Streaming | Two-phase commit | Disable on error | Origin | Password required | Run as owner? | Failover | Server | Retain dead tuples | Max retention duration | Retention active | Synchronous commit | Conninfo | Receiver timeout | Skip LSN | Description ------------------+---------------------------+---------+-------------+--------+-----------+------------------+------------------+--------+-------------------+---------------+----------+--------+--------------------+------------------------+------------------+--------------------+-----------------------------+------------------+------------+------------- - regress_testsub | regress_subscription_user | f | {testpub} | f | off | d | f | any | t | f | f | | f | 0 | f | off | dbname=regress_doesnotexist | -1 | 0/00000000 | + List of subscriptions + Name | Owner | Enabled | Publication | Binary | Streaming | Two-phase commit | Disable on error | Origin | Password required | Run as owner? | Failover | Server | Retain dead tuples | Max retention duration | Retention active | Message | Synchronous commit | Conninfo | Receiver timeout | Skip LSN | Description +-----------------+---------------------------+---------+-------------+--------+-----------+------------------+------------------+--------+-------------------+---------------+----------+--------+--------------------+------------------------+------------------+---------+--------------------+-----------------------------+------------------+------------+------------- + regress_testsub | regress_subscription_user | f | {testpub} | f | off | d | f | any | t | f | f | | f | 0 | f | f | off | dbname=regress_doesnotexist | -1 | 0/00000000 | (1 row) -- fail - publication already exists @@ -410,10 +410,10 @@ ALTER SUBSCRIPTION regress_testsub ADD PUBLICATION testpub1, testpub2 WITH (refr ALTER SUBSCRIPTION regress_testsub ADD PUBLICATION testpub1, testpub2 WITH (refresh = false); ERROR: publication "testpub1" is already in subscription "regress_testsub" \dRs+ - List of subscriptions - Name | Owner | Enabled | Publication | Binary | Streaming | Two-phase commit | Disable on error | Origin | Password required | Run as owner? | Failover | Server | Retain dead tuples | Max retention duration | Retention active | Synchronous commit | Conninfo | Receiver timeout | Skip LSN | Description ------------------+---------------------------+---------+-----------------------------+--------+-----------+------------------+------------------+--------+-------------------+---------------+----------+--------+--------------------+------------------------+------------------+--------------------+-----------------------------+------------------+------------+------------- - regress_testsub | regress_subscription_user | f | {testpub,testpub1,testpub2} | f | off | d | f | any | t | f | f | | f | 0 | f | off | dbname=regress_doesnotexist | -1 | 0/00000000 | + List of subscriptions + Name | Owner | Enabled | Publication | Binary | Streaming | Two-phase commit | Disable on error | Origin | Password required | Run as owner? | Failover | Server | Retain dead tuples | Max retention duration | Retention active | Message | Synchronous commit | Conninfo | Receiver timeout | Skip LSN | Description +-----------------+---------------------------+---------+-----------------------------+--------+-----------+------------------+------------------+--------+-------------------+---------------+----------+--------+--------------------+------------------------+------------------+---------+--------------------+-----------------------------+------------------+------------+------------- + regress_testsub | regress_subscription_user | f | {testpub,testpub1,testpub2} | f | off | d | f | any | t | f | f | | f | 0 | f | f | off | dbname=regress_doesnotexist | -1 | 0/00000000 | (1 row) -- fail - publication used more than once @@ -428,10 +428,10 @@ ERROR: publication "testpub3" is not in subscription "regress_testsub" -- ok - delete publications ALTER SUBSCRIPTION regress_testsub DROP PUBLICATION testpub1, testpub2 WITH (refresh = false); \dRs+ - List of subscriptions - Name | Owner | Enabled | Publication | Binary | Streaming | Two-phase commit | Disable on error | Origin | Password required | Run as owner? | Failover | Server | Retain dead tuples | Max retention duration | Retention active | Synchronous commit | Conninfo | Receiver timeout | Skip LSN | Description ------------------+---------------------------+---------+-------------+--------+-----------+------------------+------------------+--------+-------------------+---------------+----------+--------+--------------------+------------------------+------------------+--------------------+-----------------------------+------------------+------------+------------- - regress_testsub | regress_subscription_user | f | {testpub} | f | off | d | f | any | t | f | f | | f | 0 | f | off | dbname=regress_doesnotexist | -1 | 0/00000000 | + List of subscriptions + Name | Owner | Enabled | Publication | Binary | Streaming | Two-phase commit | Disable on error | Origin | Password required | Run as owner? | Failover | Server | Retain dead tuples | Max retention duration | Retention active | Message | Synchronous commit | Conninfo | Receiver timeout | Skip LSN | Description +-----------------+---------------------------+---------+-------------+--------+-----------+------------------+------------------+--------+-------------------+---------------+----------+--------+--------------------+------------------------+------------------+---------+--------------------+-----------------------------+------------------+------------+------------- + regress_testsub | regress_subscription_user | f | {testpub} | f | off | d | f | any | t | f | f | | f | 0 | f | f | off | dbname=regress_doesnotexist | -1 | 0/00000000 | (1 row) DROP SUBSCRIPTION regress_testsub; @@ -467,19 +467,19 @@ CREATE SUBSCRIPTION regress_testsub CONNECTION 'dbname=regress_doesnotexist' PUB WARNING: subscription was created, but is not connected HINT: To initiate replication, you must manually create the replication slot, enable the subscription, and alter the subscription to refresh publications. \dRs+ - List of subscriptions - Name | Owner | Enabled | Publication | Binary | Streaming | Two-phase commit | Disable on error | Origin | Password required | Run as owner? | Failover | Server | Retain dead tuples | Max retention duration | Retention active | Synchronous commit | Conninfo | Receiver timeout | Skip LSN | Description ------------------+---------------------------+---------+-------------+--------+-----------+------------------+------------------+--------+-------------------+---------------+----------+--------+--------------------+------------------------+------------------+--------------------+-----------------------------+------------------+------------+------------- - regress_testsub | regress_subscription_user | f | {testpub} | f | parallel | p | f | any | t | f | f | | f | 0 | f | off | dbname=regress_doesnotexist | -1 | 0/00000000 | + List of subscriptions + Name | Owner | Enabled | Publication | Binary | Streaming | Two-phase commit | Disable on error | Origin | Password required | Run as owner? | Failover | Server | Retain dead tuples | Max retention duration | Retention active | Message | Synchronous commit | Conninfo | Receiver timeout | Skip LSN | Description +-----------------+---------------------------+---------+-------------+--------+-----------+------------------+------------------+--------+-------------------+---------------+----------+--------+--------------------+------------------------+------------------+---------+--------------------+-----------------------------+------------------+------------+------------- + regress_testsub | regress_subscription_user | f | {testpub} | f | parallel | p | f | any | t | f | f | | f | 0 | f | f | off | dbname=regress_doesnotexist | -1 | 0/00000000 | (1 row) -- we can alter streaming when two_phase enabled ALTER SUBSCRIPTION regress_testsub SET (streaming = true); \dRs+ - List of subscriptions - Name | Owner | Enabled | Publication | Binary | Streaming | Two-phase commit | Disable on error | Origin | Password required | Run as owner? | Failover | Server | Retain dead tuples | Max retention duration | Retention active | Synchronous commit | Conninfo | Receiver timeout | Skip LSN | Description ------------------+---------------------------+---------+-------------+--------+-----------+------------------+------------------+--------+-------------------+---------------+----------+--------+--------------------+------------------------+------------------+--------------------+-----------------------------+------------------+------------+------------- - regress_testsub | regress_subscription_user | f | {testpub} | f | on | p | f | any | t | f | f | | f | 0 | f | off | dbname=regress_doesnotexist | -1 | 0/00000000 | + List of subscriptions + Name | Owner | Enabled | Publication | Binary | Streaming | Two-phase commit | Disable on error | Origin | Password required | Run as owner? | Failover | Server | Retain dead tuples | Max retention duration | Retention active | Message | Synchronous commit | Conninfo | Receiver timeout | Skip LSN | Description +-----------------+---------------------------+---------+-------------+--------+-----------+------------------+------------------+--------+-------------------+---------------+----------+--------+--------------------+------------------------+------------------+---------+--------------------+-----------------------------+------------------+------------+------------- + regress_testsub | regress_subscription_user | f | {testpub} | f | on | p | f | any | t | f | f | | f | 0 | f | f | off | dbname=regress_doesnotexist | -1 | 0/00000000 | (1 row) ALTER SUBSCRIPTION regress_testsub SET (slot_name = NONE); @@ -489,10 +489,10 @@ CREATE SUBSCRIPTION regress_testsub CONNECTION 'dbname=regress_doesnotexist' PUB WARNING: subscription was created, but is not connected HINT: To initiate replication, you must manually create the replication slot, enable the subscription, and alter the subscription to refresh publications. \dRs+ - List of subscriptions - Name | Owner | Enabled | Publication | Binary | Streaming | Two-phase commit | Disable on error | Origin | Password required | Run as owner? | Failover | Server | Retain dead tuples | Max retention duration | Retention active | Synchronous commit | Conninfo | Receiver timeout | Skip LSN | Description ------------------+---------------------------+---------+-------------+--------+-----------+------------------+------------------+--------+-------------------+---------------+----------+--------+--------------------+------------------------+------------------+--------------------+-----------------------------+------------------+------------+------------- - regress_testsub | regress_subscription_user | f | {testpub} | f | on | p | f | any | t | f | f | | f | 0 | f | off | dbname=regress_doesnotexist | -1 | 0/00000000 | + List of subscriptions + Name | Owner | Enabled | Publication | Binary | Streaming | Two-phase commit | Disable on error | Origin | Password required | Run as owner? | Failover | Server | Retain dead tuples | Max retention duration | Retention active | Message | Synchronous commit | Conninfo | Receiver timeout | Skip LSN | Description +-----------------+---------------------------+---------+-------------+--------+-----------+------------------+------------------+--------+-------------------+---------------+----------+--------+--------------------+------------------------+------------------+---------+--------------------+-----------------------------+------------------+------------+------------- + regress_testsub | regress_subscription_user | f | {testpub} | f | on | p | f | any | t | f | f | | f | 0 | f | f | off | dbname=regress_doesnotexist | -1 | 0/00000000 | (1 row) ALTER SUBSCRIPTION regress_testsub SET (slot_name = NONE); @@ -505,18 +505,18 @@ CREATE SUBSCRIPTION regress_testsub CONNECTION 'dbname=regress_doesnotexist' PUB WARNING: subscription was created, but is not connected HINT: To initiate replication, you must manually create the replication slot, enable the subscription, and alter the subscription to refresh publications. \dRs+ - List of subscriptions - Name | Owner | Enabled | Publication | Binary | Streaming | Two-phase commit | Disable on error | Origin | Password required | Run as owner? | Failover | Server | Retain dead tuples | Max retention duration | Retention active | Synchronous commit | Conninfo | Receiver timeout | Skip LSN | Description ------------------+---------------------------+---------+-------------+--------+-----------+------------------+------------------+--------+-------------------+---------------+----------+--------+--------------------+------------------------+------------------+--------------------+-----------------------------+------------------+------------+------------- - regress_testsub | regress_subscription_user | f | {testpub} | f | parallel | d | f | any | t | f | f | | f | 0 | f | off | dbname=regress_doesnotexist | -1 | 0/00000000 | + List of subscriptions + Name | Owner | Enabled | Publication | Binary | Streaming | Two-phase commit | Disable on error | Origin | Password required | Run as owner? | Failover | Server | Retain dead tuples | Max retention duration | Retention active | Message | Synchronous commit | Conninfo | Receiver timeout | Skip LSN | Description +-----------------+---------------------------+---------+-------------+--------+-----------+------------------+------------------+--------+-------------------+---------------+----------+--------+--------------------+------------------------+------------------+---------+--------------------+-----------------------------+------------------+------------+------------- + regress_testsub | regress_subscription_user | f | {testpub} | f | parallel | d | f | any | t | f | f | | f | 0 | f | f | off | dbname=regress_doesnotexist | -1 | 0/00000000 | (1 row) ALTER SUBSCRIPTION regress_testsub SET (disable_on_error = true); \dRs+ - List of subscriptions - Name | Owner | Enabled | Publication | Binary | Streaming | Two-phase commit | Disable on error | Origin | Password required | Run as owner? | Failover | Server | Retain dead tuples | Max retention duration | Retention active | Synchronous commit | Conninfo | Receiver timeout | Skip LSN | Description ------------------+---------------------------+---------+-------------+--------+-----------+------------------+------------------+--------+-------------------+---------------+----------+--------+--------------------+------------------------+------------------+--------------------+-----------------------------+------------------+------------+------------- - regress_testsub | regress_subscription_user | f | {testpub} | f | parallel | d | t | any | t | f | f | | f | 0 | f | off | dbname=regress_doesnotexist | -1 | 0/00000000 | + List of subscriptions + Name | Owner | Enabled | Publication | Binary | Streaming | Two-phase commit | Disable on error | Origin | Password required | Run as owner? | Failover | Server | Retain dead tuples | Max retention duration | Retention active | Message | Synchronous commit | Conninfo | Receiver timeout | Skip LSN | Description +-----------------+---------------------------+---------+-------------+--------+-----------+------------------+------------------+--------+-------------------+---------------+----------+--------+--------------------+------------------------+------------------+---------+--------------------+-----------------------------+------------------+------------+------------- + regress_testsub | regress_subscription_user | f | {testpub} | f | parallel | d | t | any | t | f | f | | f | 0 | f | f | off | dbname=regress_doesnotexist | -1 | 0/00000000 | (1 row) ALTER SUBSCRIPTION regress_testsub SET (slot_name = NONE); @@ -529,10 +529,10 @@ CREATE SUBSCRIPTION regress_testsub CONNECTION 'dbname=regress_doesnotexist' PUB WARNING: subscription was created, but is not connected HINT: To initiate replication, you must manually create the replication slot, enable the subscription, and alter the subscription to refresh publications. \dRs+ - List of subscriptions - Name | Owner | Enabled | Publication | Binary | Streaming | Two-phase commit | Disable on error | Origin | Password required | Run as owner? | Failover | Server | Retain dead tuples | Max retention duration | Retention active | Synchronous commit | Conninfo | Receiver timeout | Skip LSN | Description ------------------+---------------------------+---------+-------------+--------+-----------+------------------+------------------+--------+-------------------+---------------+----------+--------+--------------------+------------------------+------------------+--------------------+-----------------------------+------------------+------------+------------- - regress_testsub | regress_subscription_user | f | {testpub} | f | parallel | d | f | any | t | f | f | | f | 0 | f | off | dbname=regress_doesnotexist | -1 | 0/00000000 | + List of subscriptions + Name | Owner | Enabled | Publication | Binary | Streaming | Two-phase commit | Disable on error | Origin | Password required | Run as owner? | Failover | Server | Retain dead tuples | Max retention duration | Retention active | Message | Synchronous commit | Conninfo | Receiver timeout | Skip LSN | Description +-----------------+---------------------------+---------+-------------+--------+-----------+------------------+------------------+--------+-------------------+---------------+----------+--------+--------------------+------------------------+------------------+---------+--------------------+-----------------------------+------------------+------------+------------- + regress_testsub | regress_subscription_user | f | {testpub} | f | parallel | d | f | any | t | f | f | | f | 0 | f | f | off | dbname=regress_doesnotexist | -1 | 0/00000000 | (1 row) ALTER SUBSCRIPTION regress_testsub SET (slot_name = NONE); @@ -549,10 +549,10 @@ NOTICE: max_retention_duration is ineffective when retain_dead_tuples is disabl WARNING: subscription was created, but is not connected HINT: To initiate replication, you must manually create the replication slot, enable the subscription, and alter the subscription to refresh publications. \dRs+ - List of subscriptions - Name | Owner | Enabled | Publication | Binary | Streaming | Two-phase commit | Disable on error | Origin | Password required | Run as owner? | Failover | Server | Retain dead tuples | Max retention duration | Retention active | Synchronous commit | Conninfo | Receiver timeout | Skip LSN | Description ------------------+---------------------------+---------+-------------+--------+-----------+------------------+------------------+--------+-------------------+---------------+----------+--------+--------------------+------------------------+------------------+--------------------+-----------------------------+------------------+------------+------------- - regress_testsub | regress_subscription_user | f | {testpub} | f | parallel | d | f | any | t | f | f | | f | 1000 | f | off | dbname=regress_doesnotexist | -1 | 0/00000000 | + List of subscriptions + Name | Owner | Enabled | Publication | Binary | Streaming | Two-phase commit | Disable on error | Origin | Password required | Run as owner? | Failover | Server | Retain dead tuples | Max retention duration | Retention active | Message | Synchronous commit | Conninfo | Receiver timeout | Skip LSN | Description +-----------------+---------------------------+---------+-------------+--------+-----------+------------------+------------------+--------+-------------------+---------------+----------+--------+--------------------+------------------------+------------------+---------+--------------------+-----------------------------+------------------+------------+------------- + regress_testsub | regress_subscription_user | f | {testpub} | f | parallel | d | f | any | t | f | f | | f | 1000 | f | f | off | dbname=regress_doesnotexist | -1 | 0/00000000 | (1 row) -- fail - max_retention_duration must be non-negative @@ -561,12 +561,38 @@ ERROR: max_retention_duration cannot be negative -- ok ALTER SUBSCRIPTION regress_testsub SET (max_retention_duration = 0); \dRs+ - List of subscriptions - Name | Owner | Enabled | Publication | Binary | Streaming | Two-phase commit | Disable on error | Origin | Password required | Run as owner? | Failover | Server | Retain dead tuples | Max retention duration | Retention active | Synchronous commit | Conninfo | Receiver timeout | Skip LSN | Description ------------------+---------------------------+---------+-------------+--------+-----------+------------------+------------------+--------+-------------------+---------------+----------+--------+--------------------+------------------------+------------------+--------------------+-----------------------------+------------------+------------+------------- - regress_testsub | regress_subscription_user | f | {testpub} | f | parallel | d | f | any | t | f | f | | f | 0 | f | off | dbname=regress_doesnotexist | -1 | 0/00000000 | + List of subscriptions + Name | Owner | Enabled | Publication | Binary | Streaming | Two-phase commit | Disable on error | Origin | Password required | Run as owner? | Failover | Server | Retain dead tuples | Max retention duration | Retention active | Message | Synchronous commit | Conninfo | Receiver timeout | Skip LSN | Description +-----------------+---------------------------+---------+-------------+--------+-----------+------------------+------------------+--------+-------------------+---------------+----------+--------+--------------------+------------------------+------------------+---------+--------------------+-----------------------------+------------------+------------+------------- + regress_testsub | regress_subscription_user | f | {testpub} | f | parallel | d | f | any | t | f | f | | f | 0 | f | f | off | dbname=regress_doesnotexist | -1 | 0/00000000 | +(1 row) + +ALTER SUBSCRIPTION regress_testsub SET (slot_name = NONE); +DROP SUBSCRIPTION regress_testsub; +-- fail - message must be boolean +CREATE SUBSCRIPTION regress_testsub CONNECTION 'dbname=regress_doesnotexist' PUBLICATION testpub WITH (connect = false, message = foo); +ERROR: message requires a Boolean value +-- ok +CREATE SUBSCRIPTION regress_testsub CONNECTION 'dbname=regress_doesnotexist' PUBLICATION testpub WITH (connect = false, message = true); +WARNING: subscription was created, but is not connected +HINT: To initiate replication, you must manually create the replication slot, enable the subscription, and alter the subscription to refresh publications. +\dRs+ + List of subscriptions + Name | Owner | Enabled | Publication | Binary | Streaming | Two-phase commit | Disable on error | Origin | Password required | Run as owner? | Failover | Server | Retain dead tuples | Max retention duration | Retention active | Message | Synchronous commit | Conninfo | Receiver timeout | Skip LSN | Description +-----------------+---------------------------+---------+-------------+--------+-----------+------------------+------------------+--------+-------------------+---------------+----------+--------+--------------------+------------------------+------------------+---------+--------------------+-----------------------------+------------------+------------+------------- + regress_testsub | regress_subscription_user | f | {testpub} | f | parallel | d | f | any | t | f | f | | f | 0 | f | t | off | dbname=regress_doesnotexist | -1 | 0/00000000 | +(1 row) + +-- ok +ALTER SUBSCRIPTION regress_testsub SET (message = false); +\dRs+ + List of subscriptions + Name | Owner | Enabled | Publication | Binary | Streaming | Two-phase commit | Disable on error | Origin | Password required | Run as owner? | Failover | Server | Retain dead tuples | Max retention duration | Retention active | Message | Synchronous commit | Conninfo | Receiver timeout | Skip LSN | Description +-----------------+---------------------------+---------+-------------+--------+-----------+------------------+------------------+--------+-------------------+---------------+----------+--------+--------------------+------------------------+------------------+---------+--------------------+-----------------------------+------------------+------------+------------- + regress_testsub | regress_subscription_user | f | {testpub} | f | parallel | d | f | any | t | f | f | | f | 0 | f | f | off | dbname=regress_doesnotexist | -1 | 0/00000000 | (1 row) +-- cleanup ALTER SUBSCRIPTION regress_testsub SET (slot_name = NONE); DROP SUBSCRIPTION regress_testsub; -- let's do some tests with pg_create_subscription rather than superuser diff --git a/src/test/regress/sql/subscription.sql b/src/test/regress/sql/subscription.sql index 05533d66675..7eef3721a9a 100644 --- a/src/test/regress/sql/subscription.sql +++ b/src/test/regress/sql/subscription.sql @@ -423,6 +423,23 @@ ALTER SUBSCRIPTION regress_testsub SET (max_retention_duration = 0); ALTER SUBSCRIPTION regress_testsub SET (slot_name = NONE); DROP SUBSCRIPTION regress_testsub; +-- fail - message must be boolean +CREATE SUBSCRIPTION regress_testsub CONNECTION 'dbname=regress_doesnotexist' PUBLICATION testpub WITH (connect = false, message = foo); + +-- ok +CREATE SUBSCRIPTION regress_testsub CONNECTION 'dbname=regress_doesnotexist' PUBLICATION testpub WITH (connect = false, message = true); + +\dRs+ + +-- ok +ALTER SUBSCRIPTION regress_testsub SET (message = false); + +\dRs+ + +-- cleanup +ALTER SUBSCRIPTION regress_testsub SET (slot_name = NONE); +DROP SUBSCRIPTION regress_testsub; + -- let's do some tests with pg_create_subscription rather than superuser SET SESSION AUTHORIZATION regress_subscription_user3; diff --git a/src/tools/pgindent/typedefs.list b/src/tools/pgindent/typedefs.list index 1969d467c1d..09c9b87d483 100644 --- a/src/tools/pgindent/typedefs.list +++ b/src/tools/pgindent/typedefs.list @@ -1701,6 +1701,7 @@ LogicalRepBeginData LogicalRepCommitData LogicalRepCommitPreparedTxnData LogicalRepCtxStruct +LogicalRepMessageData LogicalRepMsgType LogicalRepPartMapEntry LogicalRepPreparedTxnData -- 2.54.0
^ permalink raw reply [nested|flat] 44+ messages in thread

* Re: Add a hook for handling logical decoding messages on subscribers.
@ 2026-06-22 22:39  Bharath Rupireddy <bharath.rupireddyforpostgres@gmail.com>
  parent: Masahiko Sawada <sawada.mshk@gmail.com>
  0 siblings, 2 replies; 44+ messages in thread

From: Bharath Rupireddy @ 2026-06-22 22:39 UTC (permalink / raw)
  To: Masahiko Sawada <sawada.mshk@gmail.com>; +Cc: pgsql-hackers

Hi,

On Fri, Jun 19, 2026 at 3:34 PM Masahiko Sawada <sawada.mshk@gmail.com> wrote:
>
> Hi all,
>
> Commit ac4645c015 allows pgoutput to send logical decoding messages,
> but it's limited to applications that use the pgoutput plugin -- the
> built-in logical replication doesn't use it. I'd like to propose
> introducing a hook to the logical replication message handling so that
> extensions can plug in their own handling routine. This feature can be
> used for extensions to implement DDL replication, function
> replication, or trigger user-specific routines on the subscriber side.

Thanks for working on this!

> I've attached the PoC patch; it adds a hook function, and adds a new
> 'message' subscription option that allows the user to request the
> publisher to send logical decoding messages. Therefore, users need to
> enable the 'message' option and set up the hook function at server
> startup in order to receive the messages and trigger the hook
> function.

I understand the intent of the proposal, but I'd like to get the
bigger picture first.

Do we have any external modules that actually implement DDL
replication (or any of the listed use-cases) with a similar hook? Or
any existing discussion? I could be missing something because I
haven't looked at all the DDL replication related threads.

Another thing I'm curious about - why a hook? Is the plan to implement
DDL replication as an external module rather than in core? If DDL
replication eventually gets into core, I'd expect it to be apply-side
logic executing the decoded DDL messages directly, not something going
through a hook.

Why not a hook at apply_dispatch to give external modules more freedom
with the pgoutput plugin?

> I I went with a hook function in the patch. While it lets you chain
> the multiple hook functions, providing the registration API might be
> better, or other types of registry can also be considered.

It's hard to tell how many external modules would make use of this
hook (rather, how many external modules implementing this hook one
would allow to be installed in a production database requiring
chaining), but my first thought is that a registration-based API along
the lines of RegisterXactCallback would be cleaner and work better.

> Feedback is very welcome.

A few comments on the patch:

1/
+ bool message; /* True if the subscription wants to receive
+ * logical messages */
 } Subscription;

Nit: I'd call these logical decoding messages or generic logical
messages - something to match the docs and pg_logical_emit_message.

2/
+void
+test_logical_message_handler(LogicalRepMessageData *msg)
+{
+ ereport(LOG,
+ (errmsg("received message: LSN %X/%08X, prefix: %s, message: %s",
+ LSN_FORMAT_ARGS(msg->lsn),
+ msg->prefix,
+ msg->message)));
+}

Why not have the test extension do a simple DDL/function/event-trigger
based replication? It doesn't need to be a full-blown implementation,
but to show the usefulness of this hook, it's better to have one
demonstrating the listed use-cases.

3/
+ if (subinfo->submessage)
+ appendPQExpBufferStr(query, ", message = true");

Why a subscription-level option? Why not leave the decision of whether
or not to act on the message to the external module implementers?

4/
+ /*
+ * Logical messages are handled only the (parallel) apply workers
+ */
+ if (am_tablesync_worker() || am_sequencesync_worker())
+ return;

Why these restrictions? Why not leave the decision to external module
implementers? Isn't this limiting - what if someone wants to use this
hook for the schema sync during the initial table sync phase?

5/ With this change, pg_logical_emit_message does affect the logical
replication apply if the subscriber has defined this hook. I think
it's worth mentioning in the docs for pg_logical_emit_message.

--
Bharath Rupireddy
Amazon Web Services: https://aws.amazon.com





^ permalink  raw  reply  [nested|flat] 44+ messages in thread

* Re: Add a hook for handling logical decoding messages on subscribers.
@ 2026-06-23 08:52  Amit Kapila <amit.kapila16@gmail.com>
  parent: Bharath Rupireddy <bharath.rupireddyforpostgres@gmail.com>
  1 sibling, 1 reply; 44+ messages in thread

From: Amit Kapila @ 2026-06-23 08:52 UTC (permalink / raw)
  To: Bharath Rupireddy <bharath.rupireddyforpostgres@gmail.com>; +Cc: Masahiko Sawada <sawada.mshk@gmail.com>; pgsql-hackers

On Tue, Jun 23, 2026 at 4:09 AM Bharath Rupireddy
<bharath.rupireddyforpostgres@gmail.com> wrote:
>
> On Fri, Jun 19, 2026 at 3:34 PM Masahiko Sawada <sawada.mshk@gmail.com> wrote:
> >
> > Hi all,
> >
> > Commit ac4645c015 allows pgoutput to send logical decoding messages,
> > but it's limited to applications that use the pgoutput plugin -- the
> > built-in logical replication doesn't use it. I'd like to propose
> > introducing a hook to the logical replication message handling so that
> > extensions can plug in their own handling routine. This feature can be
> > used for extensions to implement DDL replication, function
> > replication, or trigger user-specific routines on the subscriber side.
>
> Thanks for working on this!
>
> > I've attached the PoC patch; it adds a hook function, and adds a new
> > 'message' subscription option that allows the user to request the
> > publisher to send logical decoding messages. Therefore, users need to
> > enable the 'message' option and set up the hook function at server
> > startup in order to receive the messages and trigger the hook
> > function.
>
> I understand the intent of the proposal, but I'd like to get the
> bigger picture first.
>
> Do we have any external modules that actually implement DDL
> replication (or any of the listed use-cases) with a similar hook? Or
> any existing discussion? I could be missing something because I
> haven't looked at all the DDL replication related threads.
>
> Another thing I'm curious about - why a hook? Is the plan to implement
> DDL replication as an external module rather than in core? If DDL
> replication eventually gets into core, I'd expect it to be apply-side
> logic executing the decoded DDL messages directly, not something going
> through a hook.
>

I think it is important to have some example extension implementation
to see how the hook could be utilized. One more use of such a hook
could be to use for audit of DDLs replayed on subscribers. BTW, can we
also consider it as a solution implementing basic DDL replication for
tables? The key question is what if someday we have in-core DDL
replication. I think extensions can still be used to implement
filtering or transformation of DDL. We can implement capture of DDL
using JSON format [1] so that it is forward compatible with in-core
DDL replication. So considering that, the extension handlers will look
like:

 _PG_init(void)
  {
      /* Publisher side */
      prev_ProcessUtility = ProcessUtility_hook;
      ProcessUtility_hook = ddlrep_ProcessUtility;

.....
      /* Subscriber side */
      prev_message_handler  = logical_message_handler;
      logical_message_handler = ddlrep_message_handler;

static void
  ddlrep_ProcessUtility(PlannedStmt *pstmt,
                         const char *queryString,
                         bool readOnlyTree,
                         ProcessUtilityContext context,
                         ParamListInfo params,
                         QueryEnvironment *queryEnv,
                         DestReceiver *dest,
                         QueryCompletion *qc)
  {
      bool replicate = should_replicate(pstmt);

      /*
       * Execute the DDL first.
       */
      if (prev_ProcessUtility)
          prev_ProcessUtility(pstmt, queryString, readOnlyTree, context,
                              params, queryEnv, dest, qc);
      else
          standard_ProcessUtility(pstmt, queryString, readOnlyTree, context,
                                  params, queryEnv, dest, qc);

      if (replicate)
      {
          const char *tag =
GetCommandTagName(CreateCommandTag(pstmt->utilityStmt));
          char       *msg = build_ddl_message(tag, queryString);

          /*
           * transactional = true  → message held in ReorderBuffer, emitted
           * to subscribers at COMMIT in WAL order relative to any DML in
           * the same transaction.
           *
           * omit_lsn = false → include the LSN so the subscriber can log
           * exactly which WAL position a given DDL came from.
           */
          LogLogicalMessage("pg_ddl", msg, strlen(msg),
                            true  /* transactional */,
                            false /* omit_lsn */);

...

 static void
  ddlrep_message_handler(const char *prefix,
                          Size sz,
                          const char *message,
                          bool transactional,
                          XLogRecPtr lsn)
  {
      char           *payload;
      char           *ddl;
      char           *search_path;
      StringInfoData  cmd;
      int             spi_rc;

      /*
       * Always pass through to the previous handler first.  This ensures
       * correct behaviour when chained with other extensions.
       */
      if (prev_message_handler)
          prev_message_handler(prefix, sz, message, transactional, lsn);

      if (strcmp(prefix, "pg_ddl") != 0)
          return;

    /* Write code to execute/perform DDL. */

When we have a built-in handler then the apply worker carefully
registers the same and gives an ERROR if the extension one is already
registered.

void
  ApplyWorkerMain(Datum main_arg)
  {
      /* ... existing initialisation ... */

      /*
       * If the subscription requests built-in DDL replication and an
       * extension has also registered a logical message hook, both would
       * process the same "pg_ddl" messages and execute DDL twice.
       * Refuse to start rather than silently corrupt.
       */
      if (MySubscription->ddloption != DDL_OPTION_NONE &&
          logical_message_hook != NULL)
          ereport(ERROR,
                  (errcode(ERRCODE_OBJECT_NOT_IN_PREREQUISITE_STATE),
                   ....

      /*
       * No conflict — register the built-in DDL handler when the
       * subscription requests DDL replication and no extension owns
       * the hook.
       */
      if (MySubscription->ddloption != DDL_OPTION_NONE)
          logical_message_hook = builtin_ddl_message_handler;

[1]
The JSON format for WAL could be of form to keep it extendable:

{
    "version": 1,
    "command_tag": "CREATE TABLE",
    "object_type": "table",
    "schema": "public",
    "identity": "public.foo",
    "ddl_text": "CREATE TABLE public.foo (id int PRIMARY KEY)",
    "search_path": "public"
 }


>
> Why not a hook at apply_dispatch to give external modules more freedom
> with the pgoutput plugin?
>

What advantage do you see with the same?

-- 
With Regards,
Amit Kapila.





^ permalink  raw  reply  [nested|flat] 44+ messages in thread

* Re: Add a hook for handling logical decoding messages on subscribers.
@ 2026-06-23 18:01  Masahiko Sawada <sawada.mshk@gmail.com>
  parent: Amit Kapila <amit.kapila16@gmail.com>
  0 siblings, 2 replies; 44+ messages in thread

From: Masahiko Sawada @ 2026-06-23 18:01 UTC (permalink / raw)
  To: Amit Kapila <amit.kapila16@gmail.com>; +Cc: Bharath Rupireddy <bharath.rupireddyforpostgres@gmail.com>; pgsql-hackers

On Tue, Jun 23, 2026 at 1:52 AM Amit Kapila <amit.kapila16@gmail.com> wrote:
>
> On Tue, Jun 23, 2026 at 4:09 AM Bharath Rupireddy
> <bharath.rupireddyforpostgres@gmail.com> wrote:
> >
> > On Fri, Jun 19, 2026 at 3:34 PM Masahiko Sawada <sawada.mshk@gmail.com> wrote:
> > >
> > > Hi all,
> > >
> > > Commit ac4645c015 allows pgoutput to send logical decoding messages,
> > > but it's limited to applications that use the pgoutput plugin -- the
> > > built-in logical replication doesn't use it. I'd like to propose
> > > introducing a hook to the logical replication message handling so that
> > > extensions can plug in their own handling routine. This feature can be
> > > used for extensions to implement DDL replication, function
> > > replication, or trigger user-specific routines on the subscriber side.
> >
> > Thanks for working on this!
> >
> > > I've attached the PoC patch; it adds a hook function, and adds a new
> > > 'message' subscription option that allows the user to request the
> > > publisher to send logical decoding messages. Therefore, users need to
> > > enable the 'message' option and set up the hook function at server
> > > startup in order to receive the messages and trigger the hook
> > > function.
> >
> > I understand the intent of the proposal, but I'd like to get the
> > bigger picture first.
> >
> > Do we have any external modules that actually implement DDL
> > replication (or any of the listed use-cases) with a similar hook? Or
> > any existing discussion? I could be missing something because I
> > haven't looked at all the DDL replication related threads.
> >
> > Another thing I'm curious about - why a hook? Is the plan to implement
> > DDL replication as an external module rather than in core? If DDL
> > replication eventually gets into core, I'd expect it to be apply-side
> > logic executing the decoded DDL messages directly, not something going
> > through a hook.
> >
>
> I think it is important to have some example extension implementation
> to see how the hook could be utilized. One more use of such a hook
> could be to use for audit of DDLs replayed on subscribers.

Right. I'm implementing a basic DDL replication solution using this
hook as a sample implementation. Other than that, this feature can be
used to add additional information for the replicated transaction that
is not replicated via logical replication protocol, such as the
executed user or context-specific information, which can then be
dispatched to a CDC system connected to the subscriber.

> BTW, can we
> also consider it as a solution implementing basic DDL replication for
> tables? The key question is what if someday we have in-core DDL
> replication. I think extensions can still be used to implement
> filtering or transformation of DDL. We can implement capture of DDL
> using JSON format [1] so that it is forward compatible with in-core
> DDL replication. So considering that, the extension handlers will look
> like:

While a common JSON format might be helpful for forward compatibility
with in-core DDL replication, I think that we need to keep the generic
logical decoding messages used by this feature distinct from the
messages that would be used for DDL replication. A generic logical
decoding message can be written via pg_logical_emit_message() SQL
function, which is granted to PUBLIC, so any user could directly emit
a DDL command in the JSON format with the 'pg_ddl' prefix and create
arbitrary tables on the subscriber, which could cause a privilege
escalation problem. I don't think we should build something as
powerful as DDL execution on top of a message channel that any user
can write to.

Regards,

-- 
Masahiko Sawada
Amazon Web Services: https://aws.amazon.com





^ permalink  raw  reply  [nested|flat] 44+ messages in thread

* Re: Add a hook for handling logical decoding messages on subscribers.
@ 2026-06-23 18:48  Masahiko Sawada <sawada.mshk@gmail.com>
  parent: Bharath Rupireddy <bharath.rupireddyforpostgres@gmail.com>
  1 sibling, 0 replies; 44+ messages in thread

From: Masahiko Sawada @ 2026-06-23 18:48 UTC (permalink / raw)
  To: Bharath Rupireddy <bharath.rupireddyforpostgres@gmail.com>; +Cc: pgsql-hackers

On Mon, Jun 22, 2026 at 3:39 PM Bharath Rupireddy
<bharath.rupireddyforpostgres@gmail.com> wrote:
>
> Hi,
>
> On Fri, Jun 19, 2026 at 3:34 PM Masahiko Sawada <sawada.mshk@gmail.com> wrote:
> >
> > Hi all,
> >
> > Commit ac4645c015 allows pgoutput to send logical decoding messages,
> > but it's limited to applications that use the pgoutput plugin -- the
> > built-in logical replication doesn't use it. I'd like to propose
> > introducing a hook to the logical replication message handling so that
> > extensions can plug in their own handling routine. This feature can be
> > used for extensions to implement DDL replication, function
> > replication, or trigger user-specific routines on the subscriber side.
>
> Thanks for working on this!
>
> > I've attached the PoC patch; it adds a hook function, and adds a new
> > 'message' subscription option that allows the user to request the
> > publisher to send logical decoding messages. Therefore, users need to
> > enable the 'message' option and set up the hook function at server
> > startup in order to receive the messages and trigger the hook
> > function.
>
> I understand the intent of the proposal, but I'd like to get the
> bigger picture first.
>
> Do we have any external modules that actually implement DDL
> replication (or any of the listed use-cases) with a similar hook? Or
> any existing discussion? I could be missing something because I
> haven't looked at all the DDL replication related threads.

Not that I'm aware of. That said, in past DDL replication discussions
several approaches were proposed (e.g. sending the DDL query text vs.
sending schema diffs in some form of intermediate representation),
each with its own pros and cons. So even once we have an in-core DDL
replication implementation, I think it's still plausible that an
extension would want to implement a different approach, and this hook
would let it do so without patching core.

>
> Another thing I'm curious about - why a hook? Is the plan to implement
> DDL replication as an external module rather than in core? If DDL
> replication eventually gets into core, I'd expect it to be apply-side
> logic executing the decoded DDL messages directly, not something going
> through a hook.

I'm working on in-core DDL replication too, and I agree that the
in-core version would be apply-side logic executing the decoded DDL
directly, not something going through this hook. The hook isn't meant
to be to implement the in-core DDL replication. It's there for things
core wouldn't cover: auditing replayed DDLs, propagating
context-specific information to the subscriber, or an extension
implementing a different DDL replication approach than the in-core
one.

> Why not a hook at apply_dispatch to give external modules more freedom
> with the pgoutput plugin?

It would let an extension intercept every message type
(INSERT/UPDATE/DELETE, COMMIT, etc.) and could easily break apply
consistency. At the same time, the incoming data is just the standard
logical replication protocol, so there isn't much an extension could
usefully do with the non-MESSAGE types beyond what core already does.

>
> > I I went with a hook function in the patch. While it lets you chain
> > the multiple hook functions, providing the registration API might be
> > better, or other types of registry can also be considered.
>
> It's hard to tell how many external modules would make use of this
> hook (rather, how many external modules implementing this hook one
> would allow to be installed in a production database requiring
> chaining), but my first thought is that a registration-based API along
> the lines of RegisterXactCallback would be cleaner and work better.

Yeah, it would be better.

>
> > Feedback is very welcome.
>
> A few comments on the patch:
>
> 1/
> + bool message; /* True if the subscription wants to receive
> + * logical messages */
>  } Subscription;
>
> Nit: I'd call these logical decoding messages or generic logical
> messages - something to match the docs and pg_logical_emit_message.

Will fix.

> 2/
> +void
> +test_logical_message_handler(LogicalRepMessageData *msg)
> +{
> + ereport(LOG,
> + (errmsg("received message: LSN %X/%08X, prefix: %s, message: %s",
> + LSN_FORMAT_ARGS(msg->lsn),
> + msg->prefix,
> + msg->message)));
> +}
>
> Why not have the test extension do a simple DDL/function/event-trigger
> based replication? It doesn't need to be a full-blown implementation,
> but to show the usefulness of this hook, it's better to have one
> demonstrating the listed use-cases.

I'd prefer to keep this test module minimal, since its purpose is to
improve coverage of the newly added code. A worked example that
demonstrates the usefulness of the hook is valuable, but I think it
belongs in a separate contrib module rather than in the test module,
and that's probably a separate discussion.

>
> 3/
> + if (subinfo->submessage)
> + appendPQExpBufferStr(query, ", message = true");
>
> Why a subscription-level option? Why not leave the decision of whether
> or not to act on the message to the external module implementers?

This is because not to affect the existing logical replication users.
Since the logical decoding messages are not sent today, I think that
it should be an explcit opt-in feature for users who don't want to
allow the publisher to send logical decoding messages.

>
> 4/
> + /*
> + * Logical messages are handled only the (parallel) apply workers
> + */
> + if (am_tablesync_worker() || am_sequencesync_worker())
> + return;
>
> Why these restrictions? Why not leave the decision to external module
> implementers? Isn't this limiting - what if someone wants to use this
> hook for the schema sync during the initial table sync phase?

Logical decoding messages are relation-agnostic, so they don't map
cleanly onto the table sync phase. If every tablesync worker processed
the messages they'd be handled multiple times, and if only one did,
there'd be no well-defined ordering between a message and the initial
copy progress. So I think letting tablesync workers run the hook seems
more confusing than useful.

>
> 5/ With this change, pg_logical_emit_message does affect the logical
> replication apply if the subscriber has defined this hook. I think
> it's worth mentioning in the docs for pg_logical_emit_message.

Agreed.

Regards,

--
Masahiko Sawada

Amazon Web Services: https://aws.amazon.com





^ permalink  raw  reply  [nested|flat] 44+ messages in thread

* Re: Add a hook for handling logical decoding messages on subscribers.
@ 2026-06-24 00:21  Masahiko Sawada <sawada.mshk@gmail.com>
  parent: Masahiko Sawada <sawada.mshk@gmail.com>
  1 sibling, 0 replies; 44+ messages in thread

From: Masahiko Sawada @ 2026-06-24 00:21 UTC (permalink / raw)
  To: Amit Kapila <amit.kapila16@gmail.com>; +Cc: Bharath Rupireddy <bharath.rupireddyforpostgres@gmail.com>; pgsql-hackers

On Tue, Jun 23, 2026 at 11:01 AM Masahiko Sawada <sawada.mshk@gmail.com> wrote:
>
> On Tue, Jun 23, 2026 at 1:52 AM Amit Kapila <amit.kapila16@gmail.com> wrote:
> >
> > On Tue, Jun 23, 2026 at 4:09 AM Bharath Rupireddy
> > <bharath.rupireddyforpostgres@gmail.com> wrote:
> > >
> > > On Fri, Jun 19, 2026 at 3:34 PM Masahiko Sawada <sawada.mshk@gmail.com> wrote:
> > > >
> > > > Hi all,
> > > >
> > > > Commit ac4645c015 allows pgoutput to send logical decoding messages,
> > > > but it's limited to applications that use the pgoutput plugin -- the
> > > > built-in logical replication doesn't use it. I'd like to propose
> > > > introducing a hook to the logical replication message handling so that
> > > > extensions can plug in their own handling routine. This feature can be
> > > > used for extensions to implement DDL replication, function
> > > > replication, or trigger user-specific routines on the subscriber side.
> > >
> > > Thanks for working on this!
> > >
> > > > I've attached the PoC patch; it adds a hook function, and adds a new
> > > > 'message' subscription option that allows the user to request the
> > > > publisher to send logical decoding messages. Therefore, users need to
> > > > enable the 'message' option and set up the hook function at server
> > > > startup in order to receive the messages and trigger the hook
> > > > function.
> > >
> > > I understand the intent of the proposal, but I'd like to get the
> > > bigger picture first.
> > >
> > > Do we have any external modules that actually implement DDL
> > > replication (or any of the listed use-cases) with a similar hook? Or
> > > any existing discussion? I could be missing something because I
> > > haven't looked at all the DDL replication related threads.
> > >
> > > Another thing I'm curious about - why a hook? Is the plan to implement
> > > DDL replication as an external module rather than in core? If DDL
> > > replication eventually gets into core, I'd expect it to be apply-side
> > > logic executing the decoded DDL messages directly, not something going
> > > through a hook.
> > >
> >
> > I think it is important to have some example extension implementation
> > to see how the hook could be utilized. One more use of such a hook
> > could be to use for audit of DDLs replayed on subscribers.
>
> Right. I'm implementing a basic DDL replication solution using this
> hook as a sample implementation.

FYI, I've implemented a test extension, pg_logical_ddl[1], which can
be used with the proposed patch.

The extension uses both hooks ProcessUtility_hook to capture DDL
events and LogicalRepMessageHandle_hook to apply the DDL message.  It
sends a JSON blob containing the DDL command, command tag, user name,
and search_path. The implementation is  straightforward and I find
that the proposed hook is useful to implement DDL replication.

One thing that would simplify the implementation, and that I'd like to
have, is an easy way to capture the OID of the affected relation (or
its object address) without event triggers. That information is useful
for figuring out which object a DDL affects: for instance, to filter
DDLs by table, or to map the command to the right object on the
subscriber.

If we use an event trigger to capture DDL events, it would be easy to
get  the object address, but it's not easy for extensions like
pg_logical_ddl that don't rely on event triggers. The OID of the
affected table can be resolved by the name and search_path but it's
not a reliable solution as its schema can be renamed without taking a
heavy lock on the table. If they have to use event triggers, it could
be cumbersome to manage additional objects (e.g., users can drop them
etc.).  It would be good to have a mechanism to easily capture the
information that is collected by event triggers today without event
triggers. FYI I proposed a similar idea before in the DDL replication
thread[2] since the same is true for the in-core DDL replication and I
have a PoC patch for that.

Also, since this is an extension, it cannot necessarily work well with
PUBLICATION; it doesn't respect the table filtering setting and some
options like publish_via_partition_root. In that sense, working with
the existing publication feature is the advantage of  the in-core DDL
replication.

Regards,

[1] https://github.com/MasahikoSawada/pg_logical_ddl
[2] https://www.postgresql.org/message-id/CAD21AoCEVO%2BzpLmQqKwZbJ5%2BrvqsJ1e0wnTNBH437p8tDw7B%3Dg%40ma...

--
Masahiko Sawada
Amazon Web Services: https://aws.amazon.com





^ permalink  raw  reply  [nested|flat] 44+ messages in thread

* Re: Add a hook for handling logical decoding messages on subscribers.
@ 2026-07-09 12:14  Fujii Masao <masao.fujii@gmail.com>
  parent: Masahiko Sawada <sawada.mshk@gmail.com>
  1 sibling, 1 reply; 44+ messages in thread

From: Fujii Masao @ 2026-07-09 12:14 UTC (permalink / raw)
  To: Masahiko Sawada <sawada.mshk@gmail.com>; +Cc: Amit Kapila <amit.kapila16@gmail.com>; Bharath Rupireddy <bharath.rupireddyforpostgres@gmail.com>; pgsql-hackers

On Wed, Jun 24, 2026 at 3:02 AM Masahiko Sawada <sawada.mshk@gmail.com> wrote:
> Right. I'm implementing a basic DDL replication solution using this
> hook as a sample implementation. Other than that, this feature can be
> used to add additional information for the replicated transaction that
> is not replicated via logical replication protocol, such as the
> executed user or context-specific information, which can then be
> dispatched to a CDC system connected to the subscriber.

+1 for the proposed hook, although I haven't read the patch yet.

I know of a system that uses logical decoding messages to propagate data
to external systems. In that system, the application writes the data to
be propagated as logical decoding messages, and a CDC pipeline consumes
them via logical decoding and sends them to other systems, for example
through Kafka.

In that system, when the remote site is maintained by physical replication,
the same logical decoding messages can be decoded on the standby
at the remote site to deliver them to external systems there. However,
if the remote site uses logical replication instead, those messages are
currently neither delivered to nor processed on the subscriber at
the remote site. As a result, the CDC pipeline at the remote site cannot
consume the data they carry.

The proposed hook might be useful for this use case. An extension could
process the incoming logical decoding messages on the subscriber and
forward them to the local CDC pipeline, or store or re-emit them in a form
that local consumers can process.

Regards,

-- 
Fujii Masao






^ permalink  raw  reply  [nested|flat] 44+ messages in thread

* Re: Add a hook for handling logical decoding messages on subscribers.
@ 2026-08-04 00:41  Masahiko Sawada <sawada.mshk@gmail.com>
  parent: Fujii Masao <masao.fujii@gmail.com>
  0 siblings, 3 replies; 44+ messages in thread

From: Masahiko Sawada @ 2026-08-04 00:41 UTC (permalink / raw)
  To: Fujii Masao <masao.fujii@gmail.com>; +Cc: Amit Kapila <amit.kapila16@gmail.com>; Bharath Rupireddy <bharath.rupireddyforpostgres@gmail.com>; pgsql-hackers

On Thu, Jul 9, 2026 at 5:14 AM Fujii Masao <masao.fujii@gmail.com> wrote:
>
> On Wed, Jun 24, 2026 at 3:02 AM Masahiko Sawada <sawada.mshk@gmail.com> wrote:
> > Right. I'm implementing a basic DDL replication solution using this
> > hook as a sample implementation. Other than that, this feature can be
> > used to add additional information for the replicated transaction that
> > is not replicated via logical replication protocol, such as the
> > executed user or context-specific information, which can then be
> > dispatched to a CDC system connected to the subscriber.
>
> +1 for the proposed hook, although I haven't read the patch yet.
>
> I know of a system that uses logical decoding messages to propagate data
> to external systems. In that system, the application writes the data to
> be propagated as logical decoding messages, and a CDC pipeline consumes
> them via logical decoding and sends them to other systems, for example
> through Kafka.
>
> In that system, when the remote site is maintained by physical replication,
> the same logical decoding messages can be decoded on the standby
> at the remote site to deliver them to external systems there. However,
> if the remote site uses logical replication instead, those messages are
> currently neither delivered to nor processed on the subscriber at
> the remote site. As a result, the CDC pipeline at the remote site cannot
> consume the data they carry.
>
> The proposed hook might be useful for this use case. An extension could
> process the incoming logical decoding messages on the subscriber and
> forward them to the local CDC pipeline, or store or re-emit them in a form
> that local consumers can process.

Thank you for sharing the concrete use case. This is one of the use
cases I initially imagined.

I've rebased and updated the patch. Please review it.

Regards,

-- 
Masahiko Sawada
Amazon Web Services: https://aws.amazon.com

Attachments:

v2-0001-Add-a-hook-for-handling-logical-messages-on-subsc.patch (text/x-patch, 108.1K) download From 12bcc34a6a47bdd04aa994f8a790155cf45b4fb6 Mon Sep 17 00:00:00 2001 From: Masahiko Sawada <sawada.mshk@gmail.com> Date: Mon, 8 Jun 2026 11:25:47 -0700 Subject: [PATCH v2] Add a hook for handling logical messages on subscribers. pgoutput can emit logical decoding messages (those written with pg_logical_emit_message()), but the built-in subscriber never requested them from the publisher, and the apply worker had only a placeholder that discarded the MESSAGE message type. Add a "message" subscription option (default false) that asks the publisher to send these messages, and dispatch each received message to a new LogicalRepMessageHandle_hook. This lets an extension act on messages on the subscriber while still relying on the built-in apply worker for streaming, conflict handling, and progress tracking, instead of writing a separate logical decoding client. When the option is off or no handler is installed, behavior is unchanged. Logical messages are a general-purpose, in-band side channel, so the possible consumers are varied: replicating DDL by emitting deparsed statements on the publisher and executing them on the subscriber; change data capture and auditing, where a message carries context (the acting user, request id, or business reason) that the row data does not; and cache invalidation or event notification, where the message is delivered in commit order relative to the data it refers to. A transactional message is applied as a step of the remote transaction it belongs to and commits atomically with it; a non-transactional message is committed as an independent unit without advancing replication progress. Either way a message may be delivered more than once (for example after an apply worker restart), so handlers must be idempotent. Transactional messages also honor transaction streaming and ALTER SUBSCRIPTION ... SKIP. XXX Bump catalog version for the new pg_subscription.submessage column. Discussion: https://www.postgresql.org/message-id/CAD21AoCTNGiddikkUcDKj5QLnsg-51bpr-o6L-GTHWZL4ZFYtQ%40mail.gmail.com --- doc/src/sgml/catalogs.sgml | 11 + doc/src/sgml/func/func-admin.sgml | 7 +- doc/src/sgml/ref/alter_subscription.sgml | 5 +- doc/src/sgml/ref/create_subscription.sgml | 40 ++++ src/backend/catalog/pg_subscription.c | 1 + src/backend/catalog/system_views.sql | 3 +- src/backend/commands/subscriptioncmds.c | 25 +- .../libpqwalreceiver/libpqwalreceiver.c | 4 + src/backend/replication/logical/proto.c | 29 +++ src/backend/replication/logical/worker.c | 59 ++++- src/bin/pg_dump/pg_dump.c | 16 +- src/bin/pg_dump/pg_dump.h | 1 + src/bin/pg_dump/t/002_pg_dump.pl | 4 +- src/bin/psql/describe.c | 7 +- src/include/catalog/pg_subscription.h | 5 + src/include/replication/logicalproto.h | 13 ++ src/include/replication/walreceiver.h | 1 + src/include/replication/worker_internal.h | 19 ++ src/test/modules/Makefile | 1 + src/test/modules/meson.build | 1 + .../modules/test_logicalmsg_hooks/.gitignore | 4 + .../modules/test_logicalmsg_hooks/Makefile | 20 ++ .../modules/test_logicalmsg_hooks/meson.build | 28 +++ .../test_logicalmsg_hooks/t/001_basic.pl | 220 ++++++++++++++++++ .../test_logicalmsg_hooks.c | 104 +++++++++ src/test/regress/expected/subscription.out | 202 +++++++++------- src/test/regress/sql/subscription.sql | 17 ++ src/tools/pgindent/typedefs.list | 1 + 28 files changed, 742 insertions(+), 106 deletions(-) create mode 100644 src/test/modules/test_logicalmsg_hooks/.gitignore create mode 100644 src/test/modules/test_logicalmsg_hooks/Makefile create mode 100644 src/test/modules/test_logicalmsg_hooks/meson.build create mode 100644 src/test/modules/test_logicalmsg_hooks/t/001_basic.pl create mode 100644 src/test/modules/test_logicalmsg_hooks/test_logicalmsg_hooks.c diff --git a/doc/src/sgml/catalogs.sgml b/doc/src/sgml/catalogs.sgml index 6066c4784f4..50c57519acd 100644 --- a/doc/src/sgml/catalogs.sgml +++ b/doc/src/sgml/catalogs.sgml @@ -8702,6 +8702,17 @@ SCRAM-SHA-256$<replaceable>&lt;iteration count&gt;</replaceable>:<replaceable>&l </para></entry> </row> + <row> + <entry role="catalog_table_entry"><para role="column_definition"> + <structfield>submessage</structfield> <type>bool</type> + </para> + <para> + If true, the subscription requests the publisher to send logical + decoding messages, which are passed to a handler registered by an + extension on the subscriber. + </para></entry> + </row> + <row> <entry role="catalog_table_entry"><para role="column_definition"> <structfield>submaxretention</structfield> <type>int4</type> diff --git a/doc/src/sgml/func/func-admin.sgml b/doc/src/sgml/func/func-admin.sgml index 0eae1c1f616..3d42500f33a 100644 --- a/doc/src/sgml/func/func-admin.sgml +++ b/doc/src/sgml/func/func-admin.sgml @@ -1479,7 +1479,12 @@ postgres=# SELECT '0/0'::pg_lsn + pd.segment_number * ps.setting::int + :offset <literal>false</literal>) controls if the message is immediately flushed to WAL or not. <parameter>flush</parameter> has no effect with <parameter>transactional</parameter>, as the message's WAL - record is flushed along with its transaction. + record is flushed along with its transaction. In addition to logical + decoding clients, messages emitted by this function are delivered to + subscriber whose subscription has + <link linkend="sql-createsubscription-params-with-message"><literal>message</literal></link> + enabled, where they are passed to a handler registered by an extension + on the subscriber. </para></entry> </row> diff --git a/doc/src/sgml/ref/alter_subscription.sgml b/doc/src/sgml/ref/alter_subscription.sgml index 0f81af5608b..b41c0ca4c05 100644 --- a/doc/src/sgml/ref/alter_subscription.sgml +++ b/doc/src/sgml/ref/alter_subscription.sgml @@ -309,8 +309,9 @@ ALTER SUBSCRIPTION <replaceable class="parameter">name</replaceable> RENAME TO < <link linkend="sql-createsubscription-params-with-two-phase"><literal>two_phase</literal></link>, <link linkend="sql-createsubscription-params-with-retain-dead-tuples"><literal>retain_dead_tuples</literal></link>, <link linkend="sql-createsubscription-params-with-max-retention-duration"><literal>max_retention_duration</literal></link>, - <link linkend="sql-createsubscription-params-with-wal-receiver-timeout"><literal>wal_receiver_timeout</literal></link>, and - <link linkend="sql-createsubscription-params-with-conflict-log-destination"><literal>conflict_log_destination</literal></link>. + <link linkend="sql-createsubscription-params-with-wal-receiver-timeout"><literal>wal_receiver_timeout</literal></link>, + <link linkend="sql-createsubscription-params-with-conflict-log-destination"><literal>conflict_log_destination</literal></link>, and + <link linkend="sql-createsubscription-params-with-message"><literal>message</literal></link>. Only a superuser can set <literal>password_required = false</literal>. </para> diff --git a/doc/src/sgml/ref/create_subscription.sgml b/doc/src/sgml/ref/create_subscription.sgml index 81fbf3487a4..afac5ad2442 100644 --- a/doc/src/sgml/ref/create_subscription.sgml +++ b/doc/src/sgml/ref/create_subscription.sgml @@ -478,6 +478,46 @@ CREATE SUBSCRIPTION <replaceable class="parameter">subscription_name</replaceabl </listitem> </varlistentry> + <varlistentry id="sql-createsubscription-params-with-message"> + <term><literal>message</literal> (<type>boolean</type>)</term> + <listitem> + <para> + Specifies whether the subscription requests the publisher to send + logical decoding messages, that is, those emitted with + <link linkend="pg-logical-emit-message"><function>pg_logical_emit_message</function></link>. + The default is <literal>false</literal>. + </para> + <para> + Received messages are only acted upon if an extension installed on + the subscriber has registered a handler for them; otherwise they are + received and discarded. A message may be passed to the handler more + than once, for example after an apply worker restart, so a handler + must be prepared to process the same message repeatedly. + </para> + <para> + A transactional message is applied as part of the remote transaction + that emitted it, and is therefore delivered in commit order with + respect to other transactions, not in the order in which it was written. + Note that + <link linkend="sql-createsubscription-params-with-streaming"><literal>streaming</literal></link> + and <link linkend="sql-altersubscription-params-skip"><literal>SKIP</literal></link> + apply to transactional messages just as they do to other changes. A + non-transactional message is delivered as soon as it is decoded, even + if the transaction that emitted it later aborted. + </para> + <para> + Messages are not restricted to the tables or schemas of the + subscribed publications: every message emitted in the publisher + database is sent, subject only to + <link linkend="sql-createsubscription-params-with-origin"><literal>origin</literal></link> + filtering. Message contents are supplied by whoever calls + <function>pg_logical_emit_message</function> on the publisher, which + requires no special privileges, so a handler should treat them as + untrusted input. + </para> + </listitem> + </varlistentry> + <varlistentry id="sql-createsubscription-params-with-origin"> <term><literal>origin</literal> (<type>string</type>)</term> <listitem> diff --git a/src/backend/catalog/pg_subscription.c b/src/backend/catalog/pg_subscription.c index 5ff61edb989..3618161ea96 100644 --- a/src/backend/catalog/pg_subscription.c +++ b/src/backend/catalog/pg_subscription.c @@ -139,6 +139,7 @@ GetSubscription(Oid subid, bool missing_ok, bool conninfo_needed, sub->maxretention = subform->submaxretention; sub->retentionactive = subform->subretentionactive; sub->conflictlogrelid = subform->subconflictlogrelid; + sub->message = subform->submessage; if (conninfo_needed) { diff --git a/src/backend/catalog/system_views.sql b/src/backend/catalog/system_views.sql index 090281a03dd..b726f6e71a8 100644 --- a/src/backend/catalog/system_views.sql +++ b/src/backend/catalog/system_views.sql @@ -1539,7 +1539,8 @@ GRANT SELECT (oid, subdbid, subskiplsn, subname, subowner, subenabled, subpasswordrequired, subrunasowner, subfailover, subretaindeadtuples, submaxretention, subretentionactive, subserver, subconflictlogrelid, subconflictlogdest, subslotname, - subsynccommit, subwalrcvtimeout, subpublications, suborigin) + subsynccommit, subwalrcvtimeout, subpublications, suborigin, + submessage) ON pg_subscription TO public; CREATE VIEW pg_stat_subscription_stats AS diff --git a/src/backend/commands/subscriptioncmds.c b/src/backend/commands/subscriptioncmds.c index 67f5699b2c7..c13e4c84f85 100644 --- a/src/backend/commands/subscriptioncmds.c +++ b/src/backend/commands/subscriptioncmds.c @@ -83,6 +83,7 @@ #define SUBOPT_LSN 0x00020000 #define SUBOPT_ORIGIN 0x00040000 #define SUBOPT_CONFLICT_LOG_DEST 0x00080000 +#define SUBOPT_MESSAGE 0x00100000 /* check if the 'val' has 'bits' set */ #define IsSet(val, bits) (((val) & (bits)) == (bits)) @@ -109,6 +110,7 @@ typedef struct SubOpts bool runasowner; bool failover; bool retaindeadtuples; + bool message; int32 maxretention; char *origin; ConflictLogDest conflictlogdest; @@ -202,6 +204,8 @@ parse_subscription_options(ParseState *pstate, List *stmt_options, opts->failover = false; if (IsSet(supported_opts, SUBOPT_RETAIN_DEAD_TUPLES)) opts->retaindeadtuples = false; + if (IsSet(supported_opts, SUBOPT_MESSAGE)) + opts->message = false; if (IsSet(supported_opts, SUBOPT_MAX_RETENTION_DURATION)) opts->maxretention = 0; if (IsSet(supported_opts, SUBOPT_ORIGIN)) @@ -360,6 +364,15 @@ parse_subscription_options(ParseState *pstate, List *stmt_options, opts->specified_opts |= SUBOPT_RETAIN_DEAD_TUPLES; opts->retaindeadtuples = defGetBoolean(defel); } + else if (IsSet(supported_opts, SUBOPT_MESSAGE) && + strcmp(defel->defname, "message") == 0) + { + if (IsSet(opts->specified_opts, SUBOPT_MESSAGE)) + errorConflictingDefElem(defel, pstate); + + opts->specified_opts |= SUBOPT_MESSAGE; + opts->message = defGetBoolean(defel); + } else if (IsSet(supported_opts, SUBOPT_MAX_RETENTION_DURATION) && strcmp(defel->defname, "max_retention_duration") == 0) { @@ -700,7 +713,7 @@ CreateSubscription(ParseState *pstate, CreateSubscriptionStmt *stmt, SUBOPT_RETAIN_DEAD_TUPLES | SUBOPT_MAX_RETENTION_DURATION | SUBOPT_WAL_RECEIVER_TIMEOUT | SUBOPT_ORIGIN | - SUBOPT_CONFLICT_LOG_DEST); + SUBOPT_CONFLICT_LOG_DEST | SUBOPT_MESSAGE); parse_subscription_options(pstate, stmt->options, supported_opts, &opts); /* @@ -850,6 +863,7 @@ CreateSubscription(ParseState *pstate, CreateSubscriptionStmt *stmt, values[Anum_pg_subscription_subfailover - 1] = BoolGetDatum(opts.failover); values[Anum_pg_subscription_subretaindeadtuples - 1] = BoolGetDatum(opts.retaindeadtuples); + values[Anum_pg_subscription_submessage - 1] = BoolGetDatum(opts.message); values[Anum_pg_subscription_submaxretention - 1] = Int32GetDatum(opts.maxretention); values[Anum_pg_subscription_subretentionactive - 1] = @@ -1667,7 +1681,8 @@ AlterSubscription(ParseState *pstate, AlterSubscriptionStmt *stmt, SUBOPT_MAX_RETENTION_DURATION | SUBOPT_WAL_RECEIVER_TIMEOUT | SUBOPT_ORIGIN | - SUBOPT_CONFLICT_LOG_DEST); + SUBOPT_CONFLICT_LOG_DEST | + SUBOPT_MESSAGE); break; case ALTER_SUBSCRIPTION_ENABLED: @@ -2054,6 +2069,12 @@ AlterSubscription(ParseState *pstate, AlterSubscriptionStmt *stmt, } } + if (IsSet(opts.specified_opts, SUBOPT_MESSAGE)) + { + values[Anum_pg_subscription_submessage - 1] = BoolGetDatum(opts.message); + replaces[Anum_pg_subscription_submessage - 1] = true; + } + update_tuple = true; break; } diff --git a/src/backend/replication/libpqwalreceiver/libpqwalreceiver.c b/src/backend/replication/libpqwalreceiver/libpqwalreceiver.c index 029990d9fce..e345b80690f 100644 --- a/src/backend/replication/libpqwalreceiver/libpqwalreceiver.c +++ b/src/backend/replication/libpqwalreceiver/libpqwalreceiver.c @@ -631,6 +631,10 @@ libpqrcv_startstreaming(WalReceiverConn *conn, appendQuotedLiteral(&cmd, options->proto.logical.origin); } + if (options->proto.logical.messages && + PQserverVersion(conn->streamConn) >= 140000) + appendStringInfo(&cmd, ", messages 'on'"); + pubnames = options->proto.logical.publication_names; pubnames_str = stringlist_to_identifierstr(pubnames); appendStringInfoString(&cmd, ", publication_names "); diff --git a/src/backend/replication/logical/proto.c b/src/backend/replication/logical/proto.c index 86ad97cd937..973a67606be 100644 --- a/src/backend/replication/logical/proto.c +++ b/src/backend/replication/logical/proto.c @@ -660,6 +660,35 @@ logicalrep_write_message(StringInfo out, TransactionId xid, XLogRecPtr lsn, pq_sendbytes(out, message, sz); } +/* + * Read MESSAGE from stream. + */ +void +logicalrep_read_message(StringInfo in, LogicalRepMessageData *msg_data) +{ + Size len; + char *msg; + uint8 flags; + + /* read and decode flags */ + flags = pq_getmsgint(in, 1); + msg_data->transactional = (flags & MESSAGE_TRANSACTIONAL) != 0; + + msg_data->lsn = pq_getmsgint64(in); + msg_data->prefix = pstrdup(pq_getmsgstring(in)); + + /* read message length */ + len = pq_getmsgint(in, 4); + msg_data->message_size = len; + + /* and data */ + msg = palloc(len + 1); + pq_copymsgbytes(in, msg, len); + + msg[len] = '\0'; + msg_data->message = msg; +} + /* * Write relation description to the output stream. */ diff --git a/src/backend/replication/logical/worker.c b/src/backend/replication/logical/worker.c index 2548d3feb54..7d5706e1598 100644 --- a/src/backend/replication/logical/worker.c +++ b/src/backend/replication/logical/worker.c @@ -549,6 +549,9 @@ typedef struct ApplySubXactData static ApplySubXactData subxact_data = {0, 0, InvalidTransactionId, NULL}; +/* Hook for plugins to get control in handling logical decoding messages */ +LogicalRepMessageHandle_hook_type LogicalRepMessageHandle_hook = NULL; + static inline void subxact_filename(char *path, Oid subid, TransactionId xid); static inline void changes_filename(char *path, Oid subid, TransactionId xid); @@ -1694,6 +1697,51 @@ apply_handle_origin(StringInfo s) errmsg_internal("ORIGIN message sent out of order"))); } +/* + * Handle MESSAGE message. + * + * Invoke a hook function if set. + */ +static void +apply_handle_message(StringInfo s) +{ + LogicalRepMessageData msg; + + if (!LogicalRepMessageHandle_hook) + return; + + /* + * Logical messages are relation-agnostic, so they don't map cleanly onto + * the tablesync worker of a particular relation, and there would be no + * well-defined ordering between a message and the initial copy. Leave + * them to the (parallel) apply workers. Logical messages are handled only + * the (parallel) apply workers + */ + if (am_tablesync_worker() || am_sequencesync_worker()) + return; + + if (is_skipping_changes() || + handle_streamed_transaction(LOGICAL_REP_MSG_MESSAGE, s)) + return; + + logicalrep_read_message(s, &msg); + + begin_replication_step(); + + (*LogicalRepMessageHandle_hook) (&msg); + + end_replication_step(); + + /* + * A transactional message is applied as a step of the remote transaction + * that emitted it, and is committed together with it when applying the + * commit message. A non-transactional message belongs to no remote + * transaction, so commit it here. + */ + if (!msg.transactional) + CommitTransactionCommand(); +} + /* * Initialize fileset (if not already done). * @@ -3846,12 +3894,7 @@ apply_dispatch(StringInfo s) break; case LOGICAL_REP_MSG_MESSAGE: - - /* - * Logical replication does not use generic logical messages yet. - * Although, it could be used by other applications that use this - * output plugin. - */ + apply_handle_message(s); break; case LOGICAL_REP_MSG_STREAM_START: @@ -5128,7 +5171,8 @@ maybe_reread_subscription(void) newsub->passwordrequired != MySubscription->passwordrequired || strcmp(newsub->origin, MySubscription->origin) != 0 || newsub->owner != MySubscription->owner || - !equal(newsub->publications, MySubscription->publications)) + !equal(newsub->publications, MySubscription->publications) || + newsub->message != MySubscription->message) { if (am_parallel_apply_worker()) ereport(LOG, @@ -5614,6 +5658,7 @@ set_stream_options(WalRcvStreamOptions *options, options->proto.logical.twophase = false; options->proto.logical.origin = pstrdup(MySubscription->origin); + options->proto.logical.messages = MySubscription->message; } /* diff --git a/src/bin/pg_dump/pg_dump.c b/src/bin/pg_dump/pg_dump.c index 4948e6d80c7..015770e0b5d 100644 --- a/src/bin/pg_dump/pg_dump.c +++ b/src/bin/pg_dump/pg_dump.c @@ -5138,6 +5138,7 @@ getSubscriptions(Archive *fout) int i_subfailover; int i_subretaindeadtuples; int i_submaxretention; + int i_submessage; int i, ntups; @@ -5236,9 +5237,14 @@ getSubscriptions(Archive *fout) " '-1' AS subwalrcvtimeout,\n"); if (fout->remoteVersion >= 190000) - appendPQExpBufferStr(query, " fs.srvname AS subservername\n"); + appendPQExpBufferStr(query, " fs.srvname AS subservername,\n"); else - appendPQExpBufferStr(query, " NULL AS subservername\n"); + appendPQExpBufferStr(query, " NULL AS subservername,\n"); + + if (fout->remoteVersion >= 200000) + appendPQExpBufferStr(query, " s.submessage\n"); + else + appendPQExpBufferStr(query, "false AS submessage\n"); appendPQExpBufferStr(query, "FROM pg_subscription s\n"); @@ -5287,6 +5293,7 @@ getSubscriptions(Archive *fout) i_subpublications = PQfnumber(res, "subpublications"); i_suborigin = PQfnumber(res, "suborigin"); i_suboriginremotelsn = PQfnumber(res, "suboriginremotelsn"); + i_submessage = PQfnumber(res, "submessage"); subinfo = pg_malloc_array(SubscriptionInfo, ntups); @@ -5344,6 +5351,8 @@ getSubscriptions(Archive *fout) else subinfo[i].suboriginremotelsn = pg_strdup(PQgetvalue(res, i, i_suboriginremotelsn)); + subinfo[i].submessage = + (strcmp(PQgetvalue(res, i, i_submessage), "t") == 0); /* Decide whether we want to dump it */ selectDumpableObject(&(subinfo[i].dobj), fout); @@ -5596,6 +5605,9 @@ dumpSubscription(Archive *fout, const SubscriptionInfo *subinfo) if (subinfo->subretaindeadtuples) appendPQExpBufferStr(query, ", retain_dead_tuples = true"); + if (subinfo->submessage) + appendPQExpBufferStr(query, ", message = true"); + if (subinfo->submaxretention) appendPQExpBuffer(query, ", max_retention_duration = %d", subinfo->submaxretention); diff --git a/src/bin/pg_dump/pg_dump.h b/src/bin/pg_dump/pg_dump.h index 5a6726d8b12..30bd1575a26 100644 --- a/src/bin/pg_dump/pg_dump.h +++ b/src/bin/pg_dump/pg_dump.h @@ -721,6 +721,7 @@ typedef struct _SubscriptionInfo bool subrunasowner; bool subfailover; bool subretaindeadtuples; + bool submessage; int submaxretention; char *subservername; char *subconninfo; diff --git a/src/bin/pg_dump/t/002_pg_dump.pl b/src/bin/pg_dump/t/002_pg_dump.pl index 9258948b583..019ba9350cd 100644 --- a/src/bin/pg_dump/t/002_pg_dump.pl +++ b/src/bin/pg_dump/t/002_pg_dump.pl @@ -3317,9 +3317,9 @@ my %tests = ( create_order => 50, create_sql => 'CREATE SUBSCRIPTION sub3 CONNECTION \'dbname=doesnotexist\' PUBLICATION pub1 - WITH (connect = false, origin = any, streaming = on);', + WITH (connect = false, origin = any, streaming = on, message = on);', regexp => qr/^ - \QCREATE SUBSCRIPTION sub3 CONNECTION 'dbname=doesnotexist' PUBLICATION pub1 WITH (connect = false, slot_name = 'sub3', streaming = on);\E + \QCREATE SUBSCRIPTION sub3 CONNECTION 'dbname=doesnotexist' PUBLICATION pub1 WITH (connect = false, slot_name = 'sub3', streaming = on, message = on);\E /xm, like => { %full_runs, section_post_data => 1, }, unlike => { diff --git a/src/bin/psql/describe.c b/src/bin/psql/describe.c index ad9c8affb4f..cdb6bbb2184 100644 --- a/src/bin/psql/describe.c +++ b/src/bin/psql/describe.c @@ -6894,7 +6894,7 @@ describeSubscriptions(const char *pattern, bool verbose) printQueryOpt myopt = pset.popt; static const bool translate_columns[] = {false, false, false, false, false, false, false, false, false, false, false, false, false, false, - false, false, false, false, false, false, false}; + false, false, false, false, false, false, false, false}; initPQExpBuffer(&buf); @@ -6972,6 +6972,11 @@ describeSubscriptions(const char *pattern, bool verbose) gettext_noop("Retention active")); } + if (pset.sversion >= 200000) + appendPQExpBuffer(&buf, + ", submessage AS \"%s\"\n", + gettext_noop("Message")); + appendPQExpBuffer(&buf, ", subsynccommit AS \"%s\"\n" ", subconninfo AS \"%s\"\n", diff --git a/src/include/catalog/pg_subscription.h b/src/include/catalog/pg_subscription.h index 65ce8e145fb..1194e006715 100644 --- a/src/include/catalog/pg_subscription.h +++ b/src/include/catalog/pg_subscription.h @@ -83,6 +83,9 @@ CATALOG(pg_subscription,6100,SubscriptionRelationId) BKI_SHARED_RELATION BKI_ROW bool subretaindeadtuples; /* True if dead tuples useful for * conflict detection are retained */ + bool submessage; /* True if the subscription wants to receive + * logical messages. */ + int32 submaxretention; /* The maximum duration (in milliseconds) * for which information useful for * conflict detection can be retained */ @@ -165,6 +168,8 @@ typedef struct Subscription * to be synchronized to the standbys. */ bool retaindeadtuples; /* True if dead tuples useful for conflict * detection are retained */ + bool message; /* True if the subscription wants to receive + * logical messages */ int32 maxretention; /* The maximum duration (in milliseconds) for * which information useful for conflict * detection can be retained */ diff --git a/src/include/replication/logicalproto.h b/src/include/replication/logicalproto.h index 058a955e20c..83741a2080d 100644 --- a/src/include/replication/logicalproto.h +++ b/src/include/replication/logicalproto.h @@ -191,6 +191,18 @@ typedef struct LogicalRepStreamAbortData TimestampTz abort_time; } LogicalRepStreamAbortData; +/* + * Logical decoding message information + */ +typedef struct LogicalRepMessageData +{ + XLogRecPtr lsn; + bool transactional; + const char *prefix; + Size message_size; /* length of message in bytes */ + const char *message; /* payload; may contain embedded nulls */ +} LogicalRepMessageData; + extern void logicalrep_write_begin(StringInfo out, ReorderBufferTXN *txn); extern void logicalrep_read_begin(StringInfo in, LogicalRepBeginData *begin_data); @@ -248,6 +260,7 @@ extern List *logicalrep_read_truncate(StringInfo in, bool *cascade, bool *restart_seqs); extern void logicalrep_write_message(StringInfo out, TransactionId xid, XLogRecPtr lsn, bool transactional, const char *prefix, Size sz, const char *message); +extern void logicalrep_read_message(StringInfo in, LogicalRepMessageData *msg_data); extern void logicalrep_write_rel(StringInfo out, TransactionId xid, Relation rel, Bitmapset *columns, PublishGencolsType include_gencols_type); diff --git a/src/include/replication/walreceiver.h b/src/include/replication/walreceiver.h index 5cfe90f9989..eca847c4a8f 100644 --- a/src/include/replication/walreceiver.h +++ b/src/include/replication/walreceiver.h @@ -188,6 +188,7 @@ typedef struct * prepare time */ char *origin; /* Only publish data originating from the * specified origin */ + bool messages; /* Logical message */ } logical; } proto; } WalRcvStreamOptions; diff --git a/src/include/replication/worker_internal.h b/src/include/replication/worker_internal.h index 745b7d9e969..6bbefe5b8f8 100644 --- a/src/include/replication/worker_internal.h +++ b/src/include/replication/worker_internal.h @@ -18,6 +18,7 @@ #include "miscadmin.h" #include "replication/logicalrelation.h" #include "replication/walreceiver.h" +#include "replication/logicalproto.h" #include "storage/buffile.h" #include "storage/fileset.h" #include "storage/shm_mq.h" @@ -255,6 +256,24 @@ extern PGDLLIMPORT bool InitializingApplyWorker; extern PGDLLIMPORT List *table_states_not_ready; +/* + * Hook for extensions to handle logical decoding messages (see + * pg_logical_emit_message()) received from the publisher. Called only when + * the subscription has the "message" option enabled. + * + * The handler runs in the apply worker, inside a transaction, with an active + * snapshot pushed. For a transactional message the handler's work is part of + * the remote transaction and commits with it; a non-transactional message is + * committed as an independent unit. The same message may be delivered more + * than once, so the handler must be idempotent. + * + * Note that the apply worker runs with an empty search_path. An ERROR thrown + * by the handler restarts the apply worker, or disables the subscription if + * disable_on_error is set. + */ +typedef void (*LogicalRepMessageHandle_hook_type) (LogicalRepMessageData *msg); +extern PGDLLIMPORT LogicalRepMessageHandle_hook_type LogicalRepMessageHandle_hook; + extern void logicalrep_worker_attach(int slot); extern LogicalRepWorker *logicalrep_worker_find(LogicalRepWorkerType wtype, Oid subid, Oid relid, diff --git a/src/test/modules/Makefile b/src/test/modules/Makefile index 098bb8142ae..e8a7f50eb38 100644 --- a/src/test/modules/Makefile +++ b/src/test/modules/Makefile @@ -36,6 +36,7 @@ SUBDIRS = \ test_integerset \ test_json_parser \ test_lfind \ + test_logicalmsg_hooks \ test_lwlock_tranches \ test_misc \ test_oat_hooks \ diff --git a/src/test/modules/meson.build b/src/test/modules/meson.build index 4bca42bb370..5f6c0cae77a 100644 --- a/src/test/modules/meson.build +++ b/src/test/modules/meson.build @@ -37,6 +37,7 @@ subdir('test_int128') subdir('test_integerset') subdir('test_json_parser') subdir('test_lfind') +subdir('test_logicalmsg_hooks') subdir('test_lwlock_tranches') subdir('test_misc') subdir('test_oat_hooks') diff --git a/src/test/modules/test_logicalmsg_hooks/.gitignore b/src/test/modules/test_logicalmsg_hooks/.gitignore new file mode 100644 index 00000000000..5dcb3ff9723 --- /dev/null +++ b/src/test/modules/test_logicalmsg_hooks/.gitignore @@ -0,0 +1,4 @@ +# Generated subdirectories +/log/ +/results/ +/tmp_check/ diff --git a/src/test/modules/test_logicalmsg_hooks/Makefile b/src/test/modules/test_logicalmsg_hooks/Makefile new file mode 100644 index 00000000000..af129009b8a --- /dev/null +++ b/src/test/modules/test_logicalmsg_hooks/Makefile @@ -0,0 +1,20 @@ +# src/test/modules/test_logicalmsg_hooks/Makefile + +MODULE_big = test_logicalmsg_hooks +OBJS = \ + $(WIN32RES) \ + test_logicalmsg_hooks.o +PGFILEDESC = "test_logicalmsg_hooks - test logical replication message hooks" + +TAP_TESTS = 1 + +ifdef USE_PGXS +PG_CONFIG = pg_config +PGXS := $(shell $(PG_CONFIG) --pgxs) +include $(PGXS) +else +subdir = src/test/modules/test_logicalmsg_hooks +top_builddir = ../../../.. +include $(top_builddir)/src/Makefile.global +include $(top_srcdir)/contrib/contrib-global.mk +endif diff --git a/src/test/modules/test_logicalmsg_hooks/meson.build b/src/test/modules/test_logicalmsg_hooks/meson.build new file mode 100644 index 00000000000..8084d82e47b --- /dev/null +++ b/src/test/modules/test_logicalmsg_hooks/meson.build @@ -0,0 +1,28 @@ +# Copyright (c) 2022-2026, PostgreSQL Global Development Group + +test_logicalmsg_hooks_sources = files( + 'test_logicalmsg_hooks.c', +) + +if host_system == 'windows' + test_logicalmsg_hooks_sources += rc_lib_gen.process(win32ver_rc, extra_args: [ + '--NAME', 'test_logicalmsg_hooks', + '--FILEDESC', 'test_logicalmsg_hooks - test logical message hooks',]) +endif + +test_logicalmsg_hooks = shared_module('test_logicalmsg_hooks', + test_logicalmsg_hooks_sources, + kwargs: pg_test_mod_args, +) +test_install_libs += test_logicalmsg_hooks + +tests += { + 'name': 'test_logicalmsg_hooks', + 'sd': meson.current_source_dir(), + 'bd': meson.current_build_dir(), + 'tap': { + 'tests': [ + 't/001_basic.pl', + ], + }, +} diff --git a/src/test/modules/test_logicalmsg_hooks/t/001_basic.pl b/src/test/modules/test_logicalmsg_hooks/t/001_basic.pl new file mode 100644 index 00000000000..0d7c7d90530 --- /dev/null +++ b/src/test/modules/test_logicalmsg_hooks/t/001_basic.pl @@ -0,0 +1,220 @@ +# Copyright (c) 2026, PostgreSQL Global Development Group + +# Test LogicalRepMessageHandle_hook. The test module records every message it +# receives into public.test_logicalmsg_log on the subscriber, so most checks +# here are plain queries against that table. + +use strict; +use warnings FATAL => 'all'; + +use PostgreSQL::Test::Cluster; +use PostgreSQL::Test::Utils; +use Test::More; + +my $pub = PostgreSQL::Test::Cluster->new('publisher'); +$pub->init(allows_streaming => 'logical'); + +# Keep the memory limit low so that the transactions below are streamed. +$pub->append_conf('postgresql.conf', "logical_decoding_work_mem = 64kB\n"); +$pub->start; + +my $sub = PostgreSQL::Test::Cluster->new('subscriber'); +$sub->init(allows_streaming => 'logical'); +$sub->append_conf( + 'postgresql.conf', q{ +shared_preload_libraries = 'test_logicalmsg_hooks' +max_parallel_apply_workers_per_subscription = 2 +}); +$sub->start; + +$pub->safe_psql( + 'postgres', qq{ +CREATE TABLE test (a int); +INSERT INTO test VALUES (1); +CREATE PUBLICATION pub FOR ALL TABLES; +}); + +# The log table has to exist before any message arrives; the hook silently +# skips recording while it is missing. +$sub->safe_psql( + 'postgres', qq{ +CREATE TABLE test (a int primary key); +CREATE TABLE test_logicalmsg_log ( + lsn pg_lsn, + transactional boolean, + prefix text, + message_size int, + message text +); +}); + +my $pub_connstr = $pub->connstr . ' dbname=postgres'; + +$sub->safe_psql( + 'postgres', qq{ +CREATE SUBSCRIPTION sub CONNECTION '$pub_connstr' PUBLICATION pub WITH (message = true, disable_on_error = true) +}); + +$sub->wait_for_subscription_sync($pub, 'sub'); + +# Number of messages recorded under the given prefix. +sub logged_count +{ + my ($prefix) = @_; + + return $sub->safe_psql('postgres', + "SELECT count(*) FROM test_logicalmsg_log WHERE prefix = '$prefix'"); +} + +# Verify that a transactional message is passed to the hook, and that the +# message data given to the hook is correct. +$pub->safe_psql('postgres', + q{SELECT pg_logical_emit_message(true, 'basic', 'transactional message')} +); +$pub->wait_for_catchup('sub'); + +is( $sub->safe_psql( + 'postgres', + q{SELECT transactional, message_size, message FROM test_logicalmsg_log + WHERE prefix = 'basic'}), + 't|21|transactional message', + "transactional message is passed to the hook"); + +# Verify that a non-transactional message is applied even if the transaction +# that emitted it rolls back. +$pub->safe_psql( + 'postgres', q{ +BEGIN; +SELECT pg_logical_emit_message(false, 'nontransactional', 'emitted then rolled back'); +ROLLBACK; +}); + +# Commit another message to flush WAL, as ROLLBACK doesn't flush. +$pub->safe_psql('postgres', + q{SELECT pg_logical_emit_message(true, 'flush', 'flush')}); +$pub->wait_for_catchup('sub'); + +is( $sub->safe_psql( + 'postgres', + q{SELECT transactional, message FROM test_logicalmsg_log + WHERE prefix = 'nontransactional'}), + 'f|emitted then rolled back', + "non-transactional message of an aborted transaction is applied"); + +# Verify that a transactional message emitted in a subtransaction that is +# rolled back is not passed to the hook. +$pub->safe_psql( + 'postgres', q{ +BEGIN; +SAVEPOINT sp; +SELECT pg_logical_emit_message(true, 'subxact_aborted', 'not delivered'); +ROLLBACK TO SAVEPOINT sp; +SELECT pg_logical_emit_message(true, 'subxact_committed', 'delivered'); +COMMIT; +}); +$pub->wait_for_catchup('sub'); + +is(logged_count('subxact_aborted'), + 0, "message of an aborted subtransaction is not applied"); +is(logged_count('subxact_committed'), + 1, "message of a committed subtransaction is applied"); + +# Verify that transactional messages are applied in streaming transactions, +# both when the leader apply worker serializes the transaction to a file and +# when it hands the transaction to a parallel apply worker. +foreach my $mode ('on', 'parallel') +{ + $sub->safe_psql('postgres', + "ALTER SUBSCRIPTION sub SET (streaming = $mode)"); + + # Reconnect so that the new streaming mode takes effect immediately. + $sub->safe_psql('postgres', 'ALTER SUBSCRIPTION sub DISABLE'); + $sub->safe_psql('postgres', 'ALTER SUBSCRIPTION sub ENABLE'); + + $pub->safe_psql('postgres', + q{SELECT pg_stat_reset_replication_slot('sub')}); + + $pub->safe_psql( + 'postgres', qq{ +BEGIN; +INSERT INTO test SELECT generate_series(1000, 6000); +SELECT pg_logical_emit_message(true, 'streaming_$mode', 'streamed message'); +COMMIT; +}); + $pub->wait_for_catchup('sub'); + + # Make sure the transaction really was streamed, so that this test cannot + # silently degrade into a non-streaming one. + $pub->poll_query_until('postgres', + q{SELECT stream_txns > 0 FROM pg_stat_replication_slots WHERE slot_name = 'sub'} + ) or die "transaction was not streamed"; + + is(logged_count("streaming_$mode"), + 1, + "message in a streamed transaction is applied (streaming = $mode)"); + + $pub->safe_psql('postgres', 'TRUNCATE test'); + $pub->wait_for_catchup('sub'); +} + +$sub->safe_psql('postgres', 'ALTER SUBSCRIPTION sub SET (streaming = off)'); + +# Verify that ALTER SUBSCRIPTION ... SKIP skips a transactional message along +# with the rest of its transaction. The message is emitted before the +# conflicting change, so the hook runs and records the message on the first +# attempt; that work must be rolled back with the failed transaction, and must +# not be redone once the transaction is skipped. +$pub->safe_psql('postgres', 'INSERT INTO test VALUES (1)'); +$pub->wait_for_catchup('sub'); + +my $log_location = -s $sub->logfile; + +$pub->safe_psql( + 'postgres', q{ +BEGIN; +SELECT pg_logical_emit_message(true, 'skipped', 'rolled back, then skipped'); +INSERT INTO test VALUES (1); +COMMIT; +}); + +# Wait until the conflict disables the subscription. +$sub->poll_query_until('postgres', + q{SELECT subenabled = FALSE FROM pg_subscription WHERE subname = 'sub'}); + +# The hook did run before the transaction failed. +ok( $sub->log_contains( + qr/LOG[^\n]+received message: [^\n]+prefix: skipped/, + $log_location), + "hook is called for a message in a transaction that later fails"); + +# ... but its work was rolled back with the transaction. +is(logged_count('skipped'), 0, + "work done by the hook is rolled back with the remote transaction"); + +# Get the finish LSN of the failed transaction. +my $contents = slurp_file($sub->logfile, $log_location); +$contents =~ + qr/conflict detected on relation "public.test".*\n.*DETAIL:.*Could not apply remote change.*\n.*Key already exists in unique index "test_pkey", modified in transaction \d+: key .*, local row .*\n.*CONTEXT:.* for replication target relation "public.test" in transaction \d+, finished at ([[:xdigit:]]+\/[[:xdigit:]]+)/m + or die "could not get error-LSN"; +my $lsn = $1; + +$log_location = -s $sub->logfile; + +# Set skip LSN and re-enable the subscription. +$sub->safe_psql('postgres', qq{ALTER SUBSCRIPTION sub SKIP (lsn = '$lsn')}); +$sub->safe_psql('postgres', 'ALTER SUBSCRIPTION sub ENABLE'); + +# Wait for the failed transaction to be skipped. +$sub->poll_query_until('postgres', + q{SELECT subskiplsn = '0/0' FROM pg_subscription WHERE subname = 'sub'}); + +ok( !$sub->log_contains( + qr/LOG[^\n]+received message: [^\n]+prefix: skipped/, + $log_location), + "hook is not called for a message in a skipped transaction"); +is(logged_count('skipped'), 0, + "message in a skipped transaction is not applied"); + +$sub->stop; +$pub->stop; +done_testing(); diff --git a/src/test/modules/test_logicalmsg_hooks/test_logicalmsg_hooks.c b/src/test/modules/test_logicalmsg_hooks/test_logicalmsg_hooks.c new file mode 100644 index 00000000000..1e3120f4ee1 --- /dev/null +++ b/src/test/modules/test_logicalmsg_hooks/test_logicalmsg_hooks.c @@ -0,0 +1,104 @@ +/*-------------------------------------------------------------------------- + * + * test_logicalmsg_hooks.c + * Code for testing LogicalRepMessageHandle_hook + * + * The handler records every logical decoding message it receives into the + * table public.test_logicalmsg_log, so that tests can assert on the contents + * of a table rather than on the server log. Recording the messages this way + * also exercises the transactional behavior of the hook: a transactional + * message is recorded as part of the remote transaction that emitted it, and + * therefore disappears together with it if that transaction is rolled back or + * skipped, while a non-transactional message is recorded independently. + * + * The messages are also logged, which lets a test tell "the handler ran but + * its work was rolled back" apart from "the handler was never called". + * + * Portions Copyright (c) 1996-2026, PostgreSQL Global Development Group + * Portions Copyright (c) 1994, Regents of the University of California + * + * IDENTIFICATION + * src/test/modules/test_logicalmsg_hooks/test_logicalmsg_hooks.c + * + * ------------------------------------------------------------------------- + */ + +#include "postgres.h" + +#include "catalog/namespace.h" +#include "catalog/pg_type.h" +#include "executor/spi.h" +#include "nodes/makefuncs.h" +#include "replication/worker_internal.h" +#include "utils/builtins.h" +#include "utils/pg_lsn.h" + +PG_MODULE_MAGIC; + +#define LOG_SCHEMA "public" +#define LOG_TABLE "test_logicalmsg_log" + +static void test_logical_message_handler(LogicalRepMessageData *msg); + +/* + * Module load callback + */ +void +_PG_init(void) +{ + LogicalRepMessageHandle_hook = &test_logical_message_handler; +} + +static void +test_logical_message_handler(LogicalRepMessageData *msg) +{ + RangeVar *rv; + Oid argtypes[5] = {LSNOID, BOOLOID, TEXTOID, INT4OID, TEXTOID}; + Datum values[5]; + int ret; + + ereport(LOG, + (errmsg("received message: LSN %X/%08X, prefix: %s, message: %s, transactional: %d", + LSN_FORMAT_ARGS(msg->lsn), + msg->prefix, msg->message, msg->transactional))); + + /* + * The log table is created by the test, not by this module, so it may not + * exist yet. Skip recording rather than throwing an error, which would + * put the apply worker into a restart loop and make the test hang instead + * of fail. No lock is needed for this probe; the INSERT below takes the + * proper locks. + */ + rv = makeRangeVar(LOG_SCHEMA, LOG_TABLE, -1); + if (!OidIsValid(RangeVarGetRelid(rv, NoLock, true))) + return; + + /* + * Pass the message as query parameters rather than interpolating it into + * the query text. Message contents come from the publisher and are not + * to be trusted, and the payload may contain characters that would need + * quoting. Note also that message_size, not strlen(), is authoritative + * for the payload length. + */ + values[0] = LSNGetDatum(msg->lsn); + values[1] = BoolGetDatum(msg->transactional); + values[2] = CStringGetTextDatum(msg->prefix); + values[3] = Int32GetDatum((int32) msg->message_size); + values[4] = PointerGetDatum(cstring_to_text_with_len(msg->message, + msg->message_size)); + + SPI_connect(); + + /* + * The apply worker runs with an empty search_path, so the table name must + * be schema-qualified. + */ + ret = SPI_execute_with_args("INSERT INTO " LOG_SCHEMA "." LOG_TABLE + " (lsn, transactional, prefix, message_size, message)" + " VALUES ($1, $2, $3, $4, $5)", + 5, argtypes, values, NULL, false, 0); + if (ret != SPI_OK_INSERT) + elog(ERROR, "could not record logical message: SPI returned %d", ret); + + SPI_finish(); +} diff --git a/src/test/regress/expected/subscription.out b/src/test/regress/expected/subscription.out index 259db747334..83642fc03ee 100644 --- a/src/test/regress/expected/subscription.out +++ b/src/test/regress/expected/subscription.out @@ -143,18 +143,18 @@ CREATE SUBSCRIPTION regress_testsub4 CONNECTION 'dbname=regress_doesnotexist' PU WARNING: subscription was created, but is not connected HINT: To initiate replication, you must manually create the replication slot, enable the subscription, and alter the subscription to refresh publications. \dRs+ regress_testsub4 - List of subscriptions - Name | Owner | Enabled | Publication | Binary | Streaming | Two-phase commit | Disable on error | Origin | Password required | Run as owner? | Failover | Server | Retain dead tuples | Max retention duration | Retention active | Synchronous commit | Conninfo | Receiver timeout | Skip LSN | Description -------------------+---------------------------+---------+-------------+--------+-----------+------------------+------------------+--------+-------------------+---------------+----------+--------+--------------------+------------------------+------------------+--------------------+-----------------------------+------------------+------------+------------- - regress_testsub4 | regress_subscription_user | f | {testpub} | f | parallel | d | f | none | t | f | f | | f | 0 | f | off | dbname=regress_doesnotexist | -1 | 0/00000000 | + List of subscriptions + Name | Owner | Enabled | Publication | Binary | Streaming | Two-phase commit | Disable on error | Origin | Password required | Run as owner? | Failover | Server | Retain dead tuples | Max retention duration | Retention active | Message | Synchronous commit | Conninfo | Receiver timeout | Skip LSN | Description +------------------+---------------------------+---------+-------------+--------+-----------+------------------+------------------+--------+-------------------+---------------+----------+--------+--------------------+------------------------+------------------+---------+--------------------+-----------------------------+------------------+------------+------------- + regress_testsub4 | regress_subscription_user | f | {testpub} | f | parallel | d | f | none | t | f | f | | f | 0 | f | f | off | dbname=regress_doesnotexist | -1 | 0/00000000 | (1 row) ALTER SUBSCRIPTION regress_testsub4 SET (origin = any); \dRs+ regress_testsub4 - List of subscriptions - Name | Owner | Enabled | Publication | Binary | Streaming | Two-phase commit | Disable on error | Origin | Password required | Run as owner? | Failover | Server | Retain dead tuples | Max retention duration | Retention active | Synchronous commit | Conninfo | Receiver timeout | Skip LSN | Description -------------------+---------------------------+---------+-------------+--------+-----------+------------------+------------------+--------+-------------------+---------------+----------+--------+--------------------+------------------------+------------------+--------------------+-----------------------------+------------------+------------+------------- - regress_testsub4 | regress_subscription_user | f | {testpub} | f | parallel | d | f | any | t | f | f | | f | 0 | f | off | dbname=regress_doesnotexist | -1 | 0/00000000 | + List of subscriptions + Name | Owner | Enabled | Publication | Binary | Streaming | Two-phase commit | Disable on error | Origin | Password required | Run as owner? | Failover | Server | Retain dead tuples | Max retention duration | Retention active | Message | Synchronous commit | Conninfo | Receiver timeout | Skip LSN | Description +------------------+---------------------------+---------+-------------+--------+-----------+------------------+------------------+--------+-------------------+---------------+----------+--------+--------------------+------------------------+------------------+---------+--------------------+-----------------------------+------------------+------------+------------- + regress_testsub4 | regress_subscription_user | f | {testpub} | f | parallel | d | f | any | t | f | f | | f | 0 | f | f | off | dbname=regress_doesnotexist | -1 | 0/00000000 | (1 row) DROP SUBSCRIPTION regress_testsub3; @@ -258,10 +258,10 @@ ALTER SUBSCRIPTION regress_testsub CONNECTION 'foobar'; ERROR: invalid connection string syntax: missing "=" after "foobar" in connection info string \dRs+ - List of subscriptions - Name | Owner | Enabled | Publication | Binary | Streaming | Two-phase commit | Disable on error | Origin | Password required | Run as owner? | Failover | Server | Retain dead tuples | Max retention duration | Retention active | Synchronous commit | Conninfo | Receiver timeout | Skip LSN | Description ------------------+---------------------------+---------+-------------+--------+-----------+------------------+------------------+--------+-------------------+---------------+----------+--------+--------------------+------------------------+------------------+--------------------+-----------------------------+------------------+------------+------------------- - regress_testsub | regress_subscription_user | f | {testpub} | f | parallel | d | f | any | t | f | f | | f | 0 | f | off | dbname=regress_doesnotexist | -1 | 0/00000000 | test subscription + List of subscriptions + Name | Owner | Enabled | Publication | Binary | Streaming | Two-phase commit | Disable on error | Origin | Password required | Run as owner? | Failover | Server | Retain dead tuples | Max retention duration | Retention active | Message | Synchronous commit | Conninfo | Receiver timeout | Skip LSN | Description +-----------------+---------------------------+---------+-------------+--------+-----------+------------------+------------------+--------+-------------------+---------------+----------+--------+--------------------+------------------------+------------------+---------+--------------------+-----------------------------+------------------+------------+------------------- + regress_testsub | regress_subscription_user | f | {testpub} | f | parallel | d | f | any | t | f | f | | f | 0 | f | f | off | dbname=regress_doesnotexist | -1 | 0/00000000 | test subscription (1 row) ALTER SUBSCRIPTION regress_testsub SET PUBLICATION testpub2, testpub3 WITH (refresh = false); @@ -270,10 +270,10 @@ ALTER SUBSCRIPTION regress_testsub SET (slot_name = 'newname'); ALTER SUBSCRIPTION regress_testsub SET (password_required = false); ALTER SUBSCRIPTION regress_testsub SET (run_as_owner = true); \dRs+ - List of subscriptions - Name | Owner | Enabled | Publication | Binary | Streaming | Two-phase commit | Disable on error | Origin | Password required | Run as owner? | Failover | Server | Retain dead tuples | Max retention duration | Retention active | Synchronous commit | Conninfo | Receiver timeout | Skip LSN | Description ------------------+---------------------------+---------+---------------------+--------+-----------+------------------+------------------+--------+-------------------+---------------+----------+--------+--------------------+------------------------+------------------+--------------------+------------------------------+------------------+------------+------------------- - regress_testsub | regress_subscription_user | f | {testpub2,testpub3} | f | parallel | d | f | any | f | t | f | | f | 0 | f | off | dbname=regress_doesnotexist2 | -1 | 0/00000000 | test subscription + List of subscriptions + Name | Owner | Enabled | Publication | Binary | Streaming | Two-phase commit | Disable on error | Origin | Password required | Run as owner? | Failover | Server | Retain dead tuples | Max retention duration | Retention active | Message | Synchronous commit | Conninfo | Receiver timeout | Skip LSN | Description +-----------------+---------------------------+---------+---------------------+--------+-----------+------------------+------------------+--------+-------------------+---------------+----------+--------+--------------------+------------------------+------------------+---------+--------------------+------------------------------+------------------+------------+------------------- + regress_testsub | regress_subscription_user | f | {testpub2,testpub3} | f | parallel | d | f | any | f | t | f | | f | 0 | f | f | off | dbname=regress_doesnotexist2 | -1 | 0/00000000 | test subscription (1 row) ALTER SUBSCRIPTION regress_testsub SET (password_required = true); @@ -289,10 +289,10 @@ ERROR: unrecognized subscription parameter: "create_slot" -- ok ALTER SUBSCRIPTION regress_testsub SKIP (lsn = '0/12345'); \dRs+ - List of subscriptions - Name | Owner | Enabled | Publication | Binary | Streaming | Two-phase commit | Disable on error | Origin | Password required | Run as owner? | Failover | Server | Retain dead tuples | Max retention duration | Retention active | Synchronous commit | Conninfo | Receiver timeout | Skip LSN | Description ------------------+---------------------------+---------+---------------------+--------+-----------+------------------+------------------+--------+-------------------+---------------+----------+--------+--------------------+------------------------+------------------+--------------------+------------------------------+------------------+------------+------------------- - regress_testsub | regress_subscription_user | f | {testpub2,testpub3} | f | parallel | d | f | any | t | f | f | | f | 0 | f | off | dbname=regress_doesnotexist2 | -1 | 0/00012345 | test subscription + List of subscriptions + Name | Owner | Enabled | Publication | Binary | Streaming | Two-phase commit | Disable on error | Origin | Password required | Run as owner? | Failover | Server | Retain dead tuples | Max retention duration | Retention active | Message | Synchronous commit | Conninfo | Receiver timeout | Skip LSN | Description +-----------------+---------------------------+---------+---------------------+--------+-----------+------------------+------------------+--------+-------------------+---------------+----------+--------+--------------------+------------------------+------------------+---------+--------------------+------------------------------+------------------+------------+------------------- + regress_testsub | regress_subscription_user | f | {testpub2,testpub3} | f | parallel | d | f | any | t | f | f | | f | 0 | f | f | off | dbname=regress_doesnotexist2 | -1 | 0/00012345 | test subscription (1 row) -- ok - with lsn = NONE @@ -301,10 +301,10 @@ ALTER SUBSCRIPTION regress_testsub SKIP (lsn = NONE); ALTER SUBSCRIPTION regress_testsub SKIP (lsn = '0/0'); ERROR: invalid WAL location (LSN): 0/0 \dRs+ - List of subscriptions - Name | Owner | Enabled | Publication | Binary | Streaming | Two-phase commit | Disable on error | Origin | Password required | Run as owner? | Failover | Server | Retain dead tuples | Max retention duration | Retention active | Synchronous commit | Conninfo | Receiver timeout | Skip LSN | Description ------------------+---------------------------+---------+---------------------+--------+-----------+------------------+------------------+--------+-------------------+---------------+----------+--------+--------------------+------------------------+------------------+--------------------+------------------------------+------------------+------------+------------------- - regress_testsub | regress_subscription_user | f | {testpub2,testpub3} | f | parallel | d | f | any | t | f | f | | f | 0 | f | off | dbname=regress_doesnotexist2 | -1 | 0/00000000 | test subscription + List of subscriptions + Name | Owner | Enabled | Publication | Binary | Streaming | Two-phase commit | Disable on error | Origin | Password required | Run as owner? | Failover | Server | Retain dead tuples | Max retention duration | Retention active | Message | Synchronous commit | Conninfo | Receiver timeout | Skip LSN | Description +-----------------+---------------------------+---------+---------------------+--------+-----------+------------------+------------------+--------+-------------------+---------------+----------+--------+--------------------+------------------------+------------------+---------+--------------------+------------------------------+------------------+------------+------------------- + regress_testsub | regress_subscription_user | f | {testpub2,testpub3} | f | parallel | d | f | any | t | f | f | | f | 0 | f | f | off | dbname=regress_doesnotexist2 | -1 | 0/00000000 | test subscription (1 row) BEGIN; @@ -340,10 +340,10 @@ ALTER SUBSCRIPTION regress_testsub_foo SET (wal_receiver_timeout = '80s'); ALTER SUBSCRIPTION regress_testsub_foo SET (wal_receiver_timeout = 'foobar'); ERROR: invalid value for parameter "wal_receiver_timeout": "foobar" \dRs+ - List of subscriptions - Name | Owner | Enabled | Publication | Binary | Streaming | Two-phase commit | Disable on error | Origin | Password required | Run as owner? | Failover | Server | Retain dead tuples | Max retention duration | Retention active | Synchronous commit | Conninfo | Receiver timeout | Skip LSN | Description ----------------------+---------------------------+---------+---------------------+--------+-----------+------------------+------------------+--------+-------------------+---------------+----------+--------+--------------------+------------------------+------------------+--------------------+------------------------------+------------------+------------+------------------- - regress_testsub_foo | regress_subscription_user | f | {testpub2,testpub3} | f | parallel | d | f | any | t | f | f | | f | 0 | f | local | dbname=regress_doesnotexist2 | 80s | 0/00000000 | test subscription + List of subscriptions + Name | Owner | Enabled | Publication | Binary | Streaming | Two-phase commit | Disable on error | Origin | Password required | Run as owner? | Failover | Server | Retain dead tuples | Max retention duration | Retention active | Message | Synchronous commit | Conninfo | Receiver timeout | Skip LSN | Description +---------------------+---------------------------+---------+---------------------+--------+-----------+------------------+------------------+--------+-------------------+---------------+----------+--------+--------------------+------------------------+------------------+---------+--------------------+------------------------------+------------------+------------+------------------- + regress_testsub_foo | regress_subscription_user | f | {testpub2,testpub3} | f | parallel | d | f | any | t | f | f | | f | 0 | f | f | local | dbname=regress_doesnotexist2 | 80s | 0/00000000 | test subscription (1 row) -- rename back to keep the rest simple @@ -372,19 +372,19 @@ CREATE SUBSCRIPTION regress_testsub CONNECTION 'dbname=regress_doesnotexist' PUB WARNING: subscription was created, but is not connected HINT: To initiate replication, you must manually create the replication slot, enable the subscription, and alter the subscription to refresh publications. \dRs+ - List of subscriptions - Name | Owner | Enabled | Publication | Binary | Streaming | Two-phase commit | Disable on error | Origin | Password required | Run as owner? | Failover | Server | Retain dead tuples | Max retention duration | Retention active | Synchronous commit | Conninfo | Receiver timeout | Skip LSN | Description ------------------+---------------------------+---------+-------------+--------+-----------+------------------+------------------+--------+-------------------+---------------+----------+--------+--------------------+------------------------+------------------+--------------------+-----------------------------+------------------+------------+------------- - regress_testsub | regress_subscription_user | f | {testpub} | t | parallel | d | f | any | t | f | f | | f | 0 | f | off | dbname=regress_doesnotexist | -1 | 0/00000000 | + List of subscriptions + Name | Owner | Enabled | Publication | Binary | Streaming | Two-phase commit | Disable on error | Origin | Password required | Run as owner? | Failover | Server | Retain dead tuples | Max retention duration | Retention active | Message | Synchronous commit | Conninfo | Receiver timeout | Skip LSN | Description +-----------------+---------------------------+---------+-------------+--------+-----------+------------------+------------------+--------+-------------------+---------------+----------+--------+--------------------+------------------------+------------------+---------+--------------------+-----------------------------+------------------+------------+------------- + regress_testsub | regress_subscription_user | f | {testpub} | t | parallel | d | f | any | t | f | f | | f | 0 | f | f | off | dbname=regress_doesnotexist | -1 | 0/00000000 | (1 row) ALTER SUBSCRIPTION regress_testsub SET (binary = false); ALTER SUBSCRIPTION regress_testsub SET (slot_name = NONE); \dRs+ - List of subscriptions - Name | Owner | Enabled | Publication | Binary | Streaming | Two-phase commit | Disable on error | Origin | Password required | Run as owner? | Failover | Server | Retain dead tuples | Max retention duration | Retention active | Synchronous commit | Conninfo | Receiver timeout | Skip LSN | Description ------------------+---------------------------+---------+-------------+--------+-----------+------------------+------------------+--------+-------------------+---------------+----------+--------+--------------------+------------------------+------------------+--------------------+-----------------------------+------------------+------------+------------- - regress_testsub | regress_subscription_user | f | {testpub} | f | parallel | d | f | any | t | f | f | | f | 0 | f | off | dbname=regress_doesnotexist | -1 | 0/00000000 | + List of subscriptions + Name | Owner | Enabled | Publication | Binary | Streaming | Two-phase commit | Disable on error | Origin | Password required | Run as owner? | Failover | Server | Retain dead tuples | Max retention duration | Retention active | Message | Synchronous commit | Conninfo | Receiver timeout | Skip LSN | Description +-----------------+---------------------------+---------+-------------+--------+-----------+------------------+------------------+--------+-------------------+---------------+----------+--------+--------------------+------------------------+------------------+---------+--------------------+-----------------------------+------------------+------------+------------- + regress_testsub | regress_subscription_user | f | {testpub} | f | parallel | d | f | any | t | f | f | | f | 0 | f | f | off | dbname=regress_doesnotexist | -1 | 0/00000000 | (1 row) DROP SUBSCRIPTION regress_testsub; @@ -396,27 +396,27 @@ CREATE SUBSCRIPTION regress_testsub CONNECTION 'dbname=regress_doesnotexist' PUB WARNING: subscription was created, but is not connected HINT: To initiate replication, you must manually create the replication slot, enable the subscription, and alter the subscription to refresh publications. \dRs+ - List of subscriptions - Name | Owner | Enabled | Publication | Binary | Streaming | Two-phase commit | Disable on error | Origin | Password required | Run as owner? | Failover | Server | Retain dead tuples | Max retention duration | Retention active | Synchronous commit | Conninfo | Receiver timeout | Skip LSN | Description ------------------+---------------------------+---------+-------------+--------+-----------+------------------+------------------+--------+-------------------+---------------+----------+--------+--------------------+------------------------+------------------+--------------------+-----------------------------+------------------+------------+------------- - regress_testsub | regress_subscription_user | f | {testpub} | f | on | d | f | any | t | f | f | | f | 0 | f | off | dbname=regress_doesnotexist | -1 | 0/00000000 | + List of subscriptions + Name | Owner | Enabled | Publication | Binary | Streaming | Two-phase commit | Disable on error | Origin | Password required | Run as owner? | Failover | Server | Retain dead tuples | Max retention duration | Retention active | Message | Synchronous commit | Conninfo | Receiver timeout | Skip LSN | Description +-----------------+---------------------------+---------+-------------+--------+-----------+------------------+------------------+--------+-------------------+---------------+----------+--------+--------------------+------------------------+------------------+---------+--------------------+-----------------------------+------------------+------------+------------- + regress_testsub | regress_subscription_user | f | {testpub} | f | on | d | f | any | t | f | f | | f | 0 | f | f | off | dbname=regress_doesnotexist | -1 | 0/00000000 | (1 row) ALTER SUBSCRIPTION regress_testsub SET (streaming = parallel); \dRs+ - List of subscriptions - Name | Owner | Enabled | Publication | Binary | Streaming | Two-phase commit | Disable on error | Origin | Password required | Run as owner? | Failover | Server | Retain dead tuples | Max retention duration | Retention active | Synchronous commit | Conninfo | Receiver timeout | Skip LSN | Description ------------------+---------------------------+---------+-------------+--------+-----------+------------------+------------------+--------+-------------------+---------------+----------+--------+--------------------+------------------------+------------------+--------------------+-----------------------------+------------------+------------+------------- - regress_testsub | regress_subscription_user | f | {testpub} | f | parallel | d | f | any | t | f | f | | f | 0 | f | off | dbname=regress_doesnotexist | -1 | 0/00000000 | + List of subscriptions + Name | Owner | Enabled | Publication | Binary | Streaming | Two-phase commit | Disable on error | Origin | Password required | Run as owner? | Failover | Server | Retain dead tuples | Max retention duration | Retention active | Message | Synchronous commit | Conninfo | Receiver timeout | Skip LSN | Description +-----------------+---------------------------+---------+-------------+--------+-----------+------------------+------------------+--------+-------------------+---------------+----------+--------+--------------------+------------------------+------------------+---------+--------------------+-----------------------------+------------------+------------+------------- + regress_testsub | regress_subscription_user | f | {testpub} | f | parallel | d | f | any | t | f | f | | f | 0 | f | f | off | dbname=regress_doesnotexist | -1 | 0/00000000 | (1 row) ALTER SUBSCRIPTION regress_testsub SET (streaming = false); ALTER SUBSCRIPTION regress_testsub SET (slot_name = NONE); \dRs+ - List of subscriptions - Name | Owner | Enabled | Publication | Binary | Streaming | Two-phase commit | Disable on error | Origin | Password required | Run as owner? | Failover | Server | Retain dead tuples | Max retention duration | Retention active | Synchronous commit | Conninfo | Receiver timeout | Skip LSN | Description ------------------+---------------------------+---------+-------------+--------+-----------+------------------+------------------+--------+-------------------+---------------+----------+--------+--------------------+------------------------+------------------+--------------------+-----------------------------+------------------+------------+------------- - regress_testsub | regress_subscription_user | f | {testpub} | f | off | d | f | any | t | f | f | | f | 0 | f | off | dbname=regress_doesnotexist | -1 | 0/00000000 | + List of subscriptions + Name | Owner | Enabled | Publication | Binary | Streaming | Two-phase commit | Disable on error | Origin | Password required | Run as owner? | Failover | Server | Retain dead tuples | Max retention duration | Retention active | Message | Synchronous commit | Conninfo | Receiver timeout | Skip LSN | Description +-----------------+---------------------------+---------+-------------+--------+-----------+------------------+------------------+--------+-------------------+---------------+----------+--------+--------------------+------------------------+------------------+---------+--------------------+-----------------------------+------------------+------------+------------- + regress_testsub | regress_subscription_user | f | {testpub} | f | off | d | f | any | t | f | f | | f | 0 | f | f | off | dbname=regress_doesnotexist | -1 | 0/00000000 | (1 row) -- fail - publication already exists @@ -431,10 +431,10 @@ ALTER SUBSCRIPTION regress_testsub ADD PUBLICATION testpub1, testpub2 WITH (refr ALTER SUBSCRIPTION regress_testsub ADD PUBLICATION testpub1, testpub2 WITH (refresh = false); ERROR: publication "testpub1" is already in subscription "regress_testsub" \dRs+ - List of subscriptions - Name | Owner | Enabled | Publication | Binary | Streaming | Two-phase commit | Disable on error | Origin | Password required | Run as owner? | Failover | Server | Retain dead tuples | Max retention duration | Retention active | Synchronous commit | Conninfo | Receiver timeout | Skip LSN | Description ------------------+---------------------------+---------+-----------------------------+--------+-----------+------------------+------------------+--------+-------------------+---------------+----------+--------+--------------------+------------------------+------------------+--------------------+-----------------------------+------------------+------------+------------- - regress_testsub | regress_subscription_user | f | {testpub,testpub1,testpub2} | f | off | d | f | any | t | f | f | | f | 0 | f | off | dbname=regress_doesnotexist | -1 | 0/00000000 | + List of subscriptions + Name | Owner | Enabled | Publication | Binary | Streaming | Two-phase commit | Disable on error | Origin | Password required | Run as owner? | Failover | Server | Retain dead tuples | Max retention duration | Retention active | Message | Synchronous commit | Conninfo | Receiver timeout | Skip LSN | Description +-----------------+---------------------------+---------+-----------------------------+--------+-----------+------------------+------------------+--------+-------------------+---------------+----------+--------+--------------------+------------------------+------------------+---------+--------------------+-----------------------------+------------------+------------+------------- + regress_testsub | regress_subscription_user | f | {testpub,testpub1,testpub2} | f | off | d | f | any | t | f | f | | f | 0 | f | f | off | dbname=regress_doesnotexist | -1 | 0/00000000 | (1 row) -- fail - publication used more than once @@ -449,10 +449,10 @@ ERROR: publication "testpub3" is not in subscription "regress_testsub" -- ok - delete publications ALTER SUBSCRIPTION regress_testsub DROP PUBLICATION testpub1, testpub2 WITH (refresh = false); \dRs+ - List of subscriptions - Name | Owner | Enabled | Publication | Binary | Streaming | Two-phase commit | Disable on error | Origin | Password required | Run as owner? | Failover | Server | Retain dead tuples | Max retention duration | Retention active | Synchronous commit | Conninfo | Receiver timeout | Skip LSN | Description ------------------+---------------------------+---------+-------------+--------+-----------+------------------+------------------+--------+-------------------+---------------+----------+--------+--------------------+------------------------+------------------+--------------------+-----------------------------+------------------+------------+------------- - regress_testsub | regress_subscription_user | f | {testpub} | f | off | d | f | any | t | f | f | | f | 0 | f | off | dbname=regress_doesnotexist | -1 | 0/00000000 | + List of subscriptions + Name | Owner | Enabled | Publication | Binary | Streaming | Two-phase commit | Disable on error | Origin | Password required | Run as owner? | Failover | Server | Retain dead tuples | Max retention duration | Retention active | Message | Synchronous commit | Conninfo | Receiver timeout | Skip LSN | Description +-----------------+---------------------------+---------+-------------+--------+-----------+------------------+------------------+--------+-------------------+---------------+----------+--------+--------------------+------------------------+------------------+---------+--------------------+-----------------------------+------------------+------------+------------- + regress_testsub | regress_subscription_user | f | {testpub} | f | off | d | f | any | t | f | f | | f | 0 | f | f | off | dbname=regress_doesnotexist | -1 | 0/00000000 | (1 row) DROP SUBSCRIPTION regress_testsub; @@ -488,19 +488,19 @@ CREATE SUBSCRIPTION regress_testsub CONNECTION 'dbname=regress_doesnotexist' PUB WARNING: subscription was created, but is not connected HINT: To initiate replication, you must manually create the replication slot, enable the subscription, and alter the subscription to refresh publications. \dRs+ - List of subscriptions - Name | Owner | Enabled | Publication | Binary | Streaming | Two-phase commit | Disable on error | Origin | Password required | Run as owner? | Failover | Server | Retain dead tuples | Max retention duration | Retention active | Synchronous commit | Conninfo | Receiver timeout | Skip LSN | Description ------------------+---------------------------+---------+-------------+--------+-----------+------------------+------------------+--------+-------------------+---------------+----------+--------+--------------------+------------------------+------------------+--------------------+-----------------------------+------------------+------------+------------- - regress_testsub | regress_subscription_user | f | {testpub} | f | parallel | p | f | any | t | f | f | | f | 0 | f | off | dbname=regress_doesnotexist | -1 | 0/00000000 | + List of subscriptions + Name | Owner | Enabled | Publication | Binary | Streaming | Two-phase commit | Disable on error | Origin | Password required | Run as owner? | Failover | Server | Retain dead tuples | Max retention duration | Retention active | Message | Synchronous commit | Conninfo | Receiver timeout | Skip LSN | Description +-----------------+---------------------------+---------+-------------+--------+-----------+------------------+------------------+--------+-------------------+---------------+----------+--------+--------------------+------------------------+------------------+---------+--------------------+-----------------------------+------------------+------------+------------- + regress_testsub | regress_subscription_user | f | {testpub} | f | parallel | p | f | any | t | f | f | | f | 0 | f | f | off | dbname=regress_doesnotexist | -1 | 0/00000000 | (1 row) -- we can alter streaming when two_phase enabled ALTER SUBSCRIPTION regress_testsub SET (streaming = true); \dRs+ - List of subscriptions - Name | Owner | Enabled | Publication | Binary | Streaming | Two-phase commit | Disable on error | Origin | Password required | Run as owner? | Failover | Server | Retain dead tuples | Max retention duration | Retention active | Synchronous commit | Conninfo | Receiver timeout | Skip LSN | Description ------------------+---------------------------+---------+-------------+--------+-----------+------------------+------------------+--------+-------------------+---------------+----------+--------+--------------------+------------------------+------------------+--------------------+-----------------------------+------------------+------------+------------- - regress_testsub | regress_subscription_user | f | {testpub} | f | on | p | f | any | t | f | f | | f | 0 | f | off | dbname=regress_doesnotexist | -1 | 0/00000000 | + List of subscriptions + Name | Owner | Enabled | Publication | Binary | Streaming | Two-phase commit | Disable on error | Origin | Password required | Run as owner? | Failover | Server | Retain dead tuples | Max retention duration | Retention active | Message | Synchronous commit | Conninfo | Receiver timeout | Skip LSN | Description +-----------------+---------------------------+---------+-------------+--------+-----------+------------------+------------------+--------+-------------------+---------------+----------+--------+--------------------+------------------------+------------------+---------+--------------------+-----------------------------+------------------+------------+------------- + regress_testsub | regress_subscription_user | f | {testpub} | f | on | p | f | any | t | f | f | | f | 0 | f | f | off | dbname=regress_doesnotexist | -1 | 0/00000000 | (1 row) ALTER SUBSCRIPTION regress_testsub SET (slot_name = NONE); @@ -510,10 +510,10 @@ CREATE SUBSCRIPTION regress_testsub CONNECTION 'dbname=regress_doesnotexist' PUB WARNING: subscription was created, but is not connected HINT: To initiate replication, you must manually create the replication slot, enable the subscription, and alter the subscription to refresh publications. \dRs+ - List of subscriptions - Name | Owner | Enabled | Publication | Binary | Streaming | Two-phase commit | Disable on error | Origin | Password required | Run as owner? | Failover | Server | Retain dead tuples | Max retention duration | Retention active | Synchronous commit | Conninfo | Receiver timeout | Skip LSN | Description ------------------+---------------------------+---------+-------------+--------+-----------+------------------+------------------+--------+-------------------+---------------+----------+--------+--------------------+------------------------+------------------+--------------------+-----------------------------+------------------+------------+------------- - regress_testsub | regress_subscription_user | f | {testpub} | f | on | p | f | any | t | f | f | | f | 0 | f | off | dbname=regress_doesnotexist | -1 | 0/00000000 | + List of subscriptions + Name | Owner | Enabled | Publication | Binary | Streaming | Two-phase commit | Disable on error | Origin | Password required | Run as owner? | Failover | Server | Retain dead tuples | Max retention duration | Retention active | Message | Synchronous commit | Conninfo | Receiver timeout | Skip LSN | Description +-----------------+---------------------------+---------+-------------+--------+-----------+------------------+------------------+--------+-------------------+---------------+----------+--------+--------------------+------------------------+------------------+---------+--------------------+-----------------------------+------------------+------------+------------- + regress_testsub | regress_subscription_user | f | {testpub} | f | on | p | f | any | t | f | f | | f | 0 | f | f | off | dbname=regress_doesnotexist | -1 | 0/00000000 | (1 row) ALTER SUBSCRIPTION regress_testsub SET (slot_name = NONE); @@ -526,18 +526,18 @@ CREATE SUBSCRIPTION regress_testsub CONNECTION 'dbname=regress_doesnotexist' PUB WARNING: subscription was created, but is not connected HINT: To initiate replication, you must manually create the replication slot, enable the subscription, and alter the subscription to refresh publications. \dRs+ - List of subscriptions - Name | Owner | Enabled | Publication | Binary | Streaming | Two-phase commit | Disable on error | Origin | Password required | Run as owner? | Failover | Server | Retain dead tuples | Max retention duration | Retention active | Synchronous commit | Conninfo | Receiver timeout | Skip LSN | Description ------------------+---------------------------+---------+-------------+--------+-----------+------------------+------------------+--------+-------------------+---------------+----------+--------+--------------------+------------------------+------------------+--------------------+-----------------------------+------------------+------------+------------- - regress_testsub | regress_subscription_user | f | {testpub} | f | parallel | d | f | any | t | f | f | | f | 0 | f | off | dbname=regress_doesnotexist | -1 | 0/00000000 | + List of subscriptions + Name | Owner | Enabled | Publication | Binary | Streaming | Two-phase commit | Disable on error | Origin | Password required | Run as owner? | Failover | Server | Retain dead tuples | Max retention duration | Retention active | Message | Synchronous commit | Conninfo | Receiver timeout | Skip LSN | Description +-----------------+---------------------------+---------+-------------+--------+-----------+------------------+------------------+--------+-------------------+---------------+----------+--------+--------------------+------------------------+------------------+---------+--------------------+-----------------------------+------------------+------------+------------- + regress_testsub | regress_subscription_user | f | {testpub} | f | parallel | d | f | any | t | f | f | | f | 0 | f | f | off | dbname=regress_doesnotexist | -1 | 0/00000000 | (1 row) ALTER SUBSCRIPTION regress_testsub SET (disable_on_error = true); \dRs+ - List of subscriptions - Name | Owner | Enabled | Publication | Binary | Streaming | Two-phase commit | Disable on error | Origin | Password required | Run as owner? | Failover | Server | Retain dead tuples | Max retention duration | Retention active | Synchronous commit | Conninfo | Receiver timeout | Skip LSN | Description ------------------+---------------------------+---------+-------------+--------+-----------+------------------+------------------+--------+-------------------+---------------+----------+--------+--------------------+------------------------+------------------+--------------------+-----------------------------+------------------+------------+------------- - regress_testsub | regress_subscription_user | f | {testpub} | f | parallel | d | t | any | t | f | f | | f | 0 | f | off | dbname=regress_doesnotexist | -1 | 0/00000000 | + List of subscriptions + Name | Owner | Enabled | Publication | Binary | Streaming | Two-phase commit | Disable on error | Origin | Password required | Run as owner? | Failover | Server | Retain dead tuples | Max retention duration | Retention active | Message | Synchronous commit | Conninfo | Receiver timeout | Skip LSN | Description +-----------------+---------------------------+---------+-------------+--------+-----------+------------------+------------------+--------+-------------------+---------------+----------+--------+--------------------+------------------------+------------------+---------+--------------------+-----------------------------+------------------+------------+------------- + regress_testsub | regress_subscription_user | f | {testpub} | f | parallel | d | t | any | t | f | f | | f | 0 | f | f | off | dbname=regress_doesnotexist | -1 | 0/00000000 | (1 row) ALTER SUBSCRIPTION regress_testsub SET (slot_name = NONE); @@ -550,10 +550,10 @@ CREATE SUBSCRIPTION regress_testsub CONNECTION 'dbname=regress_doesnotexist' PUB WARNING: subscription was created, but is not connected HINT: To initiate replication, you must manually create the replication slot, enable the subscription, and alter the subscription to refresh publications. \dRs+ - List of subscriptions - Name | Owner | Enabled | Publication | Binary | Streaming | Two-phase commit | Disable on error | Origin | Password required | Run as owner? | Failover | Server | Retain dead tuples | Max retention duration | Retention active | Synchronous commit | Conninfo | Receiver timeout | Skip LSN | Description ------------------+---------------------------+---------+-------------+--------+-----------+------------------+------------------+--------+-------------------+---------------+----------+--------+--------------------+------------------------+------------------+--------------------+-----------------------------+------------------+------------+------------- - regress_testsub | regress_subscription_user | f | {testpub} | f | parallel | d | f | any | t | f | f | | f | 0 | f | off | dbname=regress_doesnotexist | -1 | 0/00000000 | + List of subscriptions + Name | Owner | Enabled | Publication | Binary | Streaming | Two-phase commit | Disable on error | Origin | Password required | Run as owner? | Failover | Server | Retain dead tuples | Max retention duration | Retention active | Message | Synchronous commit | Conninfo | Receiver timeout | Skip LSN | Description +-----------------+---------------------------+---------+-------------+--------+-----------+------------------+------------------+--------+-------------------+---------------+----------+--------+--------------------+------------------------+------------------+---------+--------------------+-----------------------------+------------------+------------+------------- + regress_testsub | regress_subscription_user | f | {testpub} | f | parallel | d | f | any | t | f | f | | f | 0 | f | f | off | dbname=regress_doesnotexist | -1 | 0/00000000 | (1 row) ALTER SUBSCRIPTION regress_testsub SET (slot_name = NONE); @@ -570,10 +570,10 @@ NOTICE: max_retention_duration is ineffective when retain_dead_tuples is disabl WARNING: subscription was created, but is not connected HINT: To initiate replication, you must manually create the replication slot, enable the subscription, and alter the subscription to refresh publications. \dRs+ - List of subscriptions - Name | Owner | Enabled | Publication | Binary | Streaming | Two-phase commit | Disable on error | Origin | Password required | Run as owner? | Failover | Server | Retain dead tuples | Max retention duration | Retention active | Synchronous commit | Conninfo | Receiver timeout | Skip LSN | Description ------------------+---------------------------+---------+-------------+--------+-----------+------------------+------------------+--------+-------------------+---------------+----------+--------+--------------------+------------------------+------------------+--------------------+-----------------------------+------------------+------------+------------- - regress_testsub | regress_subscription_user | f | {testpub} | f | parallel | d | f | any | t | f | f | | f | 1000 | f | off | dbname=regress_doesnotexist | -1 | 0/00000000 | + List of subscriptions + Name | Owner | Enabled | Publication | Binary | Streaming | Two-phase commit | Disable on error | Origin | Password required | Run as owner? | Failover | Server | Retain dead tuples | Max retention duration | Retention active | Message | Synchronous commit | Conninfo | Receiver timeout | Skip LSN | Description +-----------------+---------------------------+---------+-------------+--------+-----------+------------------+------------------+--------+-------------------+---------------+----------+--------+--------------------+------------------------+------------------+---------+--------------------+-----------------------------+------------------+------------+------------- + regress_testsub | regress_subscription_user | f | {testpub} | f | parallel | d | f | any | t | f | f | | f | 1000 | f | f | off | dbname=regress_doesnotexist | -1 | 0/00000000 | (1 row) -- fail - max_retention_duration must be non-negative @@ -582,12 +582,38 @@ ERROR: max_retention_duration cannot be negative -- ok ALTER SUBSCRIPTION regress_testsub SET (max_retention_duration = 0); \dRs+ - List of subscriptions - Name | Owner | Enabled | Publication | Binary | Streaming | Two-phase commit | Disable on error | Origin | Password required | Run as owner? | Failover | Server | Retain dead tuples | Max retention duration | Retention active | Synchronous commit | Conninfo | Receiver timeout | Skip LSN | Description ------------------+---------------------------+---------+-------------+--------+-----------+------------------+------------------+--------+-------------------+---------------+----------+--------+--------------------+------------------------+------------------+--------------------+-----------------------------+------------------+------------+------------- - regress_testsub | regress_subscription_user | f | {testpub} | f | parallel | d | f | any | t | f | f | | f | 0 | f | off | dbname=regress_doesnotexist | -1 | 0/00000000 | + List of subscriptions + Name | Owner | Enabled | Publication | Binary | Streaming | Two-phase commit | Disable on error | Origin | Password required | Run as owner? | Failover | Server | Retain dead tuples | Max retention duration | Retention active | Message | Synchronous commit | Conninfo | Receiver timeout | Skip LSN | Description +-----------------+---------------------------+---------+-------------+--------+-----------+------------------+------------------+--------+-------------------+---------------+----------+--------+--------------------+------------------------+------------------+---------+--------------------+-----------------------------+------------------+------------+------------- + regress_testsub | regress_subscription_user | f | {testpub} | f | parallel | d | f | any | t | f | f | | f | 0 | f | f | off | dbname=regress_doesnotexist | -1 | 0/00000000 | +(1 row) + +ALTER SUBSCRIPTION regress_testsub SET (slot_name = NONE); +DROP SUBSCRIPTION regress_testsub; +-- fail - message must be boolean +CREATE SUBSCRIPTION regress_testsub CONNECTION 'dbname=regress_doesnotexist' PUBLICATION testpub WITH (connect = false, message = foo); +ERROR: message requires a Boolean value +-- ok +CREATE SUBSCRIPTION regress_testsub CONNECTION 'dbname=regress_doesnotexist' PUBLICATION testpub WITH (connect = false, message = true); +WARNING: subscription was created, but is not connected +HINT: To initiate replication, you must manually create the replication slot, enable the subscription, and alter the subscription to refresh publications. +\dRs+ + List of subscriptions + Name | Owner | Enabled | Publication | Binary | Streaming | Two-phase commit | Disable on error | Origin | Password required | Run as owner? | Failover | Server | Retain dead tuples | Max retention duration | Retention active | Message | Synchronous commit | Conninfo | Receiver timeout | Skip LSN | Description +-----------------+---------------------------+---------+-------------+--------+-----------+------------------+------------------+--------+-------------------+---------------+----------+--------+--------------------+------------------------+------------------+---------+--------------------+-----------------------------+------------------+------------+------------- + regress_testsub | regress_subscription_user | f | {testpub} | f | parallel | d | f | any | t | f | f | | f | 0 | f | t | off | dbname=regress_doesnotexist | -1 | 0/00000000 | +(1 row) + +-- ok +ALTER SUBSCRIPTION regress_testsub SET (message = false); +\dRs+ + List of subscriptions + Name | Owner | Enabled | Publication | Binary | Streaming | Two-phase commit | Disable on error | Origin | Password required | Run as owner? | Failover | Server | Retain dead tuples | Max retention duration | Retention active | Message | Synchronous commit | Conninfo | Receiver timeout | Skip LSN | Description +-----------------+---------------------------+---------+-------------+--------+-----------+------------------+------------------+--------+-------------------+---------------+----------+--------+--------------------+------------------------+------------------+---------+--------------------+-----------------------------+------------------+------------+------------- + regress_testsub | regress_subscription_user | f | {testpub} | f | parallel | d | f | any | t | f | f | | f | 0 | f | f | off | dbname=regress_doesnotexist | -1 | 0/00000000 | (1 row) +-- cleanup ALTER SUBSCRIPTION regress_testsub SET (slot_name = NONE); DROP SUBSCRIPTION regress_testsub; -- let's do some tests with pg_create_subscription rather than superuser diff --git a/src/test/regress/sql/subscription.sql b/src/test/regress/sql/subscription.sql index 7718c742974..55682f255d1 100644 --- a/src/test/regress/sql/subscription.sql +++ b/src/test/regress/sql/subscription.sql @@ -441,6 +441,23 @@ ALTER SUBSCRIPTION regress_testsub SET (max_retention_duration = 0); ALTER SUBSCRIPTION regress_testsub SET (slot_name = NONE); DROP SUBSCRIPTION regress_testsub; +-- fail - message must be boolean +CREATE SUBSCRIPTION regress_testsub CONNECTION 'dbname=regress_doesnotexist' PUBLICATION testpub WITH (connect = false, message = foo); + +-- ok +CREATE SUBSCRIPTION regress_testsub CONNECTION 'dbname=regress_doesnotexist' PUBLICATION testpub WITH (connect = false, message = true); + +\dRs+ + +-- ok +ALTER SUBSCRIPTION regress_testsub SET (message = false); + +\dRs+ + +-- cleanup +ALTER SUBSCRIPTION regress_testsub SET (slot_name = NONE); +DROP SUBSCRIPTION regress_testsub; + -- let's do some tests with pg_create_subscription rather than superuser SET SESSION AUTHORIZATION regress_subscription_user3; diff --git a/src/tools/pgindent/typedefs.list b/src/tools/pgindent/typedefs.list index 85d989f395d..0c1ac8f6b64 100644 --- a/src/tools/pgindent/typedefs.list +++ b/src/tools/pgindent/typedefs.list @@ -1707,6 +1707,7 @@ LogicalRepBeginData LogicalRepCommitData LogicalRepCommitPreparedTxnData LogicalRepCtxStruct +LogicalRepMessageData LogicalRepMsgType LogicalRepPartMapEntry LogicalRepPreparedTxnData -- 2.55.0
^ permalink raw reply [nested|flat] 44+ messages in thread

* Re: Add a hook for handling logical decoding messages on subscribers.
@ 2026-08-04 03:03  Fujii Masao <masao.fujii@gmail.com>
  parent: Masahiko Sawada <sawada.mshk@gmail.com>
  2 siblings, 1 reply; 44+ messages in thread

From: Fujii Masao @ 2026-08-04 03:03 UTC (permalink / raw)
  To: Masahiko Sawada <sawada.mshk@gmail.com>; +Cc: Amit Kapila <amit.kapila16@gmail.com>; Bharath Rupireddy <bharath.rupireddyforpostgres@gmail.com>; pgsql-hackers

On Tue, Aug 4, 2026 at 9:42 AM Masahiko Sawada <sawada.mshk@gmail.com> wrote:
> I've rebased and updated the patch. Please review it.

Thanks for the patch!

I tried running the tests with it, and t/002_pg_dump.pl failed as follows:

t/002_pg_dump.pl .. 12151/? # Looks like you failed 26 tests of 12881.
t/002_pg_dump.pl .. Dubious, test returned 26 (wstat 6656, 0x1a00)
Failed 26/12881 subtests

Test Summary Report
-------------------
t/002_pg_dump.pl (Wstat: 6656 Tests: 12881 Failed: 26)
  Failed tests:  201, 468, 735, 1269, 1803, 2074, 2348, 2620
                2891, 3158, 3425, 3692, 3959, 4226, 5027
                5294, 5561, 5829, 6096, 6630, 7432, 7699
                8767, 9034, 10637, 11171
  Non-zero exit status: 26
Files=1, Tests=12881, 31 wallclock secs ( 0.56 usr  0.10 sys +  4.81
cusr  1.72 csys =  7.19 CPU)
Result: FAIL


  options->proto.logical.origin = pstrdup(MySubscription->origin);
+ options->proto.logical.messages = MySubscription->message;

Does this mean the tablesync worker also requests messages = true, even though
it always discards logical decoding messages in apply_handle_message()? If so,
that seems like unnecessary overhead. Shouldn't the tablesync worker always
request messages = false?


-              subsynccommit, subwalrcvtimeout, subpublications, suborigin)
+              subsynccommit, subwalrcvtimeout, subpublications, suborigin,
+       submessage)

In pg_subscription, submessage is defined just before submaxretention, but here
it's added at the end of the column list. It's not a bug, but it would be better
to keep the ordering consistent.


+ /*
+ * A transactional message is applied as a step of the remote transaction
+ * that emitted it, and is committed together with it when applying the
+ * commit message. A non-transactional message belongs to no remote
+ * transaction, so commit it here.
+ */
+ if (!msg.transactional)
+ CommitTransactionCommand();

Shouldn't store_flush_position() also be called in the non-transactional case,
so that the remote/local LSN pair is tracked just as it is during normal commit
processing? Also pgstat_report_stat() etc should be called?


+typedef void (*LogicalRepMessageHandle_hook_type) (LogicalRepMessageData *msg);
+extern PGDLLIMPORT LogicalRepMessageHandle_hook_type
LogicalRepMessageHandle_hook;

These are added to worker_internal.h. Since that header is for worker internals,
would it be better to put them in a separate header such as logicalworker.h?


+ Oid argtypes[5] = {LSNOID, BOOLOID, TEXTOID, INT4OID, TEXTOID};
+ Datum values[5];
+ int ret;
+
+ ereport(LOG,
+ (errmsg("received message: LSN %X/%08X, prefix: %s, message: %s,
transactional: %d",
+ LSN_FORMAT_ARGS(msg->lsn),
+ msg->prefix, msg->message, msg->transactional)));

pg_logical_emit_message() can emit binary messages, but test_logicalmsg_hook.c
doesn't seem to handle them correctly. Is that intentional?


Should psql tab completion also be updated for the new message option
in CREATE/ALTER SUBSCRIPTION ... WITH/SET?

Regards,

-- 
Fujii Masao





^ permalink  raw  reply  [nested|flat] 44+ messages in thread

* Re: Add a hook for handling logical decoding messages on subscribers.
@ 2026-08-04 03:04  Bharath Rupireddy <bharath.rupireddyforpostgres@gmail.com>
  parent: Masahiko Sawada <sawada.mshk@gmail.com>
  2 siblings, 1 reply; 44+ messages in thread

From: Bharath Rupireddy @ 2026-08-04 03:04 UTC (permalink / raw)
  To: Masahiko Sawada <sawada.mshk@gmail.com>; +Cc: Fujii Masao <masao.fujii@gmail.com>; Amit Kapila <amit.kapila16@gmail.com>; pgsql-hackers

Hi,

On Mon, Aug 3, 2026 at 5:42 PM Masahiko Sawada <sawada.mshk@gmail.com> wrote:
>
> Thank you for sharing the concrete use case. This is one of the use
> cases I initially imagined.
>
> I've rebased and updated the patch. Please review it.

Thanks, Sawada-san, for the v2 patch.

I reviewed it and here are some comments:

1/
+ * The handler runs in the apply worker, inside a transaction, with an active
+ * snapshot pushed. For a transactional message the handler's work is part of
+ * the remote transaction and commits with it; a non-transactional message is
+ * committed as an independent unit. The same message may be delivered more
+ * than once, so the handler must be idempotent.

Nice call-out about the more-than-once delivery and the idempotency
requirement. It's true for logical replication in general, but would
it be worth mentioning near the option description in
create_subscription.sgml too?

2/
+ * well-defined ordering between a message and the initial copy. Leave
+ * them to the (parallel) apply workers. Logical messages are handled only
+ * the (parallel) apply workers

Could you fix the last two sentences? They say the same thing.

3/
+ /*
+ * The log table is created by the test, not by this module, so it may not

+#define LOG_SCHEMA "public"
+#define LOG_TABLE "test_logicalmsg_log"

Hard-coding is fine for a test module, but would it be better to have
the module create the log table itself on first use (via SPI, or
heap_create_with_catalog) instead of relying on the test to create it
beforehand?

4/
+ msg_data->prefix = pstrdup(pq_getmsgstring(in));
+ msg = palloc(len + 1);

I understand this isn't a memory leak, because ApplyMessageContext is
reset after every message is handled at the end of the apply loop in
LogicalRepApplyLoop(). But if the intention were to hand the caller
the message in long-lived memory, this palloc in ApplyMessageContext
doesn't achieve that either, since it gets reset after every message
anyway. So why not hand the hook the message directly from the
received StringInfo input buffer and let hook implementors copy it
into whatever context they need?

The point is just that the palloc here doesn't give us anything (if
I'm not missing something), so I'm wondering whether we can avoid it.

5/
+/*
+ * Module load callback
+ */
+void
+_PG_init(void)
+{
+ LogicalRepMessageHandle_hook = &test_logical_message_handler;
+}

This overwrites LogicalRepMessageHandle_hook without saving or calling
the previous value, so any previously registered hook is lost. Is it
intentional? It's just a test module, but others may use it as a
template.

6/
+ * test_logicalmsg_hooks.c
+ * Code for testing LogicalRepMessageHandle_hook
+ *
+ * The handler records every logical decoding message it receives into the
+ * table public.test_logicalmsg_log, so that tests can assert on the contents
+ * of a table rather than on the server log. Recording the messages this way

This is fine, but I personally like the idea Amit posted above: a
simple DDL replication example on the publisher and subscriber. Why
not have that instead? I'm aware it would mean more LoC for the test
extension, but that's fine IMO, since I see DDL replication as a good
use case to demonstrate in the test module.

7/
+ /*
+ * A transactional message is applied as a step of the remote transaction
+ * that emitted it, and is committed together with it when applying the
+ * commit message. A non-transactional message belongs to no remote
+ * transaction, so commit it here.
+ */
+ if (!msg.transactional)
+ CommitTransactionCommand();

Is it safe to commit the transaction that the non-transactional
message was emitted from, given that transaction may have already done
some work?

I haven't verified this in depth, but going by intuition:

BEGIN;
INSERT ...
UPDATE ...
DELETE ...
emit non-transactional message  --> apply worker commits here
UPDATE ...
DELETE ...
COMMIT;

Is something like this safe?

--
Bharath Rupireddy
Amazon Web Services: https://aws.amazon.com






^ permalink  raw  reply  [nested|flat] 44+ messages in thread

* Re: Add a hook for handling logical decoding messages on subscribers.
@ 2026-08-04 08:05  Chao Li <li.evan.chao@gmail.com>
  parent: Masahiko Sawada <sawada.mshk@gmail.com>
  2 siblings, 1 reply; 44+ messages in thread

From: Chao Li @ 2026-08-04 08:05 UTC (permalink / raw)
  To: Masahiko Sawada <sawada.mshk@gmail.com>; +Cc: Fujii Masao <masao.fujii@gmail.com>; Amit Kapila <amit.kapila16@gmail.com>; Bharath Rupireddy <bharath.rupireddyforpostgres@gmail.com>; pgsql-hackers



> On Aug 4, 2026, at 08:41, Masahiko Sawada <sawada.mshk@gmail.com> wrote:
> 
> On Thu, Jul 9, 2026 at 5:14 AM Fujii Masao <masao.fujii@gmail.com> wrote:
>> 
>> On Wed, Jun 24, 2026 at 3:02 AM Masahiko Sawada <sawada.mshk@gmail.com> wrote:
>>> Right. I'm implementing a basic DDL replication solution using this
>>> hook as a sample implementation. Other than that, this feature can be
>>> used to add additional information for the replicated transaction that
>>> is not replicated via logical replication protocol, such as the
>>> executed user or context-specific information, which can then be
>>> dispatched to a CDC system connected to the subscriber.
>> 
>> +1 for the proposed hook, although I haven't read the patch yet.
>> 
>> I know of a system that uses logical decoding messages to propagate data
>> to external systems. In that system, the application writes the data to
>> be propagated as logical decoding messages, and a CDC pipeline consumes
>> them via logical decoding and sends them to other systems, for example
>> through Kafka.
>> 
>> In that system, when the remote site is maintained by physical replication,
>> the same logical decoding messages can be decoded on the standby
>> at the remote site to deliver them to external systems there. However,
>> if the remote site uses logical replication instead, those messages are
>> currently neither delivered to nor processed on the subscriber at
>> the remote site. As a result, the CDC pipeline at the remote site cannot
>> consume the data they carry.
>> 
>> The proposed hook might be useful for this use case. An extension could
>> process the incoming logical decoding messages on the subscriber and
>> forward them to the local CDC pipeline, or store or re-emit them in a form
>> that local consumers can process.
> 
> Thank you for sharing the concrete use case. This is one of the use
> cases I initially imagined.
> 
> I've rebased and updated the patch. Please review it.
> 
> Regards,
> 
> -- 
> Masahiko Sawada
> Amazon Web Services: https://aws.amazon.com
> <v2-0001-Add-a-hook-for-handling-logical-messages-on-subsc.patch>

I played with this feature today. I have a few comments from a design perspective:

1. Would it make sense to pass the receiving subscription’s identity explicitly to the hook? Otherwise, a hook that needs to distinguish messages received by different subscriptions has to obtain it from a global variable such as MySubscription->oid.

2. A hook is invoked by an apply worker, and since a message has no target relation, the hook runs as the subscription owner regardless of run_as_owner. I think it would be useful to document this explicitly.

3. I’m thinking out loud here. Would it be useful to provide an opt-in default hook that re-emits received messages, so that they are written to the subscriber-side WAL and can be consumed by a local logical-decoding client? I understand that a re-emitted message would have a different LSN, just as logically replicated row changes generate new local WAL records. For the use case Fujii-san described, this might allow an existing decoder to continue working after the remote site switches from physical to logical replication.

And a few comment for the code changes:

1 - worker_internal.h
```
+typedef void (*LogicalRepMessageHandle_hook_type) (LogicalRepMessageData *msg);
+extern PGDLLIMPORT LogicalRepMessageHandle_hook_type LogicalRepMessageHandle_hook;
```

I guess we don’t expect a hook function to mutate the message, so maybe make the pointer const.

2 - worker.c
```
+	/*
+	 * Logical messages are relation-agnostic, so they don't map cleanly onto
+	 * the tablesync worker of a particular relation, and there would be no
+	 * well-defined ordering between a message and the initial copy. Leave
+	 * them to the (parallel) apply workers. Logical messages are handled only
+	 * the (parallel) apply workers
+	 */
```

The last sentence looks duplicate and incomplete.

3 - proto.c
```
+	/* read message length */
+	len = pq_getmsgint(in, 4);
+	msg_data->message_size = len;
+
+	/* and data */
+	msg = palloc(len + 1);
+	pq_copymsgbytes(in, msg, len);
+
+	msg[len] = '\0';
+	msg_data->message = msg;
```

logicalrep_read_message() allocates one extra byte for NULL terminator, which is unnecessary, as the contract is to use msg_data->message_size to decide the message length, and a message contain contains 0 in the middle. But I agree it may be useful for debugging and logging, so maybe add a comment to explain why using this extra byte.

Best regards,
--
Chao Li (Evan)
HighGo Software Co., Ltd.
https://www.highgo.com/










^ permalink  raw  reply  [nested|flat] 44+ messages in thread

* Re: Add a hook for handling logical decoding messages on subscribers.
@ 2026-08-04 19:06  Masahiko Sawada <sawada.mshk@gmail.com>
  parent: Fujii Masao <masao.fujii@gmail.com>
  0 siblings, 0 replies; 44+ messages in thread

From: Masahiko Sawada @ 2026-08-04 19:06 UTC (permalink / raw)
  To: Fujii Masao <masao.fujii@gmail.com>; +Cc: Amit Kapila <amit.kapila16@gmail.com>; Bharath Rupireddy <bharath.rupireddyforpostgres@gmail.com>; pgsql-hackers

On Mon, Aug 3, 2026 at 8:03 PM Fujii Masao <masao.fujii@gmail.com> wrote:
>
> On Tue, Aug 4, 2026 at 9:42 AM Masahiko Sawada <sawada.mshk@gmail.com> wrote:
> > I've rebased and updated the patch. Please review it.
>
> Thanks for the patch!

Thank you for reviewing the patch!

> I tried running the tests with it, and t/002_pg_dump.pl failed as follows:
>
> t/002_pg_dump.pl .. 12151/? # Looks like you failed 26 tests of 12881.
> t/002_pg_dump.pl .. Dubious, test returned 26 (wstat 6656, 0x1a00)
> Failed 26/12881 subtests
>
> Test Summary Report
> -------------------
> t/002_pg_dump.pl (Wstat: 6656 Tests: 12881 Failed: 26)
>   Failed tests:  201, 468, 735, 1269, 1803, 2074, 2348, 2620
>                 2891, 3158, 3425, 3692, 3959, 4226, 5027
>                 5294, 5561, 5829, 6096, 6630, 7432, 7699
>                 8767, 9034, 10637, 11171
>   Non-zero exit status: 26
> Files=1, Tests=12881, 31 wallclock secs ( 0.56 usr  0.10 sys +  4.81
> cusr  1.72 csys =  7.19 CPU)
> Result: FAIL

Fixed.

>   options->proto.logical.origin = pstrdup(MySubscription->origin);
> + options->proto.logical.messages = MySubscription->message;
>
> Does this mean the tablesync worker also requests messages = true, even though
> it always discards logical decoding messages in apply_handle_message()? If so,
> that seems like unnecessary overhead. Shouldn't the tablesync worker always
> request messages = false?

Good catch. We can always disable messages for tablesync workers.

>
>
> -              subsynccommit, subwalrcvtimeout, subpublications, suborigin)
> +              subsynccommit, subwalrcvtimeout, subpublications, suborigin,
> +       submessage)
>
> In pg_subscription, submessage is defined just before submaxretention, but here
> it's added at the end of the column list. It's not a bug, but it would be better
> to keep the ordering consistent.

Fixed.

>
>
> + /*
> + * A transactional message is applied as a step of the remote transaction
> + * that emitted it, and is committed together with it when applying the
> + * commit message. A non-transactional message belongs to no remote
> + * transaction, so commit it here.
> + */
> + if (!msg.transactional)
> + CommitTransactionCommand();
>
> Shouldn't store_flush_position() also be called in the non-transactional case,
> so that the remote/local LSN pair is tracked just as it is during normal commit
> processing? Also pgstat_report_stat() etc should be called?

True. In addition to that, I think we should call
replorigin_xact_clear(false) before the commit. Will fix it.

> +typedef void (*LogicalRepMessageHandle_hook_type) (LogicalRepMessageData *msg);
> +extern PGDLLIMPORT LogicalRepMessageHandle_hook_type
> LogicalRepMessageHandle_hook;
>
> These are added to worker_internal.h. Since that header is for worker internals,
> would it be better to put them in a separate header such as logicalworker.h?

Agreed.

>
> + Oid argtypes[5] = {LSNOID, BOOLOID, TEXTOID, INT4OID, TEXTOID};
> + Datum values[5];
> + int ret;
> +
> + ereport(LOG,
> + (errmsg("received message: LSN %X/%08X, prefix: %s, message: %s,
> transactional: %d",
> + LSN_FORMAT_ARGS(msg->lsn),
> + msg->prefix, msg->message, msg->transactional)));
>
> pg_logical_emit_message() can emit binary messages, but test_logicalmsg_hook.c
> doesn't seem to handle them correctly. Is that intentional?

Yes. It supports only text-format messages, which seems fine to me as
we can ensure that only text messages are emitted in this test module.
Do you think it's better to make it work in more general cases?

> Should psql tab completion also be updated for the new message option
> in CREATE/ALTER SUBSCRIPTION ... WITH/SET?

Fixed.

Regards,

--
Masahiko Sawada
Amazon Web Services: https://aws.amazon.com






^ permalink  raw  reply  [nested|flat] 44+ messages in thread

* Re: Add a hook for handling logical decoding messages on subscribers.
@ 2026-08-04 20:25  Masahiko Sawada <sawada.mshk@gmail.com>
  parent: Bharath Rupireddy <bharath.rupireddyforpostgres@gmail.com>
  0 siblings, 1 reply; 44+ messages in thread

From: Masahiko Sawada @ 2026-08-04 20:25 UTC (permalink / raw)
  To: Bharath Rupireddy <bharath.rupireddyforpostgres@gmail.com>; +Cc: Fujii Masao <masao.fujii@gmail.com>; Amit Kapila <amit.kapila16@gmail.com>; pgsql-hackers

On Mon, Aug 3, 2026 at 8:04 PM Bharath Rupireddy
<bharath.rupireddyforpostgres@gmail.com> wrote:
>
> Hi,
>
> On Mon, Aug 3, 2026 at 5:42 PM Masahiko Sawada <sawada.mshk@gmail.com> wrote:
> >
> > Thank you for sharing the concrete use case. This is one of the use
> > cases I initially imagined.
> >
> > I've rebased and updated the patch. Please review it.
>
> Thanks, Sawada-san, for the v2 patch.
>
> I reviewed it and here are some comments:

Thank you for reviewing the patch!

>
> 1/
> + * The handler runs in the apply worker, inside a transaction, with an active
> + * snapshot pushed. For a transactional message the handler's work is part of
> + * the remote transaction and commits with it; a non-transactional message is
> + * committed as an independent unit. The same message may be delivered more
> + * than once, so the handler must be idempotent.
>
> Nice call-out about the more-than-once delivery and the idempotency
> requirement. It's true for logical replication in general, but would
> it be worth mentioning near the option description in
> create_subscription.sgml too?

The patch already mentions in the doc:

+         <para>
+          Received messages are only acted upon if an extension installed on
+          the subscriber has registered a handler for them; otherwise they are
+          received and discarded. A message may be passed to the handler more
+          than once, for example after an apply worker restart, so a handler
+          must be prepared to process the same message repeatedly.
+         </para>

Does it work for you?

>
> 2/
> + * well-defined ordering between a message and the initial copy. Leave
> + * them to the (parallel) apply workers. Logical messages are handled only
> + * the (parallel) apply workers
>
> Could you fix the last two sentences? They say the same thing.

Fixed.

>
> 3/
> + /*
> + * The log table is created by the test, not by this module, so it may not
>
> +#define LOG_SCHEMA "public"
> +#define LOG_TABLE "test_logicalmsg_log"
>
> Hard-coding is fine for a test module, but would it be better to have
> the module create the log table itself on first use (via SPI, or
> heap_create_with_catalog) instead of relying on the test to create it
> beforehand?

Agreed.

>
> 4/
> + msg_data->prefix = pstrdup(pq_getmsgstring(in));
> + msg = palloc(len + 1);
>
> I understand this isn't a memory leak, because ApplyMessageContext is
> reset after every message is handled at the end of the apply loop in
> LogicalRepApplyLoop(). But if the intention were to hand the caller
> the message in long-lived memory, this palloc in ApplyMessageContext
> doesn't achieve that either, since it gets reset after every message
> anyway. So why not hand the hook the message directly from the
> received StringInfo input buffer and let hook implementors copy it
> into whatever context they need?
>
> The point is just that the palloc here doesn't give us anything (if
> I'm not missing something), so I'm wondering whether we can avoid it.

It's just to make the message handling consistent with the rest of
proto.c does (see logicalrep_read_tuple()). I don't think it would be
common for the hook to keep the received message in a long-lived
memory context, so the current API seems fine to me.

> 5/
> +/*
> + * Module load callback
> + */
> +void
> +_PG_init(void)
> +{
> + LogicalRepMessageHandle_hook = &test_logical_message_handler;
> +}
>
> This overwrites LogicalRepMessageHandle_hook without saving or calling
> the previous value, so any previously registered hook is lost. Is it
> intentional? It's just a test module, but others may use it as a
> template.

Agreed.

>
> 6/
> + * test_logicalmsg_hooks.c
> + * Code for testing LogicalRepMessageHandle_hook
> + *
> + * The handler records every logical decoding message it receives into the
> + * table public.test_logicalmsg_log, so that tests can assert on the contents
> + * of a table rather than on the server log. Recording the messages this way
>
> This is fine, but I personally like the idea Amit posted above: a
> simple DDL replication example on the publisher and subscriber. Why
> not have that instead? I'm aware it would mean more LoC for the test
> extension, but that's fine IMO, since I see DDL replication as a good
> use case to demonstrate in the test module.

While query-shipping style DDL replication can be implemented using
this hook, it would not be a great solution to me as neither a test
module nor a contrib module. For a test module, it would need too much
things to be implemented as a test module and people would not use it
as it's not shipped in the package. As for implementing it as a
contrib module, I think the module would not be integrated with the
built-in logical replication well. For example, the extension cannot
respect the table filter and publication options like
publish_via_partition_root etc. Also, if we support DDL replication in
the built-in logical replication, the contrib module would no longer
be necessary but it would be hard to remove it as people might be
using it.

>
> 7/
> + /*
> + * A transactional message is applied as a step of the remote transaction
> + * that emitted it, and is committed together with it when applying the
> + * commit message. A non-transactional message belongs to no remote
> + * transaction, so commit it here.
> + */
> + if (!msg.transactional)
> + CommitTransactionCommand();
>
> Is it safe to commit the transaction that the non-transactional
> message was emitted from, given that transaction may have already done
> some work?

I think that non-transactional messages are never interleaved with the
remote transaction. We emit a non-transactional message as soon as we
decode it (see ReorderBufferQueueMessage()) instead of buffering it.

> I haven't verified this in depth, but going by intuition:
>
> BEGIN;
> INSERT ...
> UPDATE ...
> DELETE ...
> emit non-transactional message  --> apply worker commits here
> UPDATE ...
> DELETE ...
> COMMIT;
>
> Is something like this safe?

In this case, the non-transaction message is sent first and the
subscriber handles it as a single separate transaction, and then the
transaction without the message is sent to the subscriber. I think it
works fine.

Regards,


--
Masahiko Sawada
Amazon Web Services: https://aws.amazon.com






^ permalink  raw  reply  [nested|flat] 44+ messages in thread

* Re: Add a hook for handling logical decoding messages on subscribers.
@ 2026-08-04 22:25  Bharath Rupireddy <bharath.rupireddyforpostgres@gmail.com>
  parent: Masahiko Sawada <sawada.mshk@gmail.com>
  0 siblings, 0 replies; 44+ messages in thread

From: Bharath Rupireddy @ 2026-08-04 22:25 UTC (permalink / raw)
  To: Masahiko Sawada <sawada.mshk@gmail.com>; +Cc: Fujii Masao <masao.fujii@gmail.com>; Amit Kapila <amit.kapila16@gmail.com>; pgsql-hackers

Hi,

On Tue, Aug 4, 2026 at 1:26 PM Masahiko Sawada <sawada.mshk@gmail.com> wrote:
>
> > Nice call-out about the more-than-once delivery and the idempotency
> > requirement. It's true for logical replication in general, but would....
>
> The patch already mentions in the doc:
>
> +         <para>
> +          Received messages are only acted upon if an extension installed on
> +          the subscriber has registered a handler for them; otherwise they are
> +          received and discarded. A message may be passed to the handler more
> +          than once, for example after an apply worker restart, so a handler
> +          must be prepared to process the same message repeatedly.
> +         </para>
>
> Does it work for you?

WFM. I missed that, sorry for the noise!

> > 4/
> > + msg_data->prefix = pstrdup(pq_getmsgstring(in));
> > + msg = palloc(len + 1);
> >
> > I understand this isn't a memory leak, because ApplyMessageContext is
> > reset after every message is handled at the end of the apply loop in
> > LogicalRepApplyLoop(). But if the intention were to hand the caller
> > the message in long-lived memory, this palloc in ApplyMessageContext
> > doesn't achieve that either, since it gets reset after every message
> > anyway. So why not hand the hook the message directly from the
> > received StringInfo input buffer and let hook implementors copy it
> > into whatever context they need?
> >
> > The point is just that the palloc here doesn't give us anything (if
> > I'm not missing something), so I'm wondering whether we can avoid it.
>
> It's just to make the message handling consistent with the rest of
> proto.c does (see logicalrep_read_tuple()). I don't think it would be
> common for the hook to keep the received message in a long-lived
> memory context, so the current API seems fine to me.

Ah, I see it. Also, it's better to send a null-terminated string to
the hooks than sending everything across from the received input
buffer.

> > 6/
> > This is fine, but I personally like the idea Amit posted above: a
> > simple DDL replication example on the publisher and subscriber. Why
> > not have that instead? I'm aware it would mean more LoC for the test
> > extension, but that's fine IMO, since I see DDL replication as a good
> > use case to demonstrate in the test module.
>
> While query-shipping style DDL replication can be implemented using
> this hook, it would not be a great solution to me as neither a test
> module nor a contrib module. For a test module, it would need too much
> things to be implemented as a test module and people would not use it
> as it's not shipped in the package. As for implementing it as a
> contrib module, I think the module would not be integrated with the
> built-in logical replication well. For example, the extension cannot
> respect the table filter and publication options like
> publish_via_partition_root etc. Also, if we support DDL replication in
> the built-in logical replication, the contrib module would no longer
> be necessary but it would be hard to remove it as people might be
> using it.

Agreed to keep it simple the way you have it in the v2 patch.

> > 7/
> > Is it safe to commit the transaction that the non-transactional
> > message was emitted from, given that transaction may have already done
> > some work?
>
> I think that non-transactional messages are never interleaved with the
> remote transaction. We emit a non-transactional message as soon as we
> decode it (see ReorderBufferQueueMessage()) instead of buffering it.
>
> > I haven't verified this in depth, but going by intuition:
> >
> > BEGIN;
> > INSERT ...
> > UPDATE ...
> > DELETE ...
> > emit non-transactional message  --> apply worker commits here
> > UPDATE ...
> > DELETE ...
> > COMMIT;
> >
> > Is something like this safe?
>
> In this case, the non-transaction message is sent first and the
> subscriber handles it as a single separate transaction, and then the
> transaction without the message is sent to the subscriber. I think it
> works fine.

Thanks for pointing me to the code. I understand it now. So, there can
never be a case where a non-transactional message is part of an
in-progress transaction on the apply worker, even in streaming mode,
because logical replication can't interleave transactions for apply.
If so, can we assert this or enhance the comment in
apply_handle_message() a bit?

--
Bharath Rupireddy
Amazon Web Services: https://aws.amazon.com






^ permalink  raw  reply  [nested|flat] 44+ messages in thread

* Re: Add a hook for handling logical decoding messages on subscribers.
@ 2026-08-05 16:32  Masahiko Sawada <sawada.mshk@gmail.com>
  parent: Chao Li <li.evan.chao@gmail.com>
  0 siblings, 1 reply; 44+ messages in thread

From: Masahiko Sawada @ 2026-08-05 16:32 UTC (permalink / raw)
  To: Chao Li <li.evan.chao@gmail.com>; +Cc: Fujii Masao <masao.fujii@gmail.com>; Amit Kapila <amit.kapila16@gmail.com>; Bharath Rupireddy <bharath.rupireddyforpostgres@gmail.com>; pgsql-hackers

On Tue, Aug 4, 2026 at 1:05 AM Chao Li <li.evan.chao@gmail.com> wrote:
>
>
>
> > On Aug 4, 2026, at 08:41, Masahiko Sawada <sawada.mshk@gmail.com> wrote:
> >
> > On Thu, Jul 9, 2026 at 5:14 AM Fujii Masao <masao.fujii@gmail.com> wrote:
> >>
> >> On Wed, Jun 24, 2026 at 3:02 AM Masahiko Sawada <sawada.mshk@gmail.com> wrote:
> >>> Right. I'm implementing a basic DDL replication solution using this
> >>> hook as a sample implementation. Other than that, this feature can be
> >>> used to add additional information for the replicated transaction that
> >>> is not replicated via logical replication protocol, such as the
> >>> executed user or context-specific information, which can then be
> >>> dispatched to a CDC system connected to the subscriber.
> >>
> >> +1 for the proposed hook, although I haven't read the patch yet.
> >>
> >> I know of a system that uses logical decoding messages to propagate data
> >> to external systems. In that system, the application writes the data to
> >> be propagated as logical decoding messages, and a CDC pipeline consumes
> >> them via logical decoding and sends them to other systems, for example
> >> through Kafka.
> >>
> >> In that system, when the remote site is maintained by physical replication,
> >> the same logical decoding messages can be decoded on the standby
> >> at the remote site to deliver them to external systems there. However,
> >> if the remote site uses logical replication instead, those messages are
> >> currently neither delivered to nor processed on the subscriber at
> >> the remote site. As a result, the CDC pipeline at the remote site cannot
> >> consume the data they carry.
> >>
> >> The proposed hook might be useful for this use case. An extension could
> >> process the incoming logical decoding messages on the subscriber and
> >> forward them to the local CDC pipeline, or store or re-emit them in a form
> >> that local consumers can process.
> >
> > Thank you for sharing the concrete use case. This is one of the use
> > cases I initially imagined.
> >
> > I've rebased and updated the patch. Please review it.
> >
> > Regards,
> >
> > --
> > Masahiko Sawada
> > Amazon Web Services: https://aws.amazon.com
> > <v2-0001-Add-a-hook-for-handling-logical-messages-on-subsc.patch>
>
> I played with this feature today. I have a few comments from a design perspective:

Thank you for reviewing the patch!

> 1. Would it make sense to pass the receiving subscription’s identity explicitly to the hook? Otherwise, a hook that needs to distinguish messages received by different subscriptions has to obtain it from a global variable such as MySubscription->oid.

It's common for hook functions to access a global variable to get more
information such as MyDatabaseId and MyProcPort etc. Using
MySubscription to get a subscription's identity works for me.

> 2. A hook is invoked by an apply worker, and since a message has no target relation, the hook runs as the subscription owner regardless of run_as_owner. I think it would be useful to document this explicitly.

I think it depends on the hook function implementations. They can
switch the role as they want.

>
> 3. I’m thinking out loud here. Would it be useful to provide an opt-in default hook that re-emits received messages, so that they are written to the subscriber-side WAL and can be consumed by a local logical-decoding client? I understand that a re-emitted message would have a different LSN, just as logically replicated row changes generate new local WAL records. For the use case Fujii-san described, this might allow an existing decoder to continue working after the remote site switches from physical to logical replication.

Interesting idea, but I'm not sure we should have it in the core or in
the contrib. I think it's a good topic to discuss in a separate
thread.

>
> And a few comment for the code changes:
>
> 1 - worker_internal.h
> ```
> +typedef void (*LogicalRepMessageHandle_hook_type) (LogicalRepMessageData *msg);
> +extern PGDLLIMPORT LogicalRepMessageHandle_hook_type LogicalRepMessageHandle_hook;
> ```
>
> I guess we don’t expect a hook function to mutate the message, so maybe make the pointer const.
>
> 2 - worker.c
> ```
> +       /*
> +        * Logical messages are relation-agnostic, so they don't map cleanly onto
> +        * the tablesync worker of a particular relation, and there would be no
> +        * well-defined ordering between a message and the initial copy. Leave
> +        * them to the (parallel) apply workers. Logical messages are handled only
> +        * the (parallel) apply workers
> +        */
> ```
>
> The last sentence looks duplicate and incomplete.
>
> 3 - proto.c
> ```
> +       /* read message length */
> +       len = pq_getmsgint(in, 4);
> +       msg_data->message_size = len;
> +
> +       /* and data */
> +       msg = palloc(len + 1);
> +       pq_copymsgbytes(in, msg, len);
> +
> +       msg[len] = '\0';
> +       msg_data->message = msg;
> ```
>
> logicalrep_read_message() allocates one extra byte for NULL terminator, which is unnecessary, as the contract is to use msg_data->message_size to decide the message length, and a message contain contains 0 in the middle. But I agree it may be useful for debugging and logging, so maybe add a comment to explain why using this extra byte.

Agreed with the all above comments.

I'll submit the updated patch shortly.

Regards,

-- 
Masahiko Sawada
Amazon Web Services: https://aws.amazon.com






^ permalink  raw  reply  [nested|flat] 44+ messages in thread

* Re: Add a hook for handling logical decoding messages on subscribers.
@ 2026-08-05 18:55  Masahiko Sawada <sawada.mshk@gmail.com>
  parent: Masahiko Sawada <sawada.mshk@gmail.com>
  0 siblings, 2 replies; 44+ messages in thread

From: Masahiko Sawada @ 2026-08-05 18:55 UTC (permalink / raw)
  To: Chao Li <li.evan.chao@gmail.com>; +Cc: Fujii Masao <masao.fujii@gmail.com>; Amit Kapila <amit.kapila16@gmail.com>; Bharath Rupireddy <bharath.rupireddyforpostgres@gmail.com>; pgsql-hackers

On Wed, Aug 5, 2026 at 9:32 AM Masahiko Sawada <sawada.mshk@gmail.com> wrote:
>
> On Tue, Aug 4, 2026 at 1:05 AM Chao Li <li.evan.chao@gmail.com> wrote:
> >
> >
> >
> > > On Aug 4, 2026, at 08:41, Masahiko Sawada <sawada.mshk@gmail.com> wrote:
> > >
> > > On Thu, Jul 9, 2026 at 5:14 AM Fujii Masao <masao.fujii@gmail.com> wrote:
> > >>
> > >> On Wed, Jun 24, 2026 at 3:02 AM Masahiko Sawada <sawada.mshk@gmail.com> wrote:
> > >>> Right. I'm implementing a basic DDL replication solution using this
> > >>> hook as a sample implementation. Other than that, this feature can be
> > >>> used to add additional information for the replicated transaction that
> > >>> is not replicated via logical replication protocol, such as the
> > >>> executed user or context-specific information, which can then be
> > >>> dispatched to a CDC system connected to the subscriber.
> > >>
> > >> +1 for the proposed hook, although I haven't read the patch yet.
> > >>
> > >> I know of a system that uses logical decoding messages to propagate data
> > >> to external systems. In that system, the application writes the data to
> > >> be propagated as logical decoding messages, and a CDC pipeline consumes
> > >> them via logical decoding and sends them to other systems, for example
> > >> through Kafka.
> > >>
> > >> In that system, when the remote site is maintained by physical replication,
> > >> the same logical decoding messages can be decoded on the standby
> > >> at the remote site to deliver them to external systems there. However,
> > >> if the remote site uses logical replication instead, those messages are
> > >> currently neither delivered to nor processed on the subscriber at
> > >> the remote site. As a result, the CDC pipeline at the remote site cannot
> > >> consume the data they carry.
> > >>
> > >> The proposed hook might be useful for this use case. An extension could
> > >> process the incoming logical decoding messages on the subscriber and
> > >> forward them to the local CDC pipeline, or store or re-emit them in a form
> > >> that local consumers can process.
> > >
> > > Thank you for sharing the concrete use case. This is one of the use
> > > cases I initially imagined.
> > >
> > > I've rebased and updated the patch. Please review it.
> > >
> > > Regards,
> > >
> > > --
> > > Masahiko Sawada
> > > Amazon Web Services: https://aws.amazon.com
> > > <v2-0001-Add-a-hook-for-handling-logical-messages-on-subsc.patch>
> >
> > I played with this feature today. I have a few comments from a design perspective:
>
> Thank you for reviewing the patch!
>
> > 1. Would it make sense to pass the receiving subscription’s identity explicitly to the hook? Otherwise, a hook that needs to distinguish messages received by different subscriptions has to obtain it from a global variable such as MySubscription->oid.
>
> It's common for hook functions to access a global variable to get more
> information such as MyDatabaseId and MyProcPort etc. Using
> MySubscription to get a subscription's identity works for me.
>
> > 2. A hook is invoked by an apply worker, and since a message has no target relation, the hook runs as the subscription owner regardless of run_as_owner. I think it would be useful to document this explicitly.
>
> I think it depends on the hook function implementations. They can
> switch the role as they want.
>
> >
> > 3. I’m thinking out loud here. Would it be useful to provide an opt-in default hook that re-emits received messages, so that they are written to the subscriber-side WAL and can be consumed by a local logical-decoding client? I understand that a re-emitted message would have a different LSN, just as logically replicated row changes generate new local WAL records. For the use case Fujii-san described, this might allow an existing decoder to continue working after the remote site switches from physical to logical replication.
>
> Interesting idea, but I'm not sure we should have it in the core or in
> the contrib. I think it's a good topic to discuss in a separate
> thread.
>
> >
> > And a few comment for the code changes:
> >
> > 1 - worker_internal.h
> > ```
> > +typedef void (*LogicalRepMessageHandle_hook_type) (LogicalRepMessageData *msg);
> > +extern PGDLLIMPORT LogicalRepMessageHandle_hook_type LogicalRepMessageHandle_hook;
> > ```
> >
> > I guess we don’t expect a hook function to mutate the message, so maybe make the pointer const.
> >
> > 2 - worker.c
> > ```
> > +       /*
> > +        * Logical messages are relation-agnostic, so they don't map cleanly onto
> > +        * the tablesync worker of a particular relation, and there would be no
> > +        * well-defined ordering between a message and the initial copy. Leave
> > +        * them to the (parallel) apply workers. Logical messages are handled only
> > +        * the (parallel) apply workers
> > +        */
> > ```
> >
> > The last sentence looks duplicate and incomplete.
> >
> > 3 - proto.c
> > ```
> > +       /* read message length */
> > +       len = pq_getmsgint(in, 4);
> > +       msg_data->message_size = len;
> > +
> > +       /* and data */
> > +       msg = palloc(len + 1);
> > +       pq_copymsgbytes(in, msg, len);
> > +
> > +       msg[len] = '\0';
> > +       msg_data->message = msg;
> > ```
> >
> > logicalrep_read_message() allocates one extra byte for NULL terminator, which is unnecessary, as the contract is to use msg_data->message_size to decide the message length, and a message contain contains 0 in the middle. But I agree it may be useful for debugging and logging, so maybe add a comment to explain why using this extra byte.
>
> Agreed with the all above comments.
>
> I'll submit the updated patch shortly.

I've addressed all comments I got so far unless I'm missing anything,
and attached the updated patch. Feedback is very welcome.

Regards,

-- 
Masahiko Sawada
Amazon Web Services: https://aws.amazon.com

Attachments:

v3-0001-Add-a-hook-for-handling-logical-messages-on-subsc.patch (text/x-patch, 111.5K) download From d5e66435d389ae0c2dac1d28cf80f8470d751299 Mon Sep 17 00:00:00 2001 From: Masahiko Sawada <sawada.mshk@gmail.com> Date: Mon, 8 Jun 2026 11:25:47 -0700 Subject: [PATCH v3] Add a hook for handling logical messages on subscribers. Previously, pgoutput could emit logical decoding messages, those written with pg_logical_emit_message(), but the built-in subscriber never requested them from the publisher, and apply_handle_message() was only a placeholder that discarded the MESSAGE message type. Acting on such messages therefore meant writing a separate logical decoding client, and giving up the apply worker's streaming, conflict handling and progress tracking. Logical messages are a general-purpose side channel carried in band with the data, useful for replicating deparsed DDL, for carrying change-data-capture context that the row data does not have, and for cache invalidation delivered in commit order relative to the rows it refers to. This commit adds a new "message" subscription option, which asks the publisher for the pgoutput "messages" option, and dispatches each received message to a new LogicalRepMessageHandle_hook. Behavior is unchanged when the option is off or when no handler is installed. A transactional message is applied as a step of the remote transaction it belongs to, so the handler's work commits atomically with that transaction, and transaction streaming and ALTER SUBSCRIPTION ... SKIP apply to it as they do to any other change. A non-transactional message belongs to no remote transaction and is committed on its own. A message may still reach the handler more than once, for example after an apply worker restart, so handlers must be idempotent. Note also that messages are not scoped by publication: every message emitted in the publisher database is sent, subject only to origin filtering, and pg_logical_emit_message() requires no special privileges. A handler must therefore treat the payload as untrusted input, and as an arbitrary string of bytes rather than as text in the subscriber's encoding. The publisher must be running PostgreSQL 14 or later, where pgoutput gainedthe "messages" option; against an older publisher the option is accepted but no message is ever sent. Bump catalog version. Reviewed-by: Bharath Rupireddy <bharath.rupireddyforpostgres@gmail.com> Reviewed-by: Fujii Masao <masao.fujii@gmail.com> Reviewed-by: Chao Li <li.evan.chao@gmail.com> Discussion: Discussion: https://postgr.es/m/CAD21AoCTNGiddikkUcDKj5QLnsg-51bpr-o6L-GTHWZL4ZFYtQ@mail.gmail.com --- doc/src/sgml/catalogs.sgml | 11 + doc/src/sgml/func/func-admin.sgml | 7 +- doc/src/sgml/ref/alter_subscription.sgml | 5 +- doc/src/sgml/ref/create_subscription.sgml | 43 ++++ src/backend/catalog/pg_subscription.c | 1 + src/backend/catalog/system_views.sql | 5 +- src/backend/commands/subscriptioncmds.c | 25 ++- .../libpqwalreceiver/libpqwalreceiver.c | 4 + src/backend/replication/logical/proto.c | 37 ++++ src/backend/replication/logical/worker.c | 87 +++++++- src/bin/pg_dump/pg_dump.c | 16 +- src/bin/pg_dump/pg_dump.h | 1 + src/bin/pg_dump/t/002_pg_dump.pl | 4 +- src/bin/psql/describe.c | 7 +- src/include/catalog/pg_subscription.h | 5 + src/include/replication/logicalproto.h | 13 ++ src/include/replication/logicalworker.h | 31 +++ src/include/replication/walreceiver.h | 1 + src/test/modules/Makefile | 1 + src/test/modules/meson.build | 1 + .../modules/test_logicalmsg_hooks/.gitignore | 4 + .../modules/test_logicalmsg_hooks/Makefile | 20 ++ .../modules/test_logicalmsg_hooks/meson.build | 28 +++ .../test_logicalmsg_hooks/t/001_basic.pl | 207 ++++++++++++++++++ .../test_logicalmsg_hooks.c | 140 ++++++++++++ src/test/regress/expected/subscription.out | 202 +++++++++-------- src/test/regress/sql/subscription.sql | 17 ++ src/tools/pgindent/typedefs.list | 2 + 28 files changed, 818 insertions(+), 107 deletions(-) create mode 100644 src/test/modules/test_logicalmsg_hooks/.gitignore create mode 100644 src/test/modules/test_logicalmsg_hooks/Makefile create mode 100644 src/test/modules/test_logicalmsg_hooks/meson.build create mode 100644 src/test/modules/test_logicalmsg_hooks/t/001_basic.pl create mode 100644 src/test/modules/test_logicalmsg_hooks/test_logicalmsg_hooks.c diff --git a/doc/src/sgml/catalogs.sgml b/doc/src/sgml/catalogs.sgml index 6066c4784f4..78e6ccdf361 100644 --- a/doc/src/sgml/catalogs.sgml +++ b/doc/src/sgml/catalogs.sgml @@ -8737,6 +8737,17 @@ SCRAM-SHA-256$<replaceable>&lt;iteration count&gt;</replaceable>:<replaceable>&l </para></entry> </row> + <row> + <entry role="catalog_table_entry"><para role="column_definition"> + <structfield>submessage</structfield> <type>bool</type> + </para> + <para> + If true, the subscription requests the publisher to send logical + decoding messages, which are passed to a handler registered by an + extension on the subscriber. + </para></entry> + </row> + <row> <entry role="catalog_table_entry"><para role="column_definition"> <structfield>subconninfo</structfield> <type>text</type> diff --git a/doc/src/sgml/func/func-admin.sgml b/doc/src/sgml/func/func-admin.sgml index 0eae1c1f616..06005e9aa45 100644 --- a/doc/src/sgml/func/func-admin.sgml +++ b/doc/src/sgml/func/func-admin.sgml @@ -1479,7 +1479,12 @@ postgres=# SELECT '0/0'::pg_lsn + pd.segment_number * ps.setting::int + :offset <literal>false</literal>) controls if the message is immediately flushed to WAL or not. <parameter>flush</parameter> has no effect with <parameter>transactional</parameter>, as the message's WAL - record is flushed along with its transaction. + record is flushed along with its transaction. In addition to logical + decoding clients, messages emitted by this function are delivered to + subscribers whose subscriptions have + <link linkend="sql-createsubscription-params-with-message"><literal>message</literal></link> + enabled, where they are passed to a handler registered by an extension + on the subscriber. </para></entry> </row> diff --git a/doc/src/sgml/ref/alter_subscription.sgml b/doc/src/sgml/ref/alter_subscription.sgml index b6f47545682..0e45e7e98ab 100644 --- a/doc/src/sgml/ref/alter_subscription.sgml +++ b/doc/src/sgml/ref/alter_subscription.sgml @@ -309,8 +309,9 @@ ALTER SUBSCRIPTION <replaceable class="parameter">name</replaceable> RENAME TO < <link linkend="sql-createsubscription-params-with-two-phase"><literal>two_phase</literal></link>, <link linkend="sql-createsubscription-params-with-retain-dead-tuples"><literal>retain_dead_tuples</literal></link>, <link linkend="sql-createsubscription-params-with-max-retention-duration"><literal>max_retention_duration</literal></link>, - <link linkend="sql-createsubscription-params-with-wal-receiver-timeout"><literal>wal_receiver_timeout</literal></link>, and - <link linkend="sql-createsubscription-params-with-conflict-log-destination"><literal>conflict_log_destination</literal></link>. + <link linkend="sql-createsubscription-params-with-wal-receiver-timeout"><literal>wal_receiver_timeout</literal></link>, + <link linkend="sql-createsubscription-params-with-conflict-log-destination"><literal>conflict_log_destination</literal></link>, and + <link linkend="sql-createsubscription-params-with-message"><literal>message</literal></link>. Only a superuser can set <literal>password_required = false</literal>. </para> diff --git a/doc/src/sgml/ref/create_subscription.sgml b/doc/src/sgml/ref/create_subscription.sgml index 25a81e2e62a..b8ed5c7816d 100644 --- a/doc/src/sgml/ref/create_subscription.sgml +++ b/doc/src/sgml/ref/create_subscription.sgml @@ -478,6 +478,49 @@ CREATE SUBSCRIPTION <replaceable class="parameter">subscription_name</replaceabl </listitem> </varlistentry> + <varlistentry id="sql-createsubscription-params-with-message"> + <term><literal>message</literal> (<type>boolean</type>)</term> + <listitem> + <para> + Specifies whether the subscription requests the publisher to send + logical decoding messages, that is, those emitted with + <link linkend="pg-logical-emit-message"><function>pg_logical_emit_message</function></link>. + The default is <literal>false</literal>. + </para> + <para> + Received messages are only acted upon if an extension installed on + the subscriber has registered a handler for them; otherwise they are + received and discarded. A message may be passed to the handler more + than once, for example after an apply worker restart, so a handler + must be prepared to process the same message repeatedly. + </para> + <para> + A transactional message is applied as part of the remote transaction + that emitted it, and is therefore delivered in commit order with + respect to other transactions, not in the order in which it was written. + Note that + <link linkend="sql-createsubscription-params-with-streaming"><literal>streaming</literal></link> + and <link linkend="sql-altersubscription-params-skip"><literal>SKIP</literal></link> + apply to transactional messages just as they do to other changes. A + non-transactional message is delivered as soon as it is decoded, even + if the transaction that emitted it later aborted. + </para> + <para> + Messages are not restricted to the tables or schemas of the + subscribed publications: every message emitted in the publisher + database is sent, subject only to + <link linkend="sql-createsubscription-params-with-origin"><literal>origin</literal></link> + filtering. Message contents are supplied by whoever calls + <function>pg_logical_emit_message</function> on the publisher, which + requires no special privileges, so a handler should treat them as + untrusted input. A message is an arbitrary string of bytes, and is + transmitted without character set conversion; it may therefore + contain embedded nulls, and need not be valid text in the encoding + of either database. + </para> + </listitem> + </varlistentry> + <varlistentry id="sql-createsubscription-params-with-origin"> <term><literal>origin</literal> (<type>string</type>)</term> <listitem> diff --git a/src/backend/catalog/pg_subscription.c b/src/backend/catalog/pg_subscription.c index 5ff61edb989..3618161ea96 100644 --- a/src/backend/catalog/pg_subscription.c +++ b/src/backend/catalog/pg_subscription.c @@ -139,6 +139,7 @@ GetSubscription(Oid subid, bool missing_ok, bool conninfo_needed, sub->maxretention = subform->submaxretention; sub->retentionactive = subform->subretentionactive; sub->conflictlogrelid = subform->subconflictlogrelid; + sub->message = subform->submessage; if (conninfo_needed) { diff --git a/src/backend/catalog/system_views.sql b/src/backend/catalog/system_views.sql index 090281a03dd..aab999d4ff8 100644 --- a/src/backend/catalog/system_views.sql +++ b/src/backend/catalog/system_views.sql @@ -1538,8 +1538,9 @@ GRANT SELECT (oid, subdbid, subskiplsn, subname, subowner, subenabled, subbinary, substream, subtwophasestate, subdisableonerr, subpasswordrequired, subrunasowner, subfailover, subretaindeadtuples, submaxretention, subretentionactive, - subserver, subconflictlogrelid, subconflictlogdest, subslotname, - subsynccommit, subwalrcvtimeout, subpublications, suborigin) + subserver, subconflictlogrelid, subconflictlogdest, submessage, + subslotname, subsynccommit, subwalrcvtimeout, subpublications, + suborigin) ON pg_subscription TO public; CREATE VIEW pg_stat_subscription_stats AS diff --git a/src/backend/commands/subscriptioncmds.c b/src/backend/commands/subscriptioncmds.c index d3eef2a3efa..c510679fabf 100644 --- a/src/backend/commands/subscriptioncmds.c +++ b/src/backend/commands/subscriptioncmds.c @@ -83,6 +83,7 @@ #define SUBOPT_LSN 0x00020000 #define SUBOPT_ORIGIN 0x00040000 #define SUBOPT_CONFLICT_LOG_DEST 0x00080000 +#define SUBOPT_MESSAGE 0x00100000 /* check if the 'val' has 'bits' set */ #define IsSet(val, bits) (((val) & (bits)) == (bits)) @@ -109,6 +110,7 @@ typedef struct SubOpts bool runasowner; bool failover; bool retaindeadtuples; + bool message; int32 maxretention; char *origin; ConflictLogDest conflictlogdest; @@ -207,6 +209,8 @@ parse_subscription_options(ParseState *pstate, List *stmt_options, opts->origin = pstrdup(LOGICALREP_ORIGIN_ANY); if (IsSet(supported_opts, SUBOPT_CONFLICT_LOG_DEST)) opts->conflictlogdest = CONFLICT_LOG_DEST_LOG; + if (IsSet(supported_opts, SUBOPT_MESSAGE)) + opts->message = false; /* Parse options */ foreach(lc, stmt_options) @@ -459,6 +463,15 @@ parse_subscription_options(ParseState *pstate, List *stmt_options, opts->conflictlogdest = GetConflictLogDest(val); opts->specified_opts |= SUBOPT_CONFLICT_LOG_DEST; } + else if (IsSet(supported_opts, SUBOPT_MESSAGE) && + strcmp(defel->defname, "message") == 0) + { + if (IsSet(opts->specified_opts, SUBOPT_MESSAGE)) + errorConflictingDefElem(defel, pstate); + + opts->specified_opts |= SUBOPT_MESSAGE; + opts->message = defGetBoolean(defel); + } else ereport(ERROR, (errcode(ERRCODE_SYNTAX_ERROR), @@ -699,7 +712,7 @@ CreateSubscription(ParseState *pstate, CreateSubscriptionStmt *stmt, SUBOPT_RETAIN_DEAD_TUPLES | SUBOPT_MAX_RETENTION_DURATION | SUBOPT_WAL_RECEIVER_TIMEOUT | SUBOPT_ORIGIN | - SUBOPT_CONFLICT_LOG_DEST); + SUBOPT_CONFLICT_LOG_DEST | SUBOPT_MESSAGE); parse_subscription_options(pstate, stmt->options, supported_opts, &opts); /* @@ -854,6 +867,7 @@ CreateSubscription(ParseState *pstate, CreateSubscriptionStmt *stmt, values[Anum_pg_subscription_subretentionactive - 1] = BoolGetDatum(opts.retaindeadtuples); values[Anum_pg_subscription_subserver - 1] = ObjectIdGetDatum(serverid); + values[Anum_pg_subscription_submessage - 1] = BoolGetDatum(opts.message); if (!OidIsValid(serverid)) values[Anum_pg_subscription_subconninfo - 1] = CStringGetTextDatum(conninfo); @@ -1666,7 +1680,8 @@ AlterSubscription(ParseState *pstate, AlterSubscriptionStmt *stmt, SUBOPT_MAX_RETENTION_DURATION | SUBOPT_WAL_RECEIVER_TIMEOUT | SUBOPT_ORIGIN | - SUBOPT_CONFLICT_LOG_DEST); + SUBOPT_CONFLICT_LOG_DEST | + SUBOPT_MESSAGE); break; case ALTER_SUBSCRIPTION_ENABLED: @@ -2053,6 +2068,12 @@ AlterSubscription(ParseState *pstate, AlterSubscriptionStmt *stmt, } } + if (IsSet(opts.specified_opts, SUBOPT_MESSAGE)) + { + values[Anum_pg_subscription_submessage - 1] = BoolGetDatum(opts.message); + replaces[Anum_pg_subscription_submessage - 1] = true; + } + update_tuple = true; break; } diff --git a/src/backend/replication/libpqwalreceiver/libpqwalreceiver.c b/src/backend/replication/libpqwalreceiver/libpqwalreceiver.c index 029990d9fce..e345b80690f 100644 --- a/src/backend/replication/libpqwalreceiver/libpqwalreceiver.c +++ b/src/backend/replication/libpqwalreceiver/libpqwalreceiver.c @@ -631,6 +631,10 @@ libpqrcv_startstreaming(WalReceiverConn *conn, appendQuotedLiteral(&cmd, options->proto.logical.origin); } + if (options->proto.logical.messages && + PQserverVersion(conn->streamConn) >= 140000) + appendStringInfo(&cmd, ", messages 'on'"); + pubnames = options->proto.logical.publication_names; pubnames_str = stringlist_to_identifierstr(pubnames); appendStringInfoString(&cmd, ", publication_names "); diff --git a/src/backend/replication/logical/proto.c b/src/backend/replication/logical/proto.c index 86ad97cd937..6461361d68d 100644 --- a/src/backend/replication/logical/proto.c +++ b/src/backend/replication/logical/proto.c @@ -660,6 +660,43 @@ logicalrep_write_message(StringInfo out, TransactionId xid, XLogRecPtr lsn, pq_sendbytes(out, message, sz); } +/* + * Read MESSAGE from stream. + */ +void +logicalrep_read_message(StringInfo in, LogicalRepMessageData *msg_data) +{ + Size len; + char *msg; + uint8 flags; + + /* read and decode flags */ + flags = pq_getmsgint(in, 1); + msg_data->transactional = (flags & MESSAGE_TRANSACTIONAL) != 0; + + msg_data->lsn = pq_getmsgint64(in); + msg_data->prefix = pstrdup(pq_getmsgstring(in)); + + /* read message length */ + len = pq_getmsgint(in, 4); + msg_data->message_size = len; + + /* and data */ + msg = palloc(len + 1); + pq_copymsgbytes(in, msg, len); + + /* + * The payload is an arbitrary string of bytes, so message_size, not + * strlen(), is what gives its length. NULL-terminate it anyway, as + * logicalrep_read_tuple() does for column values: that way it can be + * handed to code expecting a C string without reading past the end, which + * is convenient for logging and debugging. Such code will of course stop + * at the first embedded null, if any. + */ + msg[len] = '\0'; + msg_data->message = msg; +} + /* * Write relation description to the output stream. */ diff --git a/src/backend/replication/logical/worker.c b/src/backend/replication/logical/worker.c index ce4f1b93b66..3b3d421c2b9 100644 --- a/src/backend/replication/logical/worker.c +++ b/src/backend/replication/logical/worker.c @@ -549,6 +549,9 @@ typedef struct ApplySubXactData static ApplySubXactData subxact_data = {0, 0, InvalidTransactionId, NULL}; +/* Hook for plugins to get control in handling logical decoding messages */ +LogicalRepMessageHandle_hook_type LogicalRepMessageHandle_hook = NULL; + static inline void subxact_filename(char *path, Oid subid, TransactionId xid); static inline void changes_filename(char *path, Oid subid, TransactionId xid); @@ -1694,6 +1697,71 @@ apply_handle_origin(StringInfo s) errmsg_internal("ORIGIN message sent out of order"))); } +/* + * Handle MESSAGE message. + * + * Invoke a hook function if set. + */ +static void +apply_handle_message(StringInfo s) +{ + LogicalRepMessageData msg; + + /* Tablesync worker should never receive MESSAGE */ + if (am_tablesync_worker()) + ereport(ERROR, + errcode(ERRCODE_PROTOCOL_VIOLATION), + errmsg_internal("tablesync worker received a MESSAGE message")); + + if (!LogicalRepMessageHandle_hook) + return; + + if (is_skipping_changes() || + handle_streamed_transaction(LOGICAL_REP_MSG_MESSAGE, s)) + return; + + logicalrep_read_message(s, &msg); + + begin_replication_step(); + + (*LogicalRepMessageHandle_hook) (&msg); + + end_replication_step(); + + /* + * A transactional message is applied as a step of the remote transaction + * that emitted it, and is committed together with it when applying the + * commit message. A non-transactional message belongs to no remote + * transaction, so commit it here. + */ + if (!msg.transactional) + { + Assert(!in_remote_transaction); + Assert(!in_streamed_transaction); + + /* + * The message doesn't belong to any remote transaction, so there is + * no remote commit LSN nor timestamp to record. Clear the state left + * over by the previously applied transaction so that this commit + * doesn't inherit it. + */ + replorigin_xact_clear(false); + + CommitTransactionCommand(); + + pgstat_report_stat(false); + + /* + * Report the flush position only after the local commit is durable. + * Otherwise send_feedback() would report everything received so far + * as flushed, the publisher would advance confirmed_flush_lsn past + * this message, and a crash before the commit reaches disk would lose + * the handler's work without the message ever being sent again. + */ + store_flush_position(msg.lsn, XactLastCommitEnd); + } +} + /* * Initialize fileset (if not already done). * @@ -3846,12 +3914,7 @@ apply_dispatch(StringInfo s) break; case LOGICAL_REP_MSG_MESSAGE: - - /* - * Logical replication does not use generic logical messages yet. - * Although, it could be used by other applications that use this - * output plugin. - */ + apply_handle_message(s); break; case LOGICAL_REP_MSG_STREAM_START: @@ -5128,7 +5191,8 @@ maybe_reread_subscription(void) newsub->passwordrequired != MySubscription->passwordrequired || strcmp(newsub->origin, MySubscription->origin) != 0 || newsub->owner != MySubscription->owner || - !equal(newsub->publications, MySubscription->publications)) + !equal(newsub->publications, MySubscription->publications) || + newsub->message != MySubscription->message) { if (am_parallel_apply_worker()) ereport(LOG, @@ -5614,6 +5678,15 @@ set_stream_options(WalRcvStreamOptions *options, options->proto.logical.twophase = false; options->proto.logical.origin = pstrdup(MySubscription->origin); + + /* + * Logical messages are relation-agnostic, so they don't map cleanly onto + * the tablesync worker of a particular relation, and there would be no + * well-defined ordering between a message and the initial copy. Leave + * them to the (parallel) apply workers. + */ + options->proto.logical.messages = (MySubscription->message && + !am_tablesync_worker()); } /* diff --git a/src/bin/pg_dump/pg_dump.c b/src/bin/pg_dump/pg_dump.c index 4948e6d80c7..31481720168 100644 --- a/src/bin/pg_dump/pg_dump.c +++ b/src/bin/pg_dump/pg_dump.c @@ -5138,6 +5138,7 @@ getSubscriptions(Archive *fout) int i_subfailover; int i_subretaindeadtuples; int i_submaxretention; + int i_submessage; int i, ntups; @@ -5236,9 +5237,14 @@ getSubscriptions(Archive *fout) " '-1' AS subwalrcvtimeout,\n"); if (fout->remoteVersion >= 190000) - appendPQExpBufferStr(query, " fs.srvname AS subservername\n"); + appendPQExpBufferStr(query, " fs.srvname AS subservername,\n"); else - appendPQExpBufferStr(query, " NULL AS subservername\n"); + appendPQExpBufferStr(query, " NULL AS subservername,\n"); + + if (fout->remoteVersion >= 200000) + appendPQExpBufferStr(query, " s.submessage\n"); + else + appendPQExpBufferStr(query, " false AS submessage\n"); appendPQExpBufferStr(query, "FROM pg_subscription s\n"); @@ -5287,6 +5293,7 @@ getSubscriptions(Archive *fout) i_subpublications = PQfnumber(res, "subpublications"); i_suborigin = PQfnumber(res, "suborigin"); i_suboriginremotelsn = PQfnumber(res, "suboriginremotelsn"); + i_submessage = PQfnumber(res, "submessage"); subinfo = pg_malloc_array(SubscriptionInfo, ntups); @@ -5344,6 +5351,8 @@ getSubscriptions(Archive *fout) else subinfo[i].suboriginremotelsn = pg_strdup(PQgetvalue(res, i, i_suboriginremotelsn)); + subinfo[i].submessage = + (strcmp(PQgetvalue(res, i, i_submessage), "t") == 0); /* Decide whether we want to dump it */ selectDumpableObject(&(subinfo[i].dobj), fout); @@ -5596,6 +5605,9 @@ dumpSubscription(Archive *fout, const SubscriptionInfo *subinfo) if (subinfo->subretaindeadtuples) appendPQExpBufferStr(query, ", retain_dead_tuples = true"); + if (subinfo->submessage) + appendPQExpBufferStr(query, ", message = true"); + if (subinfo->submaxretention) appendPQExpBuffer(query, ", max_retention_duration = %d", subinfo->submaxretention); diff --git a/src/bin/pg_dump/pg_dump.h b/src/bin/pg_dump/pg_dump.h index 5a6726d8b12..30bd1575a26 100644 --- a/src/bin/pg_dump/pg_dump.h +++ b/src/bin/pg_dump/pg_dump.h @@ -721,6 +721,7 @@ typedef struct _SubscriptionInfo bool subrunasowner; bool subfailover; bool subretaindeadtuples; + bool submessage; int submaxretention; char *subservername; char *subconninfo; diff --git a/src/bin/pg_dump/t/002_pg_dump.pl b/src/bin/pg_dump/t/002_pg_dump.pl index 9258948b583..70fe1276d0a 100644 --- a/src/bin/pg_dump/t/002_pg_dump.pl +++ b/src/bin/pg_dump/t/002_pg_dump.pl @@ -3317,9 +3317,9 @@ my %tests = ( create_order => 50, create_sql => 'CREATE SUBSCRIPTION sub3 CONNECTION \'dbname=doesnotexist\' PUBLICATION pub1 - WITH (connect = false, origin = any, streaming = on);', + WITH (connect = false, origin = any, streaming = on, message = on);', regexp => qr/^ - \QCREATE SUBSCRIPTION sub3 CONNECTION 'dbname=doesnotexist' PUBLICATION pub1 WITH (connect = false, slot_name = 'sub3', streaming = on);\E + \QCREATE SUBSCRIPTION sub3 CONNECTION 'dbname=doesnotexist' PUBLICATION pub1 WITH (connect = false, slot_name = 'sub3', streaming = on, message = true);\E /xm, like => { %full_runs, section_post_data => 1, }, unlike => { diff --git a/src/bin/psql/describe.c b/src/bin/psql/describe.c index ad9c8affb4f..cdb6bbb2184 100644 --- a/src/bin/psql/describe.c +++ b/src/bin/psql/describe.c @@ -6894,7 +6894,7 @@ describeSubscriptions(const char *pattern, bool verbose) printQueryOpt myopt = pset.popt; static const bool translate_columns[] = {false, false, false, false, false, false, false, false, false, false, false, false, false, false, - false, false, false, false, false, false, false}; + false, false, false, false, false, false, false, false}; initPQExpBuffer(&buf); @@ -6972,6 +6972,11 @@ describeSubscriptions(const char *pattern, bool verbose) gettext_noop("Retention active")); } + if (pset.sversion >= 200000) + appendPQExpBuffer(&buf, + ", submessage AS \"%s\"\n", + gettext_noop("Message")); + appendPQExpBuffer(&buf, ", subsynccommit AS \"%s\"\n" ", subconninfo AS \"%s\"\n", diff --git a/src/include/catalog/pg_subscription.h b/src/include/catalog/pg_subscription.h index 65ce8e145fb..7c0df9827bf 100644 --- a/src/include/catalog/pg_subscription.h +++ b/src/include/catalog/pg_subscription.h @@ -96,6 +96,9 @@ CATALOG(pg_subscription,6100,SubscriptionRelationId) BKI_SHARED_RELATION BKI_ROW * server */ Oid subconflictlogrelid; /* Relid of the conflict log table. */ + + bool submessage; /* True if the subscription wants to receive + * logical messages. */ #ifdef CATALOG_VARLEN /* variable-length fields start here */ /* @@ -173,6 +176,8 @@ typedef struct Subscription * exceeded max_retention_duration, when * defined */ Oid conflictlogrelid; /* conflict log table Oid */ + bool message; /* True if the subscription wants to receive + * logical messages */ char *conninfo; /* Connection string to the publisher */ char *slotname; /* Name of the replication slot */ char *synccommit; /* Synchronous commit setting for worker */ diff --git a/src/include/replication/logicalproto.h b/src/include/replication/logicalproto.h index 058a955e20c..83741a2080d 100644 --- a/src/include/replication/logicalproto.h +++ b/src/include/replication/logicalproto.h @@ -191,6 +191,18 @@ typedef struct LogicalRepStreamAbortData TimestampTz abort_time; } LogicalRepStreamAbortData; +/* + * Logical decoding message information + */ +typedef struct LogicalRepMessageData +{ + XLogRecPtr lsn; + bool transactional; + const char *prefix; + Size message_size; /* length of message in bytes */ + const char *message; /* payload; may contain embedded nulls */ +} LogicalRepMessageData; + extern void logicalrep_write_begin(StringInfo out, ReorderBufferTXN *txn); extern void logicalrep_read_begin(StringInfo in, LogicalRepBeginData *begin_data); @@ -248,6 +260,7 @@ extern List *logicalrep_read_truncate(StringInfo in, bool *cascade, bool *restart_seqs); extern void logicalrep_write_message(StringInfo out, TransactionId xid, XLogRecPtr lsn, bool transactional, const char *prefix, Size sz, const char *message); +extern void logicalrep_read_message(StringInfo in, LogicalRepMessageData *msg_data); extern void logicalrep_write_rel(StringInfo out, TransactionId xid, Relation rel, Bitmapset *columns, PublishGencolsType include_gencols_type); diff --git a/src/include/replication/logicalworker.h b/src/include/replication/logicalworker.h index 7d748a28da8..6a68c47f12b 100644 --- a/src/include/replication/logicalworker.h +++ b/src/include/replication/logicalworker.h @@ -16,6 +16,37 @@ extern PGDLLIMPORT volatile sig_atomic_t ParallelApplyMessagePending; +/* avoid include logicalproto.h */ +struct LogicalRepMessageData; + +/* + * Hook for extensions to handle logical decoding messages (see + * pg_logical_emit_message()) received from the publisher. Called only when + * the subscription has the "message" option enabled. + * + * The handler runs in the apply worker, inside a transaction, with an active + * snapshot pushed. For a transactional message the handler's work is part of + * the remote transaction and commits with it; a non-transactional message is + * committed as an independent unit. The same message may be delivered more + * than once, so the handler must be idempotent. + * + * The payload is an arbitrary string of bytes. The WAL record makes no + * distinction between the text and the bytea form of + * pg_logical_emit_message(), and the bytes are transmitted without character + * set conversion, so the payload may contain embedded nulls and bytes that + * are not valid in the subscriber's encoding. Its length is given by + * msg->message_size, not by strlen(); the buffer is null-terminated only for + * convenience, and that null is not part of the payload. + * + * Note that the apply worker runs with an empty search_path. An ERROR thrown + * by the handler restarts the apply worker, or disables the subscription if + * disable_on_error is set. Because the message is delivered again after the + * restart, a handler cannot reject a message by throwing an error; that only + * loops. A message the handler does not want must simply be ignored. + */ +typedef void (*LogicalRepMessageHandle_hook_type) (const struct LogicalRepMessageData *msg); +extern PGDLLIMPORT LogicalRepMessageHandle_hook_type LogicalRepMessageHandle_hook; + extern void ApplyWorkerMain(Datum main_arg); extern void ParallelApplyWorkerMain(Datum main_arg); extern void TableSyncWorkerMain(Datum main_arg); diff --git a/src/include/replication/walreceiver.h b/src/include/replication/walreceiver.h index 5cfe90f9989..eca847c4a8f 100644 --- a/src/include/replication/walreceiver.h +++ b/src/include/replication/walreceiver.h @@ -188,6 +188,7 @@ typedef struct * prepare time */ char *origin; /* Only publish data originating from the * specified origin */ + bool messages; /* Logical message */ } logical; } proto; } WalRcvStreamOptions; diff --git a/src/test/modules/Makefile b/src/test/modules/Makefile index 098bb8142ae..e8a7f50eb38 100644 --- a/src/test/modules/Makefile +++ b/src/test/modules/Makefile @@ -36,6 +36,7 @@ SUBDIRS = \ test_integerset \ test_json_parser \ test_lfind \ + test_logicalmsg_hooks \ test_lwlock_tranches \ test_misc \ test_oat_hooks \ diff --git a/src/test/modules/meson.build b/src/test/modules/meson.build index 4bca42bb370..5f6c0cae77a 100644 --- a/src/test/modules/meson.build +++ b/src/test/modules/meson.build @@ -37,6 +37,7 @@ subdir('test_int128') subdir('test_integerset') subdir('test_json_parser') subdir('test_lfind') +subdir('test_logicalmsg_hooks') subdir('test_lwlock_tranches') subdir('test_misc') subdir('test_oat_hooks') diff --git a/src/test/modules/test_logicalmsg_hooks/.gitignore b/src/test/modules/test_logicalmsg_hooks/.gitignore new file mode 100644 index 00000000000..5dcb3ff9723 --- /dev/null +++ b/src/test/modules/test_logicalmsg_hooks/.gitignore @@ -0,0 +1,4 @@ +# Generated subdirectories +/log/ +/results/ +/tmp_check/ diff --git a/src/test/modules/test_logicalmsg_hooks/Makefile b/src/test/modules/test_logicalmsg_hooks/Makefile new file mode 100644 index 00000000000..af129009b8a --- /dev/null +++ b/src/test/modules/test_logicalmsg_hooks/Makefile @@ -0,0 +1,20 @@ +# src/test/modules/test_logicalmsg_hooks/Makefile + +MODULE_big = test_logicalmsg_hooks +OBJS = \ + $(WIN32RES) \ + test_logicalmsg_hooks.o +PGFILEDESC = "test_logicalmsg_hooks - test logical replication message hooks" + +TAP_TESTS = 1 + +ifdef USE_PGXS +PG_CONFIG = pg_config +PGXS := $(shell $(PG_CONFIG) --pgxs) +include $(PGXS) +else +subdir = src/test/modules/test_logicalmsg_hooks +top_builddir = ../../../.. +include $(top_builddir)/src/Makefile.global +include $(top_srcdir)/contrib/contrib-global.mk +endif diff --git a/src/test/modules/test_logicalmsg_hooks/meson.build b/src/test/modules/test_logicalmsg_hooks/meson.build new file mode 100644 index 00000000000..8084d82e47b --- /dev/null +++ b/src/test/modules/test_logicalmsg_hooks/meson.build @@ -0,0 +1,28 @@ +# Copyright (c) 2022-2026, PostgreSQL Global Development Group + +test_logicalmsg_hooks_sources = files( + 'test_logicalmsg_hooks.c', +) + +if host_system == 'windows' + test_logicalmsg_hooks_sources += rc_lib_gen.process(win32ver_rc, extra_args: [ + '--NAME', 'test_logicalmsg_hooks', + '--FILEDESC', 'test_logicalmsg_hooks - test logical message hooks',]) +endif + +test_logicalmsg_hooks = shared_module('test_logicalmsg_hooks', + test_logicalmsg_hooks_sources, + kwargs: pg_test_mod_args, +) +test_install_libs += test_logicalmsg_hooks + +tests += { + 'name': 'test_logicalmsg_hooks', + 'sd': meson.current_source_dir(), + 'bd': meson.current_build_dir(), + 'tap': { + 'tests': [ + 't/001_basic.pl', + ], + }, +} diff --git a/src/test/modules/test_logicalmsg_hooks/t/001_basic.pl b/src/test/modules/test_logicalmsg_hooks/t/001_basic.pl new file mode 100644 index 00000000000..65d246c5031 --- /dev/null +++ b/src/test/modules/test_logicalmsg_hooks/t/001_basic.pl @@ -0,0 +1,207 @@ +# Copyright (c) 2026, PostgreSQL Global Development Group + +# Test LogicalRepMessageHandle_hook. The test module records every message it +# receives into public.test_logicalmsg_log on the subscriber, so most checks +# here are plain queries against that table. + +use strict; +use warnings FATAL => 'all'; + +use PostgreSQL::Test::Cluster; +use PostgreSQL::Test::Utils; +use Test::More; + +my $pub = PostgreSQL::Test::Cluster->new('publisher'); +$pub->init(allows_streaming => 'logical'); + +# Keep the memory limit low so that the transactions below are streamed. +$pub->append_conf('postgresql.conf', "logical_decoding_work_mem = 64kB\n"); +$pub->start; + +my $sub = PostgreSQL::Test::Cluster->new('subscriber'); +$sub->init(allows_streaming => 'logical'); +$sub->append_conf( + 'postgresql.conf', q{ +shared_preload_libraries = 'test_logicalmsg_hooks' +max_parallel_apply_workers_per_subscription = 2 +}); +$sub->start; + +$pub->safe_psql( + 'postgres', qq{ +CREATE TABLE test (a int); +INSERT INTO test VALUES (1); +CREATE PUBLICATION pub FOR ALL TABLES; +}); + +my $pub_connstr = $pub->connstr . ' dbname=postgres'; + +$sub->safe_psql( + 'postgres', qq{ +CREATE TABLE test (a int primary key); +CREATE SUBSCRIPTION sub CONNECTION '$pub_connstr' PUBLICATION pub WITH (message = true, disable_on_error = true) +}); + +$sub->wait_for_subscription_sync($pub, 'sub'); + +# Number of messages recorded under the given prefix. +sub logged_count +{ + my ($prefix) = @_; + + return $sub->safe_psql('postgres', + "SELECT count(*) FROM test_logicalmsg_log WHERE prefix = '$prefix'"); +} + +# Verify that a transactional message is passed to the hook, and that the +# message data given to the hook is correct. +$pub->safe_psql('postgres', + q{SELECT pg_logical_emit_message(true, 'basic', 'transactional message')} +); +$pub->wait_for_catchup('sub'); + +is( $sub->safe_psql( + 'postgres', + q{SELECT transactional, message_size, message FROM test_logicalmsg_log + WHERE prefix = 'basic'}), + 't|21|transactional message', + "transactional message is passed to the hook"); + +# Verify that a non-transactional message is applied even if the transaction +# that emitted it rolls back. +$pub->safe_psql( + 'postgres', q{ +BEGIN; +SELECT pg_logical_emit_message(false, 'nontransactional', 'emitted then rolled back'); +ROLLBACK; +}); + +# Commit another message to flush WAL, as ROLLBACK doesn't flush. +$pub->safe_psql('postgres', + q{SELECT pg_logical_emit_message(true, 'flush', 'flush')}); +$pub->wait_for_catchup('sub'); + +is( $sub->safe_psql( + 'postgres', + q{SELECT transactional, message FROM test_logicalmsg_log + WHERE prefix = 'nontransactional'}), + 'f|emitted then rolled back', + "non-transactional message of an aborted transaction is applied"); + +# Verify that a transactional message emitted in a subtransaction that is +# rolled back is not passed to the hook. +$pub->safe_psql( + 'postgres', q{ +BEGIN; +SAVEPOINT sp; +SELECT pg_logical_emit_message(true, 'subxact_aborted', 'not delivered'); +ROLLBACK TO SAVEPOINT sp; +SELECT pg_logical_emit_message(true, 'subxact_committed', 'delivered'); +COMMIT; +}); +$pub->wait_for_catchup('sub'); + +is(logged_count('subxact_aborted'), + 0, "message of an aborted subtransaction is not applied"); +is(logged_count('subxact_committed'), + 1, "message of a committed subtransaction is applied"); + +# Verify that transactional messages are applied in streaming transactions, +# both when the leader apply worker serializes the transaction to a file and +# when it hands the transaction to a parallel apply worker. +foreach my $mode ('on', 'parallel') +{ + $sub->safe_psql('postgres', + "ALTER SUBSCRIPTION sub SET (streaming = $mode)"); + + # Reconnect so that the new streaming mode takes effect immediately. + $sub->safe_psql('postgres', 'ALTER SUBSCRIPTION sub DISABLE'); + $sub->safe_psql('postgres', 'ALTER SUBSCRIPTION sub ENABLE'); + + $pub->safe_psql('postgres', + q{SELECT pg_stat_reset_replication_slot('sub')}); + + $pub->safe_psql( + 'postgres', qq{ +BEGIN; +INSERT INTO test SELECT generate_series(1000, 6000); +SELECT pg_logical_emit_message(true, 'streaming_$mode', 'streamed message'); +COMMIT; +}); + $pub->wait_for_catchup('sub'); + + # Make sure the transaction really was streamed, so that this test cannot + # silently degrade into a non-streaming one. + $pub->poll_query_until('postgres', + q{SELECT stream_txns > 0 FROM pg_stat_replication_slots WHERE slot_name = 'sub'} + ) or die "transaction was not streamed"; + + is(logged_count("streaming_$mode"), + 1, + "message in a streamed transaction is applied (streaming = $mode)"); + + $pub->safe_psql('postgres', 'TRUNCATE test'); + $pub->wait_for_catchup('sub'); +} + +$sub->safe_psql('postgres', 'ALTER SUBSCRIPTION sub SET (streaming = off)'); + +# Verify that ALTER SUBSCRIPTION ... SKIP skips a transactional message along +# with the rest of its transaction. The message is emitted before the +# conflicting change, so the hook runs and records the message on the first +# attempt; that work must be rolled back with the failed transaction, and must +# not be redone once the transaction is skipped. +$pub->safe_psql('postgres', 'INSERT INTO test VALUES (1)'); +$pub->wait_for_catchup('sub'); + +my $log_location = -s $sub->logfile; + +$pub->safe_psql( + 'postgres', q{ +BEGIN; +SELECT pg_logical_emit_message(true, 'skipped', 'rolled back, then skipped'); +INSERT INTO test VALUES (1); +COMMIT; +}); + +# Wait until the conflict disables the subscription. +$sub->poll_query_until('postgres', + q{SELECT subenabled = FALSE FROM pg_subscription WHERE subname = 'sub'}); + +# The hook did run before the transaction failed. +ok( $sub->log_contains( + qr/LOG[^\n]+received message: [^\n]+prefix: skipped/, + $log_location), + "hook is called for a message in a transaction that later fails"); + +# ... but its work was rolled back with the transaction. +is(logged_count('skipped'), 0, + "work done by the hook is rolled back with the remote transaction"); + +# Get the finish LSN of the failed transaction. +my $contents = slurp_file($sub->logfile, $log_location); +$contents =~ + qr/conflict detected on relation "public.test".*\n.*DETAIL:.*Could not apply remote change.*\n.*Key already exists in unique index "test_pkey", modified in transaction \d+: key .*, local row .*\n.*CONTEXT:.* for replication target relation "public.test" in transaction \d+, finished at ([[:xdigit:]]+\/[[:xdigit:]]+)/m + or die "could not get error-LSN"; +my $lsn = $1; + +$log_location = -s $sub->logfile; + +# Set skip LSN and re-enable the subscription. +$sub->safe_psql('postgres', qq{ALTER SUBSCRIPTION sub SKIP (lsn = '$lsn')}); +$sub->safe_psql('postgres', 'ALTER SUBSCRIPTION sub ENABLE'); + +# Wait for the failed transaction to be skipped. +$sub->poll_query_until('postgres', + q{SELECT subskiplsn = '0/0' FROM pg_subscription WHERE subname = 'sub'}); + +ok( !$sub->log_contains( + qr/LOG[^\n]+received message: [^\n]+prefix: skipped/, + $log_location), + "hook is not called for a message in a skipped transaction"); +is(logged_count('skipped'), 0, + "message in a skipped transaction is not applied"); + +$sub->stop; +$pub->stop; +done_testing(); diff --git a/src/test/modules/test_logicalmsg_hooks/test_logicalmsg_hooks.c b/src/test/modules/test_logicalmsg_hooks/test_logicalmsg_hooks.c new file mode 100644 index 00000000000..d8d6fc7c619 --- /dev/null +++ b/src/test/modules/test_logicalmsg_hooks/test_logicalmsg_hooks.c @@ -0,0 +1,140 @@ +/*-------------------------------------------------------------------------- + * + * test_logicalmsg_hooks.c + * Code for testing LogicalRepMessageHandle_hook + * + * The handler records every logical decoding message it receives into the + * table public.test_logicalmsg_log, so that tests can assert on the contents + * of a table rather than on the server log. Recording the messages this way + * also exercises the transactional behavior of the hook: a transactional + * message is recorded as part of the remote transaction that emitted it, and + * therefore disappears together with it if that transaction is rolled back or + * skipped, while a non-transactional message is recorded independently. + * + * The handler creates that table itself on the first message, so a test does + * not have to set it up. For a transactional message the creation is part of + * the remote transaction too, and is rolled back with it, so the table does + * not exist until some message has been applied successfully; a test must not + * query it before then. + * + * The messages are also logged, which lets a test tell "the handler ran but + * its work was rolled back" apart from "the handler was never called". + * + * Portions Copyright (c) 1996-2026, PostgreSQL Global Development Group + * Portions Copyright (c) 1994, Regents of the University of California + * + * IDENTIFICATION + * src/test/modules/test_logicalmsg_hooks/test_logicalmsg_hooks.c + * + * ------------------------------------------------------------------------- + */ + +#include "postgres.h" + +#include "catalog/namespace.h" +#include "catalog/pg_type.h" +#include "executor/spi.h" +#include "replication/logicalproto.h" +#include "replication/logicalworker.h" +#include "utils/builtins.h" +#include "utils/lsyscache.h" +#include "utils/pg_lsn.h" + +PG_MODULE_MAGIC; + +#define LOG_SCHEMA "public" +#define LOG_TABLE "test_logicalmsg_log" + +static LogicalRepMessageHandle_hook_type prev_logical_message_handler = NULL; +static void test_logical_message_handler(const LogicalRepMessageData *msg); + +/* + * Module load callback + */ +void +_PG_init(void) +{ + prev_logical_message_handler = LogicalRepMessageHandle_hook; + LogicalRepMessageHandle_hook = &test_logical_message_handler; +} + +/* + * Create the log table if not exists. + */ +static void +ensure_log_table_exists(void) +{ + Oid relid; + int ret; + + relid = get_relname_relid(LOG_TABLE, get_namespace_oid(LOG_SCHEMA, true)); + if (OidIsValid(relid)) + return; + + SPI_connect(); + + ret = SPI_execute("CREATE TABLE " LOG_SCHEMA "." LOG_TABLE + "(lsn pg_lsn, transactional boolean, prefix text, message_size int, message text)", + false, 0); + if (ret != SPI_OK_UTILITY) + elog(ERROR, "could not create " LOG_SCHEMA "." LOG_TABLE); + + SPI_finish(); +} + +static void +test_logical_message_handler(const LogicalRepMessageData *msg) +{ + Oid argtypes[5] = {LSNOID, BOOLOID, TEXTOID, INT4OID, TEXTOID}; + Datum values[5]; + int ret; + + /* + * Give a handler installed before ours its turn, so that several + * extensions can act on the same message. + */ + if (prev_logical_message_handler) + (*prev_logical_message_handler) (msg); + + ereport(LOG, + (errmsg("received message: LSN %X/%08X, prefix: %s, size: %zu, transactional: %d", + LSN_FORMAT_ARGS(msg->lsn), msg->prefix, + msg->message_size, msg->transactional))); + + ensure_log_table_exists(); + + /* + * Pass the message as query parameters rather than interpolating it into + * the query text. Message contents come from the publisher and are not to + * be trusted, and the payload may contain characters that would need + * quoting. Note also that message_size, not strlen(), is authoritative + * for the payload length. + * + * The payload is recorded as text because the accompanying test only ever + * emits text messages. A handler must not assume that in general: a + * payload is an arbitrary string of bytes, which may contain embedded + * nulls and bytes that are not valid in the subscriber's encoding. See + * the comments on LogicalRepMessageHandle_hook_type. + */ + values[0] = LSNGetDatum(msg->lsn); + values[1] = BoolGetDatum(msg->transactional); + values[2] = CStringGetTextDatum(msg->prefix); + values[3] = Int32GetDatum((int32) msg->message_size); + values[4] = PointerGetDatum(cstring_to_text_with_len(msg->message, + msg->message_size)); + + SPI_connect(); + + /* + * The apply worker runs with an empty search_path, so the table name must + * be schema-qualified. + */ + ret = SPI_execute_with_args("INSERT INTO " LOG_SCHEMA "." LOG_TABLE + " (lsn, transactional, prefix, message_size, message)" + " VALUES ($1, $2, $3, $4, $5)", + 5, argtypes, values, NULL, false, 0); + if (ret != SPI_OK_INSERT) + elog(ERROR, "could not record logical message: SPI returned %d", ret); + + SPI_finish(); +} diff --git a/src/test/regress/expected/subscription.out b/src/test/regress/expected/subscription.out index 259db747334..83642fc03ee 100644 --- a/src/test/regress/expected/subscription.out +++ b/src/test/regress/expected/subscription.out @@ -143,18 +143,18 @@ CREATE SUBSCRIPTION regress_testsub4 CONNECTION 'dbname=regress_doesnotexist' PU WARNING: subscription was created, but is not connected HINT: To initiate replication, you must manually create the replication slot, enable the subscription, and alter the subscription to refresh publications. \dRs+ regress_testsub4 - List of subscriptions - Name | Owner | Enabled | Publication | Binary | Streaming | Two-phase commit | Disable on error | Origin | Password required | Run as owner? | Failover | Server | Retain dead tuples | Max retention duration | Retention active | Synchronous commit | Conninfo | Receiver timeout | Skip LSN | Description -------------------+---------------------------+---------+-------------+--------+-----------+------------------+------------------+--------+-------------------+---------------+----------+--------+--------------------+------------------------+------------------+--------------------+-----------------------------+------------------+------------+------------- - regress_testsub4 | regress_subscription_user | f | {testpub} | f | parallel | d | f | none | t | f | f | | f | 0 | f | off | dbname=regress_doesnotexist | -1 | 0/00000000 | + List of subscriptions + Name | Owner | Enabled | Publication | Binary | Streaming | Two-phase commit | Disable on error | Origin | Password required | Run as owner? | Failover | Server | Retain dead tuples | Max retention duration | Retention active | Message | Synchronous commit | Conninfo | Receiver timeout | Skip LSN | Description +------------------+---------------------------+---------+-------------+--------+-----------+------------------+------------------+--------+-------------------+---------------+----------+--------+--------------------+------------------------+------------------+---------+--------------------+-----------------------------+------------------+------------+------------- + regress_testsub4 | regress_subscription_user | f | {testpub} | f | parallel | d | f | none | t | f | f | | f | 0 | f | f | off | dbname=regress_doesnotexist | -1 | 0/00000000 | (1 row) ALTER SUBSCRIPTION regress_testsub4 SET (origin = any); \dRs+ regress_testsub4 - List of subscriptions - Name | Owner | Enabled | Publication | Binary | Streaming | Two-phase commit | Disable on error | Origin | Password required | Run as owner? | Failover | Server | Retain dead tuples | Max retention duration | Retention active | Synchronous commit | Conninfo | Receiver timeout | Skip LSN | Description -------------------+---------------------------+---------+-------------+--------+-----------+------------------+------------------+--------+-------------------+---------------+----------+--------+--------------------+------------------------+------------------+--------------------+-----------------------------+------------------+------------+------------- - regress_testsub4 | regress_subscription_user | f | {testpub} | f | parallel | d | f | any | t | f | f | | f | 0 | f | off | dbname=regress_doesnotexist | -1 | 0/00000000 | + List of subscriptions + Name | Owner | Enabled | Publication | Binary | Streaming | Two-phase commit | Disable on error | Origin | Password required | Run as owner? | Failover | Server | Retain dead tuples | Max retention duration | Retention active | Message | Synchronous commit | Conninfo | Receiver timeout | Skip LSN | Description +------------------+---------------------------+---------+-------------+--------+-----------+------------------+------------------+--------+-------------------+---------------+----------+--------+--------------------+------------------------+------------------+---------+--------------------+-----------------------------+------------------+------------+------------- + regress_testsub4 | regress_subscription_user | f | {testpub} | f | parallel | d | f | any | t | f | f | | f | 0 | f | f | off | dbname=regress_doesnotexist | -1 | 0/00000000 | (1 row) DROP SUBSCRIPTION regress_testsub3; @@ -258,10 +258,10 @@ ALTER SUBSCRIPTION regress_testsub CONNECTION 'foobar'; ERROR: invalid connection string syntax: missing "=" after "foobar" in connection info string \dRs+ - List of subscriptions - Name | Owner | Enabled | Publication | Binary | Streaming | Two-phase commit | Disable on error | Origin | Password required | Run as owner? | Failover | Server | Retain dead tuples | Max retention duration | Retention active | Synchronous commit | Conninfo | Receiver timeout | Skip LSN | Description ------------------+---------------------------+---------+-------------+--------+-----------+------------------+------------------+--------+-------------------+---------------+----------+--------+--------------------+------------------------+------------------+--------------------+-----------------------------+------------------+------------+------------------- - regress_testsub | regress_subscription_user | f | {testpub} | f | parallel | d | f | any | t | f | f | | f | 0 | f | off | dbname=regress_doesnotexist | -1 | 0/00000000 | test subscription + List of subscriptions + Name | Owner | Enabled | Publication | Binary | Streaming | Two-phase commit | Disable on error | Origin | Password required | Run as owner? | Failover | Server | Retain dead tuples | Max retention duration | Retention active | Message | Synchronous commit | Conninfo | Receiver timeout | Skip LSN | Description +-----------------+---------------------------+---------+-------------+--------+-----------+------------------+------------------+--------+-------------------+---------------+----------+--------+--------------------+------------------------+------------------+---------+--------------------+-----------------------------+------------------+------------+------------------- + regress_testsub | regress_subscription_user | f | {testpub} | f | parallel | d | f | any | t | f | f | | f | 0 | f | f | off | dbname=regress_doesnotexist | -1 | 0/00000000 | test subscription (1 row) ALTER SUBSCRIPTION regress_testsub SET PUBLICATION testpub2, testpub3 WITH (refresh = false); @@ -270,10 +270,10 @@ ALTER SUBSCRIPTION regress_testsub SET (slot_name = 'newname'); ALTER SUBSCRIPTION regress_testsub SET (password_required = false); ALTER SUBSCRIPTION regress_testsub SET (run_as_owner = true); \dRs+ - List of subscriptions - Name | Owner | Enabled | Publication | Binary | Streaming | Two-phase commit | Disable on error | Origin | Password required | Run as owner? | Failover | Server | Retain dead tuples | Max retention duration | Retention active | Synchronous commit | Conninfo | Receiver timeout | Skip LSN | Description ------------------+---------------------------+---------+---------------------+--------+-----------+------------------+------------------+--------+-------------------+---------------+----------+--------+--------------------+------------------------+------------------+--------------------+------------------------------+------------------+------------+------------------- - regress_testsub | regress_subscription_user | f | {testpub2,testpub3} | f | parallel | d | f | any | f | t | f | | f | 0 | f | off | dbname=regress_doesnotexist2 | -1 | 0/00000000 | test subscription + List of subscriptions + Name | Owner | Enabled | Publication | Binary | Streaming | Two-phase commit | Disable on error | Origin | Password required | Run as owner? | Failover | Server | Retain dead tuples | Max retention duration | Retention active | Message | Synchronous commit | Conninfo | Receiver timeout | Skip LSN | Description +-----------------+---------------------------+---------+---------------------+--------+-----------+------------------+------------------+--------+-------------------+---------------+----------+--------+--------------------+------------------------+------------------+---------+--------------------+------------------------------+------------------+------------+------------------- + regress_testsub | regress_subscription_user | f | {testpub2,testpub3} | f | parallel | d | f | any | f | t | f | | f | 0 | f | f | off | dbname=regress_doesnotexist2 | -1 | 0/00000000 | test subscription (1 row) ALTER SUBSCRIPTION regress_testsub SET (password_required = true); @@ -289,10 +289,10 @@ ERROR: unrecognized subscription parameter: "create_slot" -- ok ALTER SUBSCRIPTION regress_testsub SKIP (lsn = '0/12345'); \dRs+ - List of subscriptions - Name | Owner | Enabled | Publication | Binary | Streaming | Two-phase commit | Disable on error | Origin | Password required | Run as owner? | Failover | Server | Retain dead tuples | Max retention duration | Retention active | Synchronous commit | Conninfo | Receiver timeout | Skip LSN | Description ------------------+---------------------------+---------+---------------------+--------+-----------+------------------+------------------+--------+-------------------+---------------+----------+--------+--------------------+------------------------+------------------+--------------------+------------------------------+------------------+------------+------------------- - regress_testsub | regress_subscription_user | f | {testpub2,testpub3} | f | parallel | d | f | any | t | f | f | | f | 0 | f | off | dbname=regress_doesnotexist2 | -1 | 0/00012345 | test subscription + List of subscriptions + Name | Owner | Enabled | Publication | Binary | Streaming | Two-phase commit | Disable on error | Origin | Password required | Run as owner? | Failover | Server | Retain dead tuples | Max retention duration | Retention active | Message | Synchronous commit | Conninfo | Receiver timeout | Skip LSN | Description +-----------------+---------------------------+---------+---------------------+--------+-----------+------------------+------------------+--------+-------------------+---------------+----------+--------+--------------------+------------------------+------------------+---------+--------------------+------------------------------+------------------+------------+------------------- + regress_testsub | regress_subscription_user | f | {testpub2,testpub3} | f | parallel | d | f | any | t | f | f | | f | 0 | f | f | off | dbname=regress_doesnotexist2 | -1 | 0/00012345 | test subscription (1 row) -- ok - with lsn = NONE @@ -301,10 +301,10 @@ ALTER SUBSCRIPTION regress_testsub SKIP (lsn = NONE); ALTER SUBSCRIPTION regress_testsub SKIP (lsn = '0/0'); ERROR: invalid WAL location (LSN): 0/0 \dRs+ - List of subscriptions - Name | Owner | Enabled | Publication | Binary | Streaming | Two-phase commit | Disable on error | Origin | Password required | Run as owner? | Failover | Server | Retain dead tuples | Max retention duration | Retention active | Synchronous commit | Conninfo | Receiver timeout | Skip LSN | Description ------------------+---------------------------+---------+---------------------+--------+-----------+------------------+------------------+--------+-------------------+---------------+----------+--------+--------------------+------------------------+------------------+--------------------+------------------------------+------------------+------------+------------------- - regress_testsub | regress_subscription_user | f | {testpub2,testpub3} | f | parallel | d | f | any | t | f | f | | f | 0 | f | off | dbname=regress_doesnotexist2 | -1 | 0/00000000 | test subscription + List of subscriptions + Name | Owner | Enabled | Publication | Binary | Streaming | Two-phase commit | Disable on error | Origin | Password required | Run as owner? | Failover | Server | Retain dead tuples | Max retention duration | Retention active | Message | Synchronous commit | Conninfo | Receiver timeout | Skip LSN | Description +-----------------+---------------------------+---------+---------------------+--------+-----------+------------------+------------------+--------+-------------------+---------------+----------+--------+--------------------+------------------------+------------------+---------+--------------------+------------------------------+------------------+------------+------------------- + regress_testsub | regress_subscription_user | f | {testpub2,testpub3} | f | parallel | d | f | any | t | f | f | | f | 0 | f | f | off | dbname=regress_doesnotexist2 | -1 | 0/00000000 | test subscription (1 row) BEGIN; @@ -340,10 +340,10 @@ ALTER SUBSCRIPTION regress_testsub_foo SET (wal_receiver_timeout = '80s'); ALTER SUBSCRIPTION regress_testsub_foo SET (wal_receiver_timeout = 'foobar'); ERROR: invalid value for parameter "wal_receiver_timeout": "foobar" \dRs+ - List of subscriptions - Name | Owner | Enabled | Publication | Binary | Streaming | Two-phase commit | Disable on error | Origin | Password required | Run as owner? | Failover | Server | Retain dead tuples | Max retention duration | Retention active | Synchronous commit | Conninfo | Receiver timeout | Skip LSN | Description ----------------------+---------------------------+---------+---------------------+--------+-----------+------------------+------------------+--------+-------------------+---------------+----------+--------+--------------------+------------------------+------------------+--------------------+------------------------------+------------------+------------+------------------- - regress_testsub_foo | regress_subscription_user | f | {testpub2,testpub3} | f | parallel | d | f | any | t | f | f | | f | 0 | f | local | dbname=regress_doesnotexist2 | 80s | 0/00000000 | test subscription + List of subscriptions + Name | Owner | Enabled | Publication | Binary | Streaming | Two-phase commit | Disable on error | Origin | Password required | Run as owner? | Failover | Server | Retain dead tuples | Max retention duration | Retention active | Message | Synchronous commit | Conninfo | Receiver timeout | Skip LSN | Description +---------------------+---------------------------+---------+---------------------+--------+-----------+------------------+------------------+--------+-------------------+---------------+----------+--------+--------------------+------------------------+------------------+---------+--------------------+------------------------------+------------------+------------+------------------- + regress_testsub_foo | regress_subscription_user | f | {testpub2,testpub3} | f | parallel | d | f | any | t | f | f | | f | 0 | f | f | local | dbname=regress_doesnotexist2 | 80s | 0/00000000 | test subscription (1 row) -- rename back to keep the rest simple @@ -372,19 +372,19 @@ CREATE SUBSCRIPTION regress_testsub CONNECTION 'dbname=regress_doesnotexist' PUB WARNING: subscription was created, but is not connected HINT: To initiate replication, you must manually create the replication slot, enable the subscription, and alter the subscription to refresh publications. \dRs+ - List of subscriptions - Name | Owner | Enabled | Publication | Binary | Streaming | Two-phase commit | Disable on error | Origin | Password required | Run as owner? | Failover | Server | Retain dead tuples | Max retention duration | Retention active | Synchronous commit | Conninfo | Receiver timeout | Skip LSN | Description ------------------+---------------------------+---------+-------------+--------+-----------+------------------+------------------+--------+-------------------+---------------+----------+--------+--------------------+------------------------+------------------+--------------------+-----------------------------+------------------+------------+------------- - regress_testsub | regress_subscription_user | f | {testpub} | t | parallel | d | f | any | t | f | f | | f | 0 | f | off | dbname=regress_doesnotexist | -1 | 0/00000000 | + List of subscriptions + Name | Owner | Enabled | Publication | Binary | Streaming | Two-phase commit | Disable on error | Origin | Password required | Run as owner? | Failover | Server | Retain dead tuples | Max retention duration | Retention active | Message | Synchronous commit | Conninfo | Receiver timeout | Skip LSN | Description +-----------------+---------------------------+---------+-------------+--------+-----------+------------------+------------------+--------+-------------------+---------------+----------+--------+--------------------+------------------------+------------------+---------+--------------------+-----------------------------+------------------+------------+------------- + regress_testsub | regress_subscription_user | f | {testpub} | t | parallel | d | f | any | t | f | f | | f | 0 | f | f | off | dbname=regress_doesnotexist | -1 | 0/00000000 | (1 row) ALTER SUBSCRIPTION regress_testsub SET (binary = false); ALTER SUBSCRIPTION regress_testsub SET (slot_name = NONE); \dRs+ - List of subscriptions - Name | Owner | Enabled | Publication | Binary | Streaming | Two-phase commit | Disable on error | Origin | Password required | Run as owner? | Failover | Server | Retain dead tuples | Max retention duration | Retention active | Synchronous commit | Conninfo | Receiver timeout | Skip LSN | Description ------------------+---------------------------+---------+-------------+--------+-----------+------------------+------------------+--------+-------------------+---------------+----------+--------+--------------------+------------------------+------------------+--------------------+-----------------------------+------------------+------------+------------- - regress_testsub | regress_subscription_user | f | {testpub} | f | parallel | d | f | any | t | f | f | | f | 0 | f | off | dbname=regress_doesnotexist | -1 | 0/00000000 | + List of subscriptions + Name | Owner | Enabled | Publication | Binary | Streaming | Two-phase commit | Disable on error | Origin | Password required | Run as owner? | Failover | Server | Retain dead tuples | Max retention duration | Retention active | Message | Synchronous commit | Conninfo | Receiver timeout | Skip LSN | Description +-----------------+---------------------------+---------+-------------+--------+-----------+------------------+------------------+--------+-------------------+---------------+----------+--------+--------------------+------------------------+------------------+---------+--------------------+-----------------------------+------------------+------------+------------- + regress_testsub | regress_subscription_user | f | {testpub} | f | parallel | d | f | any | t | f | f | | f | 0 | f | f | off | dbname=regress_doesnotexist | -1 | 0/00000000 | (1 row) DROP SUBSCRIPTION regress_testsub; @@ -396,27 +396,27 @@ CREATE SUBSCRIPTION regress_testsub CONNECTION 'dbname=regress_doesnotexist' PUB WARNING: subscription was created, but is not connected HINT: To initiate replication, you must manually create the replication slot, enable the subscription, and alter the subscription to refresh publications. \dRs+ - List of subscriptions - Name | Owner | Enabled | Publication | Binary | Streaming | Two-phase commit | Disable on error | Origin | Password required | Run as owner? | Failover | Server | Retain dead tuples | Max retention duration | Retention active | Synchronous commit | Conninfo | Receiver timeout | Skip LSN | Description ------------------+---------------------------+---------+-------------+--------+-----------+------------------+------------------+--------+-------------------+---------------+----------+--------+--------------------+------------------------+------------------+--------------------+-----------------------------+------------------+------------+------------- - regress_testsub | regress_subscription_user | f | {testpub} | f | on | d | f | any | t | f | f | | f | 0 | f | off | dbname=regress_doesnotexist | -1 | 0/00000000 | + List of subscriptions + Name | Owner | Enabled | Publication | Binary | Streaming | Two-phase commit | Disable on error | Origin | Password required | Run as owner? | Failover | Server | Retain dead tuples | Max retention duration | Retention active | Message | Synchronous commit | Conninfo | Receiver timeout | Skip LSN | Description +-----------------+---------------------------+---------+-------------+--------+-----------+------------------+------------------+--------+-------------------+---------------+----------+--------+--------------------+------------------------+------------------+---------+--------------------+-----------------------------+------------------+------------+------------- + regress_testsub | regress_subscription_user | f | {testpub} | f | on | d | f | any | t | f | f | | f | 0 | f | f | off | dbname=regress_doesnotexist | -1 | 0/00000000 | (1 row) ALTER SUBSCRIPTION regress_testsub SET (streaming = parallel); \dRs+ - List of subscriptions - Name | Owner | Enabled | Publication | Binary | Streaming | Two-phase commit | Disable on error | Origin | Password required | Run as owner? | Failover | Server | Retain dead tuples | Max retention duration | Retention active | Synchronous commit | Conninfo | Receiver timeout | Skip LSN | Description ------------------+---------------------------+---------+-------------+--------+-----------+------------------+------------------+--------+-------------------+---------------+----------+--------+--------------------+------------------------+------------------+--------------------+-----------------------------+------------------+------------+------------- - regress_testsub | regress_subscription_user | f | {testpub} | f | parallel | d | f | any | t | f | f | | f | 0 | f | off | dbname=regress_doesnotexist | -1 | 0/00000000 | + List of subscriptions + Name | Owner | Enabled | Publication | Binary | Streaming | Two-phase commit | Disable on error | Origin | Password required | Run as owner? | Failover | Server | Retain dead tuples | Max retention duration | Retention active | Message | Synchronous commit | Conninfo | Receiver timeout | Skip LSN | Description +-----------------+---------------------------+---------+-------------+--------+-----------+------------------+------------------+--------+-------------------+---------------+----------+--------+--------------------+------------------------+------------------+---------+--------------------+-----------------------------+------------------+------------+------------- + regress_testsub | regress_subscription_user | f | {testpub} | f | parallel | d | f | any | t | f | f | | f | 0 | f | f | off | dbname=regress_doesnotexist | -1 | 0/00000000 | (1 row) ALTER SUBSCRIPTION regress_testsub SET (streaming = false); ALTER SUBSCRIPTION regress_testsub SET (slot_name = NONE); \dRs+ - List of subscriptions - Name | Owner | Enabled | Publication | Binary | Streaming | Two-phase commit | Disable on error | Origin | Password required | Run as owner? | Failover | Server | Retain dead tuples | Max retention duration | Retention active | Synchronous commit | Conninfo | Receiver timeout | Skip LSN | Description ------------------+---------------------------+---------+-------------+--------+-----------+------------------+------------------+--------+-------------------+---------------+----------+--------+--------------------+------------------------+------------------+--------------------+-----------------------------+------------------+------------+------------- - regress_testsub | regress_subscription_user | f | {testpub} | f | off | d | f | any | t | f | f | | f | 0 | f | off | dbname=regress_doesnotexist | -1 | 0/00000000 | + List of subscriptions + Name | Owner | Enabled | Publication | Binary | Streaming | Two-phase commit | Disable on error | Origin | Password required | Run as owner? | Failover | Server | Retain dead tuples | Max retention duration | Retention active | Message | Synchronous commit | Conninfo | Receiver timeout | Skip LSN | Description +-----------------+---------------------------+---------+-------------+--------+-----------+------------------+------------------+--------+-------------------+---------------+----------+--------+--------------------+------------------------+------------------+---------+--------------------+-----------------------------+------------------+------------+------------- + regress_testsub | regress_subscription_user | f | {testpub} | f | off | d | f | any | t | f | f | | f | 0 | f | f | off | dbname=regress_doesnotexist | -1 | 0/00000000 | (1 row) -- fail - publication already exists @@ -431,10 +431,10 @@ ALTER SUBSCRIPTION regress_testsub ADD PUBLICATION testpub1, testpub2 WITH (refr ALTER SUBSCRIPTION regress_testsub ADD PUBLICATION testpub1, testpub2 WITH (refresh = false); ERROR: publication "testpub1" is already in subscription "regress_testsub" \dRs+ - List of subscriptions - Name | Owner | Enabled | Publication | Binary | Streaming | Two-phase commit | Disable on error | Origin | Password required | Run as owner? | Failover | Server | Retain dead tuples | Max retention duration | Retention active | Synchronous commit | Conninfo | Receiver timeout | Skip LSN | Description ------------------+---------------------------+---------+-----------------------------+--------+-----------+------------------+------------------+--------+-------------------+---------------+----------+--------+--------------------+------------------------+------------------+--------------------+-----------------------------+------------------+------------+------------- - regress_testsub | regress_subscription_user | f | {testpub,testpub1,testpub2} | f | off | d | f | any | t | f | f | | f | 0 | f | off | dbname=regress_doesnotexist | -1 | 0/00000000 | + List of subscriptions + Name | Owner | Enabled | Publication | Binary | Streaming | Two-phase commit | Disable on error | Origin | Password required | Run as owner? | Failover | Server | Retain dead tuples | Max retention duration | Retention active | Message | Synchronous commit | Conninfo | Receiver timeout | Skip LSN | Description +-----------------+---------------------------+---------+-----------------------------+--------+-----------+------------------+------------------+--------+-------------------+---------------+----------+--------+--------------------+------------------------+------------------+---------+--------------------+-----------------------------+------------------+------------+------------- + regress_testsub | regress_subscription_user | f | {testpub,testpub1,testpub2} | f | off | d | f | any | t | f | f | | f | 0 | f | f | off | dbname=regress_doesnotexist | -1 | 0/00000000 | (1 row) -- fail - publication used more than once @@ -449,10 +449,10 @@ ERROR: publication "testpub3" is not in subscription "regress_testsub" -- ok - delete publications ALTER SUBSCRIPTION regress_testsub DROP PUBLICATION testpub1, testpub2 WITH (refresh = false); \dRs+ - List of subscriptions - Name | Owner | Enabled | Publication | Binary | Streaming | Two-phase commit | Disable on error | Origin | Password required | Run as owner? | Failover | Server | Retain dead tuples | Max retention duration | Retention active | Synchronous commit | Conninfo | Receiver timeout | Skip LSN | Description ------------------+---------------------------+---------+-------------+--------+-----------+------------------+------------------+--------+-------------------+---------------+----------+--------+--------------------+------------------------+------------------+--------------------+-----------------------------+------------------+------------+------------- - regress_testsub | regress_subscription_user | f | {testpub} | f | off | d | f | any | t | f | f | | f | 0 | f | off | dbname=regress_doesnotexist | -1 | 0/00000000 | + List of subscriptions + Name | Owner | Enabled | Publication | Binary | Streaming | Two-phase commit | Disable on error | Origin | Password required | Run as owner? | Failover | Server | Retain dead tuples | Max retention duration | Retention active | Message | Synchronous commit | Conninfo | Receiver timeout | Skip LSN | Description +-----------------+---------------------------+---------+-------------+--------+-----------+------------------+------------------+--------+-------------------+---------------+----------+--------+--------------------+------------------------+------------------+---------+--------------------+-----------------------------+------------------+------------+------------- + regress_testsub | regress_subscription_user | f | {testpub} | f | off | d | f | any | t | f | f | | f | 0 | f | f | off | dbname=regress_doesnotexist | -1 | 0/00000000 | (1 row) DROP SUBSCRIPTION regress_testsub; @@ -488,19 +488,19 @@ CREATE SUBSCRIPTION regress_testsub CONNECTION 'dbname=regress_doesnotexist' PUB WARNING: subscription was created, but is not connected HINT: To initiate replication, you must manually create the replication slot, enable the subscription, and alter the subscription to refresh publications. \dRs+ - List of subscriptions - Name | Owner | Enabled | Publication | Binary | Streaming | Two-phase commit | Disable on error | Origin | Password required | Run as owner? | Failover | Server | Retain dead tuples | Max retention duration | Retention active | Synchronous commit | Conninfo | Receiver timeout | Skip LSN | Description ------------------+---------------------------+---------+-------------+--------+-----------+------------------+------------------+--------+-------------------+---------------+----------+--------+--------------------+------------------------+------------------+--------------------+-----------------------------+------------------+------------+------------- - regress_testsub | regress_subscription_user | f | {testpub} | f | parallel | p | f | any | t | f | f | | f | 0 | f | off | dbname=regress_doesnotexist | -1 | 0/00000000 | + List of subscriptions + Name | Owner | Enabled | Publication | Binary | Streaming | Two-phase commit | Disable on error | Origin | Password required | Run as owner? | Failover | Server | Retain dead tuples | Max retention duration | Retention active | Message | Synchronous commit | Conninfo | Receiver timeout | Skip LSN | Description +-----------------+---------------------------+---------+-------------+--------+-----------+------------------+------------------+--------+-------------------+---------------+----------+--------+--------------------+------------------------+------------------+---------+--------------------+-----------------------------+------------------+------------+------------- + regress_testsub | regress_subscription_user | f | {testpub} | f | parallel | p | f | any | t | f | f | | f | 0 | f | f | off | dbname=regress_doesnotexist | -1 | 0/00000000 | (1 row) -- we can alter streaming when two_phase enabled ALTER SUBSCRIPTION regress_testsub SET (streaming = true); \dRs+ - List of subscriptions - Name | Owner | Enabled | Publication | Binary | Streaming | Two-phase commit | Disable on error | Origin | Password required | Run as owner? | Failover | Server | Retain dead tuples | Max retention duration | Retention active | Synchronous commit | Conninfo | Receiver timeout | Skip LSN | Description ------------------+---------------------------+---------+-------------+--------+-----------+------------------+------------------+--------+-------------------+---------------+----------+--------+--------------------+------------------------+------------------+--------------------+-----------------------------+------------------+------------+------------- - regress_testsub | regress_subscription_user | f | {testpub} | f | on | p | f | any | t | f | f | | f | 0 | f | off | dbname=regress_doesnotexist | -1 | 0/00000000 | + List of subscriptions + Name | Owner | Enabled | Publication | Binary | Streaming | Two-phase commit | Disable on error | Origin | Password required | Run as owner? | Failover | Server | Retain dead tuples | Max retention duration | Retention active | Message | Synchronous commit | Conninfo | Receiver timeout | Skip LSN | Description +-----------------+---------------------------+---------+-------------+--------+-----------+------------------+------------------+--------+-------------------+---------------+----------+--------+--------------------+------------------------+------------------+---------+--------------------+-----------------------------+------------------+------------+------------- + regress_testsub | regress_subscription_user | f | {testpub} | f | on | p | f | any | t | f | f | | f | 0 | f | f | off | dbname=regress_doesnotexist | -1 | 0/00000000 | (1 row) ALTER SUBSCRIPTION regress_testsub SET (slot_name = NONE); @@ -510,10 +510,10 @@ CREATE SUBSCRIPTION regress_testsub CONNECTION 'dbname=regress_doesnotexist' PUB WARNING: subscription was created, but is not connected HINT: To initiate replication, you must manually create the replication slot, enable the subscription, and alter the subscription to refresh publications. \dRs+ - List of subscriptions - Name | Owner | Enabled | Publication | Binary | Streaming | Two-phase commit | Disable on error | Origin | Password required | Run as owner? | Failover | Server | Retain dead tuples | Max retention duration | Retention active | Synchronous commit | Conninfo | Receiver timeout | Skip LSN | Description ------------------+---------------------------+---------+-------------+--------+-----------+------------------+------------------+--------+-------------------+---------------+----------+--------+--------------------+------------------------+------------------+--------------------+-----------------------------+------------------+------------+------------- - regress_testsub | regress_subscription_user | f | {testpub} | f | on | p | f | any | t | f | f | | f | 0 | f | off | dbname=regress_doesnotexist | -1 | 0/00000000 | + List of subscriptions + Name | Owner | Enabled | Publication | Binary | Streaming | Two-phase commit | Disable on error | Origin | Password required | Run as owner? | Failover | Server | Retain dead tuples | Max retention duration | Retention active | Message | Synchronous commit | Conninfo | Receiver timeout | Skip LSN | Description +-----------------+---------------------------+---------+-------------+--------+-----------+------------------+------------------+--------+-------------------+---------------+----------+--------+--------------------+------------------------+------------------+---------+--------------------+-----------------------------+------------------+------------+------------- + regress_testsub | regress_subscription_user | f | {testpub} | f | on | p | f | any | t | f | f | | f | 0 | f | f | off | dbname=regress_doesnotexist | -1 | 0/00000000 | (1 row) ALTER SUBSCRIPTION regress_testsub SET (slot_name = NONE); @@ -526,18 +526,18 @@ CREATE SUBSCRIPTION regress_testsub CONNECTION 'dbname=regress_doesnotexist' PUB WARNING: subscription was created, but is not connected HINT: To initiate replication, you must manually create the replication slot, enable the subscription, and alter the subscription to refresh publications. \dRs+ - List of subscriptions - Name | Owner | Enabled | Publication | Binary | Streaming | Two-phase commit | Disable on error | Origin | Password required | Run as owner? | Failover | Server | Retain dead tuples | Max retention duration | Retention active | Synchronous commit | Conninfo | Receiver timeout | Skip LSN | Description ------------------+---------------------------+---------+-------------+--------+-----------+------------------+------------------+--------+-------------------+---------------+----------+--------+--------------------+------------------------+------------------+--------------------+-----------------------------+------------------+------------+------------- - regress_testsub | regress_subscription_user | f | {testpub} | f | parallel | d | f | any | t | f | f | | f | 0 | f | off | dbname=regress_doesnotexist | -1 | 0/00000000 | + List of subscriptions + Name | Owner | Enabled | Publication | Binary | Streaming | Two-phase commit | Disable on error | Origin | Password required | Run as owner? | Failover | Server | Retain dead tuples | Max retention duration | Retention active | Message | Synchronous commit | Conninfo | Receiver timeout | Skip LSN | Description +-----------------+---------------------------+---------+-------------+--------+-----------+------------------+------------------+--------+-------------------+---------------+----------+--------+--------------------+------------------------+------------------+---------+--------------------+-----------------------------+------------------+------------+------------- + regress_testsub | regress_subscription_user | f | {testpub} | f | parallel | d | f | any | t | f | f | | f | 0 | f | f | off | dbname=regress_doesnotexist | -1 | 0/00000000 | (1 row) ALTER SUBSCRIPTION regress_testsub SET (disable_on_error = true); \dRs+ - List of subscriptions - Name | Owner | Enabled | Publication | Binary | Streaming | Two-phase commit | Disable on error | Origin | Password required | Run as owner? | Failover | Server | Retain dead tuples | Max retention duration | Retention active | Synchronous commit | Conninfo | Receiver timeout | Skip LSN | Description ------------------+---------------------------+---------+-------------+--------+-----------+------------------+------------------+--------+-------------------+---------------+----------+--------+--------------------+------------------------+------------------+--------------------+-----------------------------+------------------+------------+------------- - regress_testsub | regress_subscription_user | f | {testpub} | f | parallel | d | t | any | t | f | f | | f | 0 | f | off | dbname=regress_doesnotexist | -1 | 0/00000000 | + List of subscriptions + Name | Owner | Enabled | Publication | Binary | Streaming | Two-phase commit | Disable on error | Origin | Password required | Run as owner? | Failover | Server | Retain dead tuples | Max retention duration | Retention active | Message | Synchronous commit | Conninfo | Receiver timeout | Skip LSN | Description +-----------------+---------------------------+---------+-------------+--------+-----------+------------------+------------------+--------+-------------------+---------------+----------+--------+--------------------+------------------------+------------------+---------+--------------------+-----------------------------+------------------+------------+------------- + regress_testsub | regress_subscription_user | f | {testpub} | f | parallel | d | t | any | t | f | f | | f | 0 | f | f | off | dbname=regress_doesnotexist | -1 | 0/00000000 | (1 row) ALTER SUBSCRIPTION regress_testsub SET (slot_name = NONE); @@ -550,10 +550,10 @@ CREATE SUBSCRIPTION regress_testsub CONNECTION 'dbname=regress_doesnotexist' PUB WARNING: subscription was created, but is not connected HINT: To initiate replication, you must manually create the replication slot, enable the subscription, and alter the subscription to refresh publications. \dRs+ - List of subscriptions - Name | Owner | Enabled | Publication | Binary | Streaming | Two-phase commit | Disable on error | Origin | Password required | Run as owner? | Failover | Server | Retain dead tuples | Max retention duration | Retention active | Synchronous commit | Conninfo | Receiver timeout | Skip LSN | Description ------------------+---------------------------+---------+-------------+--------+-----------+------------------+------------------+--------+-------------------+---------------+----------+--------+--------------------+------------------------+------------------+--------------------+-----------------------------+------------------+------------+------------- - regress_testsub | regress_subscription_user | f | {testpub} | f | parallel | d | f | any | t | f | f | | f | 0 | f | off | dbname=regress_doesnotexist | -1 | 0/00000000 | + List of subscriptions + Name | Owner | Enabled | Publication | Binary | Streaming | Two-phase commit | Disable on error | Origin | Password required | Run as owner? | Failover | Server | Retain dead tuples | Max retention duration | Retention active | Message | Synchronous commit | Conninfo | Receiver timeout | Skip LSN | Description +-----------------+---------------------------+---------+-------------+--------+-----------+------------------+------------------+--------+-------------------+---------------+----------+--------+--------------------+------------------------+------------------+---------+--------------------+-----------------------------+------------------+------------+------------- + regress_testsub | regress_subscription_user | f | {testpub} | f | parallel | d | f | any | t | f | f | | f | 0 | f | f | off | dbname=regress_doesnotexist | -1 | 0/00000000 | (1 row) ALTER SUBSCRIPTION regress_testsub SET (slot_name = NONE); @@ -570,10 +570,10 @@ NOTICE: max_retention_duration is ineffective when retain_dead_tuples is disabl WARNING: subscription was created, but is not connected HINT: To initiate replication, you must manually create the replication slot, enable the subscription, and alter the subscription to refresh publications. \dRs+ - List of subscriptions - Name | Owner | Enabled | Publication | Binary | Streaming | Two-phase commit | Disable on error | Origin | Password required | Run as owner? | Failover | Server | Retain dead tuples | Max retention duration | Retention active | Synchronous commit | Conninfo | Receiver timeout | Skip LSN | Description ------------------+---------------------------+---------+-------------+--------+-----------+------------------+------------------+--------+-------------------+---------------+----------+--------+--------------------+------------------------+------------------+--------------------+-----------------------------+------------------+------------+------------- - regress_testsub | regress_subscription_user | f | {testpub} | f | parallel | d | f | any | t | f | f | | f | 1000 | f | off | dbname=regress_doesnotexist | -1 | 0/00000000 | + List of subscriptions + Name | Owner | Enabled | Publication | Binary | Streaming | Two-phase commit | Disable on error | Origin | Password required | Run as owner? | Failover | Server | Retain dead tuples | Max retention duration | Retention active | Message | Synchronous commit | Conninfo | Receiver timeout | Skip LSN | Description +-----------------+---------------------------+---------+-------------+--------+-----------+------------------+------------------+--------+-------------------+---------------+----------+--------+--------------------+------------------------+------------------+---------+--------------------+-----------------------------+------------------+------------+------------- + regress_testsub | regress_subscription_user | f | {testpub} | f | parallel | d | f | any | t | f | f | | f | 1000 | f | f | off | dbname=regress_doesnotexist | -1 | 0/00000000 | (1 row) -- fail - max_retention_duration must be non-negative @@ -582,12 +582,38 @@ ERROR: max_retention_duration cannot be negative -- ok ALTER SUBSCRIPTION regress_testsub SET (max_retention_duration = 0); \dRs+ - List of subscriptions - Name | Owner | Enabled | Publication | Binary | Streaming | Two-phase commit | Disable on error | Origin | Password required | Run as owner? | Failover | Server | Retain dead tuples | Max retention duration | Retention active | Synchronous commit | Conninfo | Receiver timeout | Skip LSN | Description ------------------+---------------------------+---------+-------------+--------+-----------+------------------+------------------+--------+-------------------+---------------+----------+--------+--------------------+------------------------+------------------+--------------------+-----------------------------+------------------+------------+------------- - regress_testsub | regress_subscription_user | f | {testpub} | f | parallel | d | f | any | t | f | f | | f | 0 | f | off | dbname=regress_doesnotexist | -1 | 0/00000000 | + List of subscriptions + Name | Owner | Enabled | Publication | Binary | Streaming | Two-phase commit | Disable on error | Origin | Password required | Run as owner? | Failover | Server | Retain dead tuples | Max retention duration | Retention active | Message | Synchronous commit | Conninfo | Receiver timeout | Skip LSN | Description +-----------------+---------------------------+---------+-------------+--------+-----------+------------------+------------------+--------+-------------------+---------------+----------+--------+--------------------+------------------------+------------------+---------+--------------------+-----------------------------+------------------+------------+------------- + regress_testsub | regress_subscription_user | f | {testpub} | f | parallel | d | f | any | t | f | f | | f | 0 | f | f | off | dbname=regress_doesnotexist | -1 | 0/00000000 | +(1 row) + +ALTER SUBSCRIPTION regress_testsub SET (slot_name = NONE); +DROP SUBSCRIPTION regress_testsub; +-- fail - message must be boolean +CREATE SUBSCRIPTION regress_testsub CONNECTION 'dbname=regress_doesnotexist' PUBLICATION testpub WITH (connect = false, message = foo); +ERROR: message requires a Boolean value +-- ok +CREATE SUBSCRIPTION regress_testsub CONNECTION 'dbname=regress_doesnotexist' PUBLICATION testpub WITH (connect = false, message = true); +WARNING: subscription was created, but is not connected +HINT: To initiate replication, you must manually create the replication slot, enable the subscription, and alter the subscription to refresh publications. +\dRs+ + List of subscriptions + Name | Owner | Enabled | Publication | Binary | Streaming | Two-phase commit | Disable on error | Origin | Password required | Run as owner? | Failover | Server | Retain dead tuples | Max retention duration | Retention active | Message | Synchronous commit | Conninfo | Receiver timeout | Skip LSN | Description +-----------------+---------------------------+---------+-------------+--------+-----------+------------------+------------------+--------+-------------------+---------------+----------+--------+--------------------+------------------------+------------------+---------+--------------------+-----------------------------+------------------+------------+------------- + regress_testsub | regress_subscription_user | f | {testpub} | f | parallel | d | f | any | t | f | f | | f | 0 | f | t | off | dbname=regress_doesnotexist | -1 | 0/00000000 | +(1 row) + +-- ok +ALTER SUBSCRIPTION regress_testsub SET (message = false); +\dRs+ + List of subscriptions + Name | Owner | Enabled | Publication | Binary | Streaming | Two-phase commit | Disable on error | Origin | Password required | Run as owner? | Failover | Server | Retain dead tuples | Max retention duration | Retention active | Message | Synchronous commit | Conninfo | Receiver timeout | Skip LSN | Description +-----------------+---------------------------+---------+-------------+--------+-----------+------------------+------------------+--------+-------------------+---------------+----------+--------+--------------------+------------------------+------------------+---------+--------------------+-----------------------------+------------------+------------+------------- + regress_testsub | regress_subscription_user | f | {testpub} | f | parallel | d | f | any | t | f | f | | f | 0 | f | f | off | dbname=regress_doesnotexist | -1 | 0/00000000 | (1 row) +-- cleanup ALTER SUBSCRIPTION regress_testsub SET (slot_name = NONE); DROP SUBSCRIPTION regress_testsub; -- let's do some tests with pg_create_subscription rather than superuser diff --git a/src/test/regress/sql/subscription.sql b/src/test/regress/sql/subscription.sql index 7718c742974..55682f255d1 100644 --- a/src/test/regress/sql/subscription.sql +++ b/src/test/regress/sql/subscription.sql @@ -441,6 +441,23 @@ ALTER SUBSCRIPTION regress_testsub SET (max_retention_duration = 0); ALTER SUBSCRIPTION regress_testsub SET (slot_name = NONE); DROP SUBSCRIPTION regress_testsub; +-- fail - message must be boolean +CREATE SUBSCRIPTION regress_testsub CONNECTION 'dbname=regress_doesnotexist' PUBLICATION testpub WITH (connect = false, message = foo); + +-- ok +CREATE SUBSCRIPTION regress_testsub CONNECTION 'dbname=regress_doesnotexist' PUBLICATION testpub WITH (connect = false, message = true); + +\dRs+ + +-- ok +ALTER SUBSCRIPTION regress_testsub SET (message = false); + +\dRs+ + +-- cleanup +ALTER SUBSCRIPTION regress_testsub SET (slot_name = NONE); +DROP SUBSCRIPTION regress_testsub; + -- let's do some tests with pg_create_subscription rather than superuser SET SESSION AUTHORIZATION regress_subscription_user3; diff --git a/src/tools/pgindent/typedefs.list b/src/tools/pgindent/typedefs.list index 85d989f395d..6b452f8c267 100644 --- a/src/tools/pgindent/typedefs.list +++ b/src/tools/pgindent/typedefs.list @@ -1707,6 +1707,8 @@ LogicalRepBeginData LogicalRepCommitData LogicalRepCommitPreparedTxnData LogicalRepCtxStruct +LogicalRepMessageData +LogicalRepMessageHandle_hook_type LogicalRepMsgType LogicalRepPartMapEntry LogicalRepPreparedTxnData -- 2.55.0
^ permalink raw reply [nested|flat] 44+ messages in thread

* Re: Add a hook for handling logical decoding messages on subscribers.
@ 2026-08-06 04:44  Chao Li <li.evan.chao@gmail.com>
  parent: Masahiko Sawada <sawada.mshk@gmail.com>
  1 sibling, 0 replies; 44+ messages in thread

From: Chao Li @ 2026-08-06 04:44 UTC (permalink / raw)
  To: Masahiko Sawada <sawada.mshk@gmail.com>; +Cc: Fujii Masao <masao.fujii@gmail.com>; Amit Kapila <amit.kapila16@gmail.com>; Bharath Rupireddy <bharath.rupireddyforpostgres@gmail.com>; pgsql-hackers



> On Aug 6, 2026, at 02:55, Masahiko Sawada <sawada.mshk@gmail.com> wrote:
> 
> On Wed, Aug 5, 2026 at 9:32 AM Masahiko Sawada <sawada.mshk@gmail.com> wrote:
>> 
>> On Tue, Aug 4, 2026 at 1:05 AM Chao Li <li.evan.chao@gmail.com> wrote:
>>> 
>>> 
>>> 
>>>> On Aug 4, 2026, at 08:41, Masahiko Sawada <sawada.mshk@gmail.com> wrote:
>>>> 
>>>> On Thu, Jul 9, 2026 at 5:14 AM Fujii Masao <masao.fujii@gmail.com> wrote:
>>>>> 
>>>>> On Wed, Jun 24, 2026 at 3:02 AM Masahiko Sawada <sawada.mshk@gmail.com> wrote:
>>>>>> Right. I'm implementing a basic DDL replication solution using this
>>>>>> hook as a sample implementation. Other than that, this feature can be
>>>>>> used to add additional information for the replicated transaction that
>>>>>> is not replicated via logical replication protocol, such as the
>>>>>> executed user or context-specific information, which can then be
>>>>>> dispatched to a CDC system connected to the subscriber.
>>>>> 
>>>>> +1 for the proposed hook, although I haven't read the patch yet.
>>>>> 
>>>>> I know of a system that uses logical decoding messages to propagate data
>>>>> to external systems. In that system, the application writes the data to
>>>>> be propagated as logical decoding messages, and a CDC pipeline consumes
>>>>> them via logical decoding and sends them to other systems, for example
>>>>> through Kafka.
>>>>> 
>>>>> In that system, when the remote site is maintained by physical replication,
>>>>> the same logical decoding messages can be decoded on the standby
>>>>> at the remote site to deliver them to external systems there. However,
>>>>> if the remote site uses logical replication instead, those messages are
>>>>> currently neither delivered to nor processed on the subscriber at
>>>>> the remote site. As a result, the CDC pipeline at the remote site cannot
>>>>> consume the data they carry.
>>>>> 
>>>>> The proposed hook might be useful for this use case. An extension could
>>>>> process the incoming logical decoding messages on the subscriber and
>>>>> forward them to the local CDC pipeline, or store or re-emit them in a form
>>>>> that local consumers can process.
>>>> 
>>>> Thank you for sharing the concrete use case. This is one of the use
>>>> cases I initially imagined.
>>>> 
>>>> I've rebased and updated the patch. Please review it.
>>>> 
>>>> Regards,
>>>> 
>>>> --
>>>> Masahiko Sawada
>>>> Amazon Web Services: https://aws.amazon.com
>>>> <v2-0001-Add-a-hook-for-handling-logical-messages-on-subsc.patch>
>>> 
>>> I played with this feature today. I have a few comments from a design perspective:
>> 
>> Thank you for reviewing the patch!
>> 
>>> 1. Would it make sense to pass the receiving subscription’s identity explicitly to the hook? Otherwise, a hook that needs to distinguish messages received by different subscriptions has to obtain it from a global variable such as MySubscription->oid.
>> 
>> It's common for hook functions to access a global variable to get more
>> information such as MyDatabaseId and MyProcPort etc. Using
>> MySubscription to get a subscription's identity works for me.
>> 
>>> 2. A hook is invoked by an apply worker, and since a message has no target relation, the hook runs as the subscription owner regardless of run_as_owner. I think it would be useful to document this explicitly.
>> 
>> I think it depends on the hook function implementations. They can
>> switch the role as they want.
>> 
>>> 
>>> 3. I’m thinking out loud here. Would it be useful to provide an opt-in default hook that re-emits received messages, so that they are written to the subscriber-side WAL and can be consumed by a local logical-decoding client? I understand that a re-emitted message would have a different LSN, just as logically replicated row changes generate new local WAL records. For the use case Fujii-san described, this might allow an existing decoder to continue working after the remote site switches from physical to logical replication.
>> 
>> Interesting idea, but I'm not sure we should have it in the core or in
>> the contrib. I think it's a good topic to discuss in a separate
>> thread.
>> 
>>> 
>>> And a few comment for the code changes:
>>> 
>>> 1 - worker_internal.h
>>> ```
>>> +typedef void (*LogicalRepMessageHandle_hook_type) (LogicalRepMessageData *msg);
>>> +extern PGDLLIMPORT LogicalRepMessageHandle_hook_type LogicalRepMessageHandle_hook;
>>> ```
>>> 
>>> I guess we don’t expect a hook function to mutate the message, so maybe make the pointer const.
>>> 
>>> 2 - worker.c
>>> ```
>>> +       /*
>>> +        * Logical messages are relation-agnostic, so they don't map cleanly onto
>>> +        * the tablesync worker of a particular relation, and there would be no
>>> +        * well-defined ordering between a message and the initial copy. Leave
>>> +        * them to the (parallel) apply workers. Logical messages are handled only
>>> +        * the (parallel) apply workers
>>> +        */
>>> ```
>>> 
>>> The last sentence looks duplicate and incomplete.
>>> 
>>> 3 - proto.c
>>> ```
>>> +       /* read message length */
>>> +       len = pq_getmsgint(in, 4);
>>> +       msg_data->message_size = len;
>>> +
>>> +       /* and data */
>>> +       msg = palloc(len + 1);
>>> +       pq_copymsgbytes(in, msg, len);
>>> +
>>> +       msg[len] = '\0';
>>> +       msg_data->message = msg;
>>> ```
>>> 
>>> logicalrep_read_message() allocates one extra byte for NULL terminator, which is unnecessary, as the contract is to use msg_data->message_size to decide the message length, and a message contain contains 0 in the middle. But I agree it may be useful for debugging and logging, so maybe add a comment to explain why using this extra byte.
>> 
>> Agreed with the all above comments.
>> 
>> I'll submit the updated patch shortly.
> 
> I've addressed all comments I got so far unless I'm missing anything,
> and attached the updated patch. Feedback is very welcome.
> 
> Regards,
> 
> -- 
> Masahiko Sawada
> Amazon Web Services: https://aws.amazon.com
> <v3-0001-Add-a-hook-for-handling-logical-messages-on-subsc.patch>

V3 LGTM.

Best regards,
--
Chao Li (Evan)
HighGo Software Co., Ltd.
https://www.highgo.com/










^ permalink  raw  reply  [nested|flat] 44+ messages in thread

* Re: Add a hook for handling logical decoding messages on subscribers.
@ 2026-08-06 07:03  Bharath Rupireddy <bharath.rupireddyforpostgres@gmail.com>
  parent: Masahiko Sawada <sawada.mshk@gmail.com>
  1 sibling, 1 reply; 44+ messages in thread

From: Bharath Rupireddy @ 2026-08-06 07:03 UTC (permalink / raw)
  To: Masahiko Sawada <sawada.mshk@gmail.com>; +Cc: Chao Li <li.evan.chao@gmail.com>; Fujii Masao <masao.fujii@gmail.com>; Amit Kapila <amit.kapila16@gmail.com>; pgsql-hackers

Hi,

On Wed, Aug 5, 2026 at 11:56 AM Masahiko Sawada <sawada.mshk@gmail.com> wrote:
>
> I've addressed all comments I got so far unless I'm missing anything,
> and attached the updated patch. Feedback is very welcome.

Thanks for sending the v3 patch. It looks good to me as-is. I verified
the patch with different cases such as emitting a bunch of ~900MB
messages to see the memory growth, enabling/disabling the subscription
option, pg_dump with the option enabled/disabled, regression tests,
and pgindent.

Here are some comments (may not need any code changes, feel free to ignore):

1/
+ if (options->proto.logical.messages &&
+ PQserverVersion(conn->streamConn) >= 140000)
+ appendStringInfo(&cmd, ", messages 'on'");

+        <term><literal>message</literal> (<type>boolean</type>)</term>

The publisher option is "messages". Should we have the subscription
option also use the same plural form to keep it consistent, and for
the reason that we receive a stream of messages, not just one?

2/
+ /*
+ * The message doesn't belong to any remote transaction, so there is
+ * no remote commit LSN nor timestamp to record. Clear the state left
+ * over by the previously applied transaction so that this commit
+ * doesn't inherit it.
+ */
+ replorigin_xact_clear(false);

Why is this a problem if we let the non-transactional message inherit it?

3/
+ logicalrep_read_message(s, &msg);
+
+ begin_replication_step();
+
+ (*LogicalRepMessageHandle_hook) (&msg);
+
+ end_replication_step();

Wrapping the hook with begin and end replication step is nice. This
lets the hook see the correct command ID, snapshot, and memory
context. There are callers that do the begin first and read message
next (insert), but it seems okay this way because read message doesn't
do any catalog or table accesses, so it should be fine.

--
Bharath Rupireddy
Amazon Web Services: https://aws.amazon.com






^ permalink  raw  reply  [nested|flat] 44+ messages in thread

* Re: Add a hook for handling logical decoding messages on subscribers.
@ 2026-08-06 16:58  Masahiko Sawada <sawada.mshk@gmail.com>
  parent: Bharath Rupireddy <bharath.rupireddyforpostgres@gmail.com>
  0 siblings, 1 reply; 44+ messages in thread

From: Masahiko Sawada @ 2026-08-06 16:58 UTC (permalink / raw)
  To: Bharath Rupireddy <bharath.rupireddyforpostgres@gmail.com>; +Cc: Chao Li <li.evan.chao@gmail.com>; Fujii Masao <masao.fujii@gmail.com>; Amit Kapila <amit.kapila16@gmail.com>; pgsql-hackers

On Thu, Aug 6, 2026 at 12:03 AM Bharath Rupireddy
<bharath.rupireddyforpostgres@gmail.com> wrote:
>
> Hi,
>
> On Wed, Aug 5, 2026 at 11:56 AM Masahiko Sawada <sawada.mshk@gmail.com> wrote:
> >
> > I've addressed all comments I got so far unless I'm missing anything,
> > and attached the updated patch. Feedback is very welcome.
>
> Thanks for sending the v3 patch. It looks good to me as-is. I verified
> the patch with different cases such as emitting a bunch of ~900MB
> messages to see the memory growth, enabling/disabling the subscription
> option, pg_dump with the option enabled/disabled, regression tests,
> and pgindent.
>
> Here are some comments (may not need any code changes, feel free to ignore):
>
> 1/
> + if (options->proto.logical.messages &&
> + PQserverVersion(conn->streamConn) >= 140000)
> + appendStringInfo(&cmd, ", messages 'on'");
>
> +        <term><literal>message</literal> (<type>boolean</type>)</term>
>
> The publisher option is "messages". Should we have the subscription
> option also use the same plural form to keep it consistent, and for
> the reason that we receive a stream of messages, not just one?

Agreed.

>
> 2/
> + /*
> + * The message doesn't belong to any remote transaction, so there is
> + * no remote commit LSN nor timestamp to record. Clear the state left
> + * over by the previously applied transaction so that this commit
> + * doesn't inherit it.
> + */
> + replorigin_xact_clear(false);
>
> Why is this a problem if we let the non-transactional message inherit it?

IIUC non-transactional messages would have the same commit timestamp
as the previously applied transaction, which is wrong to me.

>
> 3/
> + logicalrep_read_message(s, &msg);
> +
> + begin_replication_step();
> +
> + (*LogicalRepMessageHandle_hook) (&msg);
> +
> + end_replication_step();
>
> Wrapping the hook with begin and end replication step is nice. This
> lets the hook see the correct command ID, snapshot, and memory
> context. There are callers that do the begin first and read message
> next (insert), but it seems okay this way because read message doesn't
> do any catalog or table accesses, so it should be fine.
>

begin_replication_step() switches the memory context to
ApplyMessageContext. Given logicalrep_read_message() palloc's for
messages, it should be called after begin_replication_step(). Fixed it.

I've attached the updated patch.

Regards,


--
Masahiko Sawada
Amazon Web Services: https://aws.amazon.com

Attachments:

v4-0001-Add-a-hook-for-handling-logical-messages-on-subsc.patch (text/x-patch, 111.8K) download From c5d0772b54d2738b170292a943b24713053b6718 Mon Sep 17 00:00:00 2001 From: Masahiko Sawada <sawada.mshk@gmail.com> Date: Mon, 8 Jun 2026 11:25:47 -0700 Subject: [PATCH v4] Add a hook for handling logical messages on subscribers. Previously, pgoutput could emit logical decoding messages, those written with pg_logical_emit_message(), but the built-in subscriber never requested them from the publisher, and apply_handle_message() was only a placeholder that discarded the MESSAGE message type. Acting on such messages therefore meant writing a separate logical decoding client, and giving up the apply worker's streaming, conflict handling and progress tracking. Logical messages are a general-purpose side channel carried in band with the data, useful for replicating deparsed DDL, for carrying change-data-capture context that the row data does not have, and for cache invalidation delivered in commit order relative to the rows it refers to. This commit adds a new "messages" subscription option, which asks the publisher for the pgoutput "messages" option, and dispatches each received message to a new LogicalRepMessageHandle_hook. Behavior is unchanged when the option is off or when no handler is installed. A transactional message is applied as a step of the remote transaction it belongs to, so the handler's work commits atomically with that transaction, and transaction streaming and ALTER SUBSCRIPTION ... SKIP apply to it as they do to any other change. A non-transactional message belongs to no remote transaction and is committed on its own. A message may still reach the handler more than once, for example after an apply worker restart, so handlers must be idempotent. Note also that messages are not scoped by publication: every message emitted in the publisher database is sent, subject only to origin filtering, and pg_logical_emit_message() requires no special privileges. A handler must therefore treat the payload as untrusted input, and as an arbitrary string of bytes rather than as text in the subscriber's encoding. The publisher must be running PostgreSQL 14 or later, where pgoutput gained the "messages" option; against an older publisher the option is accepted but no message is ever sent. Bump catalog version. Reviewed-by: Bharath Rupireddy <bharath.rupireddyforpostgres@gmail.com> Reviewed-by: Fujii Masao <masao.fujii@gmail.com> Reviewed-by: Chao Li <li.evan.chao@gmail.com> Discussion: https://postgr.es/m/CAD21AoCTNGiddikkUcDKj5QLnsg-51bpr-o6L-GTHWZL4ZFYtQ@mail.gmail.com --- doc/src/sgml/catalogs.sgml | 11 + doc/src/sgml/func/func-admin.sgml | 7 +- doc/src/sgml/ref/alter_subscription.sgml | 5 +- doc/src/sgml/ref/create_subscription.sgml | 43 ++++ src/backend/catalog/pg_subscription.c | 1 + src/backend/catalog/system_views.sql | 5 +- src/backend/commands/subscriptioncmds.c | 25 ++- .../libpqwalreceiver/libpqwalreceiver.c | 4 + src/backend/replication/logical/proto.c | 37 ++++ src/backend/replication/logical/worker.c | 87 +++++++- src/bin/pg_dump/pg_dump.c | 16 +- src/bin/pg_dump/pg_dump.h | 1 + src/bin/pg_dump/t/002_pg_dump.pl | 4 +- src/bin/psql/describe.c | 7 +- src/include/catalog/pg_subscription.h | 5 + src/include/replication/logicalproto.h | 13 ++ src/include/replication/logicalworker.h | 35 +++ src/include/replication/walreceiver.h | 1 + src/test/modules/Makefile | 1 + src/test/modules/meson.build | 1 + .../modules/test_logicalmsg_hooks/.gitignore | 4 + .../modules/test_logicalmsg_hooks/Makefile | 20 ++ .../modules/test_logicalmsg_hooks/meson.build | 28 +++ .../test_logicalmsg_hooks/t/001_basic.pl | 207 ++++++++++++++++++ .../test_logicalmsg_hooks.c | 140 ++++++++++++ src/test/regress/expected/subscription.out | 202 +++++++++-------- src/test/regress/sql/subscription.sql | 17 ++ src/tools/pgindent/typedefs.list | 2 + 28 files changed, 822 insertions(+), 107 deletions(-) create mode 100644 src/test/modules/test_logicalmsg_hooks/.gitignore create mode 100644 src/test/modules/test_logicalmsg_hooks/Makefile create mode 100644 src/test/modules/test_logicalmsg_hooks/meson.build create mode 100644 src/test/modules/test_logicalmsg_hooks/t/001_basic.pl create mode 100644 src/test/modules/test_logicalmsg_hooks/test_logicalmsg_hooks.c diff --git a/doc/src/sgml/catalogs.sgml b/doc/src/sgml/catalogs.sgml index 6066c4784f4..499f76b94d4 100644 --- a/doc/src/sgml/catalogs.sgml +++ b/doc/src/sgml/catalogs.sgml @@ -8737,6 +8737,17 @@ SCRAM-SHA-256$<replaceable>&lt;iteration count&gt;</replaceable>:<replaceable>&l </para></entry> </row> + <row> + <entry role="catalog_table_entry"><para role="column_definition"> + <structfield>submessages</structfield> <type>bool</type> + </para> + <para> + If true, the subscription requests the publisher to send logical + decoding messages, which are passed to a handler registered by an + extension on the subscriber. + </para></entry> + </row> + <row> <entry role="catalog_table_entry"><para role="column_definition"> <structfield>subconninfo</structfield> <type>text</type> diff --git a/doc/src/sgml/func/func-admin.sgml b/doc/src/sgml/func/func-admin.sgml index 0eae1c1f616..091c5c1416b 100644 --- a/doc/src/sgml/func/func-admin.sgml +++ b/doc/src/sgml/func/func-admin.sgml @@ -1479,7 +1479,12 @@ postgres=# SELECT '0/0'::pg_lsn + pd.segment_number * ps.setting::int + :offset <literal>false</literal>) controls if the message is immediately flushed to WAL or not. <parameter>flush</parameter> has no effect with <parameter>transactional</parameter>, as the message's WAL - record is flushed along with its transaction. + record is flushed along with its transaction. In addition to logical + decoding clients, messages emitted by this function are delivered to + subscribers whose subscriptions have + <link linkend="sql-createsubscription-params-with-messages"><literal>messages</literal></link> + enabled, where they are passed to a handler registered by an extension + on the subscriber. </para></entry> </row> diff --git a/doc/src/sgml/ref/alter_subscription.sgml b/doc/src/sgml/ref/alter_subscription.sgml index b6f47545682..26662feb544 100644 --- a/doc/src/sgml/ref/alter_subscription.sgml +++ b/doc/src/sgml/ref/alter_subscription.sgml @@ -309,8 +309,9 @@ ALTER SUBSCRIPTION <replaceable class="parameter">name</replaceable> RENAME TO < <link linkend="sql-createsubscription-params-with-two-phase"><literal>two_phase</literal></link>, <link linkend="sql-createsubscription-params-with-retain-dead-tuples"><literal>retain_dead_tuples</literal></link>, <link linkend="sql-createsubscription-params-with-max-retention-duration"><literal>max_retention_duration</literal></link>, - <link linkend="sql-createsubscription-params-with-wal-receiver-timeout"><literal>wal_receiver_timeout</literal></link>, and - <link linkend="sql-createsubscription-params-with-conflict-log-destination"><literal>conflict_log_destination</literal></link>. + <link linkend="sql-createsubscription-params-with-wal-receiver-timeout"><literal>wal_receiver_timeout</literal></link>, + <link linkend="sql-createsubscription-params-with-conflict-log-destination"><literal>conflict_log_destination</literal></link>, and + <link linkend="sql-createsubscription-params-with-messages"><literal>messages</literal></link>. Only a superuser can set <literal>password_required = false</literal>. </para> diff --git a/doc/src/sgml/ref/create_subscription.sgml b/doc/src/sgml/ref/create_subscription.sgml index 25a81e2e62a..b2337ff3d57 100644 --- a/doc/src/sgml/ref/create_subscription.sgml +++ b/doc/src/sgml/ref/create_subscription.sgml @@ -478,6 +478,49 @@ CREATE SUBSCRIPTION <replaceable class="parameter">subscription_name</replaceabl </listitem> </varlistentry> + <varlistentry id="sql-createsubscription-params-with-messages"> + <term><literal>messages</literal> (<type>boolean</type>)</term> + <listitem> + <para> + Specifies whether the subscription requests the publisher to send + logical decoding messages, that is, those emitted with + <link linkend="pg-logical-emit-message"><function>pg_logical_emit_message</function></link>. + The default is <literal>false</literal>. + </para> + <para> + Received messages are only acted upon if an extension installed on + the subscriber has registered a handler for them; otherwise they are + received and discarded. A message may be passed to the handler more + than once, for example after an apply worker restart, so a handler + must be prepared to process the same message repeatedly. + </para> + <para> + A transactional message is applied as part of the remote transaction + that emitted it, and is therefore delivered in commit order with + respect to other transactions, not in the order in which it was written. + Note that + <link linkend="sql-createsubscription-params-with-streaming"><literal>streaming</literal></link> + and <link linkend="sql-altersubscription-params-skip"><literal>SKIP</literal></link> + apply to transactional messages just as they do to other changes. A + non-transactional message is delivered as soon as it is decoded, even + if the transaction that emitted it later aborted. + </para> + <para> + Messages are not restricted to the tables or schemas of the + subscribed publications: every message emitted in the publisher + database is sent, subject only to + <link linkend="sql-createsubscription-params-with-origin"><literal>origin</literal></link> + filtering. Message contents are supplied by whoever calls + <function>pg_logical_emit_message</function> on the publisher, which + requires no special privileges, so a handler should treat them as + untrusted input. A message is an arbitrary string of bytes, and is + transmitted without character set conversion; it may therefore + contain embedded nulls, and need not be valid text in the encoding + of either database. + </para> + </listitem> + </varlistentry> + <varlistentry id="sql-createsubscription-params-with-origin"> <term><literal>origin</literal> (<type>string</type>)</term> <listitem> diff --git a/src/backend/catalog/pg_subscription.c b/src/backend/catalog/pg_subscription.c index 5ff61edb989..8ba016c25d1 100644 --- a/src/backend/catalog/pg_subscription.c +++ b/src/backend/catalog/pg_subscription.c @@ -139,6 +139,7 @@ GetSubscription(Oid subid, bool missing_ok, bool conninfo_needed, sub->maxretention = subform->submaxretention; sub->retentionactive = subform->subretentionactive; sub->conflictlogrelid = subform->subconflictlogrelid; + sub->messages = subform->submessages; if (conninfo_needed) { diff --git a/src/backend/catalog/system_views.sql b/src/backend/catalog/system_views.sql index 090281a03dd..a1ac6aef9ee 100644 --- a/src/backend/catalog/system_views.sql +++ b/src/backend/catalog/system_views.sql @@ -1538,8 +1538,9 @@ GRANT SELECT (oid, subdbid, subskiplsn, subname, subowner, subenabled, subbinary, substream, subtwophasestate, subdisableonerr, subpasswordrequired, subrunasowner, subfailover, subretaindeadtuples, submaxretention, subretentionactive, - subserver, subconflictlogrelid, subconflictlogdest, subslotname, - subsynccommit, subwalrcvtimeout, subpublications, suborigin) + subserver, subconflictlogrelid, subconflictlogdest, submessages, + subslotname, subsynccommit, subwalrcvtimeout, subpublications, + suborigin) ON pg_subscription TO public; CREATE VIEW pg_stat_subscription_stats AS diff --git a/src/backend/commands/subscriptioncmds.c b/src/backend/commands/subscriptioncmds.c index d3eef2a3efa..1cbb1c1bd5c 100644 --- a/src/backend/commands/subscriptioncmds.c +++ b/src/backend/commands/subscriptioncmds.c @@ -83,6 +83,7 @@ #define SUBOPT_LSN 0x00020000 #define SUBOPT_ORIGIN 0x00040000 #define SUBOPT_CONFLICT_LOG_DEST 0x00080000 +#define SUBOPT_MESSAGES 0x00100000 /* check if the 'val' has 'bits' set */ #define IsSet(val, bits) (((val) & (bits)) == (bits)) @@ -109,6 +110,7 @@ typedef struct SubOpts bool runasowner; bool failover; bool retaindeadtuples; + bool messages; int32 maxretention; char *origin; ConflictLogDest conflictlogdest; @@ -207,6 +209,8 @@ parse_subscription_options(ParseState *pstate, List *stmt_options, opts->origin = pstrdup(LOGICALREP_ORIGIN_ANY); if (IsSet(supported_opts, SUBOPT_CONFLICT_LOG_DEST)) opts->conflictlogdest = CONFLICT_LOG_DEST_LOG; + if (IsSet(supported_opts, SUBOPT_MESSAGES)) + opts->messages = false; /* Parse options */ foreach(lc, stmt_options) @@ -459,6 +463,15 @@ parse_subscription_options(ParseState *pstate, List *stmt_options, opts->conflictlogdest = GetConflictLogDest(val); opts->specified_opts |= SUBOPT_CONFLICT_LOG_DEST; } + else if (IsSet(supported_opts, SUBOPT_MESSAGES) && + strcmp(defel->defname, "messages") == 0) + { + if (IsSet(opts->specified_opts, SUBOPT_MESSAGES)) + errorConflictingDefElem(defel, pstate); + + opts->specified_opts |= SUBOPT_MESSAGES; + opts->messages = defGetBoolean(defel); + } else ereport(ERROR, (errcode(ERRCODE_SYNTAX_ERROR), @@ -699,7 +712,7 @@ CreateSubscription(ParseState *pstate, CreateSubscriptionStmt *stmt, SUBOPT_RETAIN_DEAD_TUPLES | SUBOPT_MAX_RETENTION_DURATION | SUBOPT_WAL_RECEIVER_TIMEOUT | SUBOPT_ORIGIN | - SUBOPT_CONFLICT_LOG_DEST); + SUBOPT_CONFLICT_LOG_DEST | SUBOPT_MESSAGES); parse_subscription_options(pstate, stmt->options, supported_opts, &opts); /* @@ -854,6 +867,7 @@ CreateSubscription(ParseState *pstate, CreateSubscriptionStmt *stmt, values[Anum_pg_subscription_subretentionactive - 1] = BoolGetDatum(opts.retaindeadtuples); values[Anum_pg_subscription_subserver - 1] = ObjectIdGetDatum(serverid); + values[Anum_pg_subscription_submessages - 1] = BoolGetDatum(opts.messages); if (!OidIsValid(serverid)) values[Anum_pg_subscription_subconninfo - 1] = CStringGetTextDatum(conninfo); @@ -1666,7 +1680,8 @@ AlterSubscription(ParseState *pstate, AlterSubscriptionStmt *stmt, SUBOPT_MAX_RETENTION_DURATION | SUBOPT_WAL_RECEIVER_TIMEOUT | SUBOPT_ORIGIN | - SUBOPT_CONFLICT_LOG_DEST); + SUBOPT_CONFLICT_LOG_DEST | + SUBOPT_MESSAGES); break; case ALTER_SUBSCRIPTION_ENABLED: @@ -2053,6 +2068,12 @@ AlterSubscription(ParseState *pstate, AlterSubscriptionStmt *stmt, } } + if (IsSet(opts.specified_opts, SUBOPT_MESSAGES)) + { + values[Anum_pg_subscription_submessages - 1] = BoolGetDatum(opts.messages); + replaces[Anum_pg_subscription_submessages - 1] = true; + } + update_tuple = true; break; } diff --git a/src/backend/replication/libpqwalreceiver/libpqwalreceiver.c b/src/backend/replication/libpqwalreceiver/libpqwalreceiver.c index 029990d9fce..e345b80690f 100644 --- a/src/backend/replication/libpqwalreceiver/libpqwalreceiver.c +++ b/src/backend/replication/libpqwalreceiver/libpqwalreceiver.c @@ -631,6 +631,10 @@ libpqrcv_startstreaming(WalReceiverConn *conn, appendQuotedLiteral(&cmd, options->proto.logical.origin); } + if (options->proto.logical.messages && + PQserverVersion(conn->streamConn) >= 140000) + appendStringInfo(&cmd, ", messages 'on'"); + pubnames = options->proto.logical.publication_names; pubnames_str = stringlist_to_identifierstr(pubnames); appendStringInfoString(&cmd, ", publication_names "); diff --git a/src/backend/replication/logical/proto.c b/src/backend/replication/logical/proto.c index 86ad97cd937..6461361d68d 100644 --- a/src/backend/replication/logical/proto.c +++ b/src/backend/replication/logical/proto.c @@ -660,6 +660,43 @@ logicalrep_write_message(StringInfo out, TransactionId xid, XLogRecPtr lsn, pq_sendbytes(out, message, sz); } +/* + * Read MESSAGE from stream. + */ +void +logicalrep_read_message(StringInfo in, LogicalRepMessageData *msg_data) +{ + Size len; + char *msg; + uint8 flags; + + /* read and decode flags */ + flags = pq_getmsgint(in, 1); + msg_data->transactional = (flags & MESSAGE_TRANSACTIONAL) != 0; + + msg_data->lsn = pq_getmsgint64(in); + msg_data->prefix = pstrdup(pq_getmsgstring(in)); + + /* read message length */ + len = pq_getmsgint(in, 4); + msg_data->message_size = len; + + /* and data */ + msg = palloc(len + 1); + pq_copymsgbytes(in, msg, len); + + /* + * The payload is an arbitrary string of bytes, so message_size, not + * strlen(), is what gives its length. NULL-terminate it anyway, as + * logicalrep_read_tuple() does for column values: that way it can be + * handed to code expecting a C string without reading past the end, which + * is convenient for logging and debugging. Such code will of course stop + * at the first embedded null, if any. + */ + msg[len] = '\0'; + msg_data->message = msg; +} + /* * Write relation description to the output stream. */ diff --git a/src/backend/replication/logical/worker.c b/src/backend/replication/logical/worker.c index ce4f1b93b66..0782757f6fa 100644 --- a/src/backend/replication/logical/worker.c +++ b/src/backend/replication/logical/worker.c @@ -549,6 +549,9 @@ typedef struct ApplySubXactData static ApplySubXactData subxact_data = {0, 0, InvalidTransactionId, NULL}; +/* Hook for plugins to get control in handling logical decoding messages */ +LogicalRepMessageHandle_hook_type LogicalRepMessageHandle_hook = NULL; + static inline void subxact_filename(char *path, Oid subid, TransactionId xid); static inline void changes_filename(char *path, Oid subid, TransactionId xid); @@ -1694,6 +1697,71 @@ apply_handle_origin(StringInfo s) errmsg_internal("ORIGIN message sent out of order"))); } +/* + * Handle MESSAGE message. + * + * Invoke a hook function if set. + */ +static void +apply_handle_message(StringInfo s) +{ + LogicalRepMessageData msg; + + /* Tablesync worker should never receive MESSAGE */ + if (am_tablesync_worker()) + ereport(ERROR, + errcode(ERRCODE_PROTOCOL_VIOLATION), + errmsg_internal("tablesync worker received a MESSAGE message")); + + if (!LogicalRepMessageHandle_hook) + return; + + if (is_skipping_changes() || + handle_streamed_transaction(LOGICAL_REP_MSG_MESSAGE, s)) + return; + + begin_replication_step(); + + logicalrep_read_message(s, &msg); + + (*LogicalRepMessageHandle_hook) (&msg); + + end_replication_step(); + + /* + * A transactional message is applied as a step of the remote transaction + * that emitted it, and is committed together with it when applying the + * commit message. A non-transactional message belongs to no remote + * transaction, so commit it here. + */ + if (!msg.transactional) + { + Assert(!in_remote_transaction); + Assert(!in_streamed_transaction); + + /* + * The message doesn't belong to any remote transaction, so there is + * no remote commit LSN nor timestamp to record. Clear the state left + * over by the previously applied transaction so that this commit + * doesn't inherit it. + */ + replorigin_xact_clear(false); + + CommitTransactionCommand(); + + pgstat_report_stat(false); + + /* + * Report the flush position only after the local commit is durable. + * Otherwise send_feedback() would report everything received so far + * as flushed, the publisher would advance confirmed_flush_lsn past + * this message, and a crash before the commit reaches disk would lose + * the handler's work without the message ever being sent again. + */ + store_flush_position(msg.lsn, XactLastCommitEnd); + } +} + /* * Initialize fileset (if not already done). * @@ -3846,12 +3914,7 @@ apply_dispatch(StringInfo s) break; case LOGICAL_REP_MSG_MESSAGE: - - /* - * Logical replication does not use generic logical messages yet. - * Although, it could be used by other applications that use this - * output plugin. - */ + apply_handle_message(s); break; case LOGICAL_REP_MSG_STREAM_START: @@ -5128,7 +5191,8 @@ maybe_reread_subscription(void) newsub->passwordrequired != MySubscription->passwordrequired || strcmp(newsub->origin, MySubscription->origin) != 0 || newsub->owner != MySubscription->owner || - !equal(newsub->publications, MySubscription->publications)) + !equal(newsub->publications, MySubscription->publications) || + newsub->messages != MySubscription->messages) { if (am_parallel_apply_worker()) ereport(LOG, @@ -5614,6 +5678,15 @@ set_stream_options(WalRcvStreamOptions *options, options->proto.logical.twophase = false; options->proto.logical.origin = pstrdup(MySubscription->origin); + + /* + * Logical messages are relation-agnostic, so they don't map cleanly onto + * the tablesync worker of a particular relation, and there would be no + * well-defined ordering between a message and the initial copy. Leave + * them to the (parallel) apply workers. + */ + options->proto.logical.messages = (MySubscription->messages && + !am_tablesync_worker()); } /* diff --git a/src/bin/pg_dump/pg_dump.c b/src/bin/pg_dump/pg_dump.c index 4948e6d80c7..6ae05d9ad0d 100644 --- a/src/bin/pg_dump/pg_dump.c +++ b/src/bin/pg_dump/pg_dump.c @@ -5138,6 +5138,7 @@ getSubscriptions(Archive *fout) int i_subfailover; int i_subretaindeadtuples; int i_submaxretention; + int i_submessages; int i, ntups; @@ -5236,9 +5237,14 @@ getSubscriptions(Archive *fout) " '-1' AS subwalrcvtimeout,\n"); if (fout->remoteVersion >= 190000) - appendPQExpBufferStr(query, " fs.srvname AS subservername\n"); + appendPQExpBufferStr(query, " fs.srvname AS subservername,\n"); else - appendPQExpBufferStr(query, " NULL AS subservername\n"); + appendPQExpBufferStr(query, " NULL AS subservername,\n"); + + if (fout->remoteVersion >= 200000) + appendPQExpBufferStr(query, " s.submessages\n"); + else + appendPQExpBufferStr(query, " false AS submessages\n"); appendPQExpBufferStr(query, "FROM pg_subscription s\n"); @@ -5287,6 +5293,7 @@ getSubscriptions(Archive *fout) i_subpublications = PQfnumber(res, "subpublications"); i_suborigin = PQfnumber(res, "suborigin"); i_suboriginremotelsn = PQfnumber(res, "suboriginremotelsn"); + i_submessages = PQfnumber(res, "submessages"); subinfo = pg_malloc_array(SubscriptionInfo, ntups); @@ -5344,6 +5351,8 @@ getSubscriptions(Archive *fout) else subinfo[i].suboriginremotelsn = pg_strdup(PQgetvalue(res, i, i_suboriginremotelsn)); + subinfo[i].submessages = + (strcmp(PQgetvalue(res, i, i_submessages), "t") == 0); /* Decide whether we want to dump it */ selectDumpableObject(&(subinfo[i].dobj), fout); @@ -5596,6 +5605,9 @@ dumpSubscription(Archive *fout, const SubscriptionInfo *subinfo) if (subinfo->subretaindeadtuples) appendPQExpBufferStr(query, ", retain_dead_tuples = true"); + if (subinfo->submessages) + appendPQExpBufferStr(query, ", messages = true"); + if (subinfo->submaxretention) appendPQExpBuffer(query, ", max_retention_duration = %d", subinfo->submaxretention); diff --git a/src/bin/pg_dump/pg_dump.h b/src/bin/pg_dump/pg_dump.h index 5a6726d8b12..b6a63004880 100644 --- a/src/bin/pg_dump/pg_dump.h +++ b/src/bin/pg_dump/pg_dump.h @@ -721,6 +721,7 @@ typedef struct _SubscriptionInfo bool subrunasowner; bool subfailover; bool subretaindeadtuples; + bool submessages; int submaxretention; char *subservername; char *subconninfo; diff --git a/src/bin/pg_dump/t/002_pg_dump.pl b/src/bin/pg_dump/t/002_pg_dump.pl index 9258948b583..49853528fa5 100644 --- a/src/bin/pg_dump/t/002_pg_dump.pl +++ b/src/bin/pg_dump/t/002_pg_dump.pl @@ -3317,9 +3317,9 @@ my %tests = ( create_order => 50, create_sql => 'CREATE SUBSCRIPTION sub3 CONNECTION \'dbname=doesnotexist\' PUBLICATION pub1 - WITH (connect = false, origin = any, streaming = on);', + WITH (connect = false, origin = any, streaming = on, messages = on);', regexp => qr/^ - \QCREATE SUBSCRIPTION sub3 CONNECTION 'dbname=doesnotexist' PUBLICATION pub1 WITH (connect = false, slot_name = 'sub3', streaming = on);\E + \QCREATE SUBSCRIPTION sub3 CONNECTION 'dbname=doesnotexist' PUBLICATION pub1 WITH (connect = false, slot_name = 'sub3', streaming = on, messages = true);\E /xm, like => { %full_runs, section_post_data => 1, }, unlike => { diff --git a/src/bin/psql/describe.c b/src/bin/psql/describe.c index ad9c8affb4f..a968ed82e60 100644 --- a/src/bin/psql/describe.c +++ b/src/bin/psql/describe.c @@ -6894,7 +6894,7 @@ describeSubscriptions(const char *pattern, bool verbose) printQueryOpt myopt = pset.popt; static const bool translate_columns[] = {false, false, false, false, false, false, false, false, false, false, false, false, false, false, - false, false, false, false, false, false, false}; + false, false, false, false, false, false, false, false}; initPQExpBuffer(&buf); @@ -6972,6 +6972,11 @@ describeSubscriptions(const char *pattern, bool verbose) gettext_noop("Retention active")); } + if (pset.sversion >= 200000) + appendPQExpBuffer(&buf, + ", submessages AS \"%s\"\n", + gettext_noop("Messages")); + appendPQExpBuffer(&buf, ", subsynccommit AS \"%s\"\n" ", subconninfo AS \"%s\"\n", diff --git a/src/include/catalog/pg_subscription.h b/src/include/catalog/pg_subscription.h index 65ce8e145fb..650663b71fd 100644 --- a/src/include/catalog/pg_subscription.h +++ b/src/include/catalog/pg_subscription.h @@ -96,6 +96,9 @@ CATALOG(pg_subscription,6100,SubscriptionRelationId) BKI_SHARED_RELATION BKI_ROW * server */ Oid subconflictlogrelid; /* Relid of the conflict log table. */ + + bool submessages; /* True if the subscription wants to receive + * logical messages. */ #ifdef CATALOG_VARLEN /* variable-length fields start here */ /* @@ -173,6 +176,8 @@ typedef struct Subscription * exceeded max_retention_duration, when * defined */ Oid conflictlogrelid; /* conflict log table Oid */ + bool messages; /* True if the subscription wants to receive + * logical messages */ char *conninfo; /* Connection string to the publisher */ char *slotname; /* Name of the replication slot */ char *synccommit; /* Synchronous commit setting for worker */ diff --git a/src/include/replication/logicalproto.h b/src/include/replication/logicalproto.h index 058a955e20c..83741a2080d 100644 --- a/src/include/replication/logicalproto.h +++ b/src/include/replication/logicalproto.h @@ -191,6 +191,18 @@ typedef struct LogicalRepStreamAbortData TimestampTz abort_time; } LogicalRepStreamAbortData; +/* + * Logical decoding message information + */ +typedef struct LogicalRepMessageData +{ + XLogRecPtr lsn; + bool transactional; + const char *prefix; + Size message_size; /* length of message in bytes */ + const char *message; /* payload; may contain embedded nulls */ +} LogicalRepMessageData; + extern void logicalrep_write_begin(StringInfo out, ReorderBufferTXN *txn); extern void logicalrep_read_begin(StringInfo in, LogicalRepBeginData *begin_data); @@ -248,6 +260,7 @@ extern List *logicalrep_read_truncate(StringInfo in, bool *cascade, bool *restart_seqs); extern void logicalrep_write_message(StringInfo out, TransactionId xid, XLogRecPtr lsn, bool transactional, const char *prefix, Size sz, const char *message); +extern void logicalrep_read_message(StringInfo in, LogicalRepMessageData *msg_data); extern void logicalrep_write_rel(StringInfo out, TransactionId xid, Relation rel, Bitmapset *columns, PublishGencolsType include_gencols_type); diff --git a/src/include/replication/logicalworker.h b/src/include/replication/logicalworker.h index 7d748a28da8..a72ee96020c 100644 --- a/src/include/replication/logicalworker.h +++ b/src/include/replication/logicalworker.h @@ -16,6 +16,41 @@ extern PGDLLIMPORT volatile sig_atomic_t ParallelApplyMessagePending; +/* + * Forward-declared so that this header, which is included by several files + * unrelated to logical replication, does not have to include logicalproto.h. + * A handler that dereferences the struct must include that header itself. + */ +struct LogicalRepMessageData; + +/* + * Hook for extensions to handle logical decoding messages (see + * pg_logical_emit_message()) received from the publisher. Called only when + * the subscription has the "messages" option enabled. + * + * The handler runs in the apply worker, inside a transaction, with an active + * snapshot pushed. For a transactional message the handler's work is part of + * the remote transaction and commits with it; a non-transactional message is + * committed as an independent unit. The same message may be delivered more + * than once, so the handler must be idempotent. + * + * The payload is an arbitrary string of bytes. The WAL record makes no + * distinction between the text and the bytea form of + * pg_logical_emit_message(), and the bytes are transmitted without character + * set conversion, so the payload may contain embedded nulls and bytes that + * are not valid in the subscriber's encoding. Its length is given by + * msg->message_size, not by strlen(); the buffer is null-terminated only for + * convenience, and that null is not part of the payload. + * + * Note that the apply worker runs with an empty search_path. An ERROR thrown + * by the handler restarts the apply worker, or disables the subscription if + * disable_on_error is set. Because the message is delivered again after the + * restart, a handler cannot reject a message by throwing an error; that only + * loops. A message the handler does not want must simply be ignored. + */ +typedef void (*LogicalRepMessageHandle_hook_type) (const struct LogicalRepMessageData *msg); +extern PGDLLIMPORT LogicalRepMessageHandle_hook_type LogicalRepMessageHandle_hook; + extern void ApplyWorkerMain(Datum main_arg); extern void ParallelApplyWorkerMain(Datum main_arg); extern void TableSyncWorkerMain(Datum main_arg); diff --git a/src/include/replication/walreceiver.h b/src/include/replication/walreceiver.h index 5cfe90f9989..6831bd1f751 100644 --- a/src/include/replication/walreceiver.h +++ b/src/include/replication/walreceiver.h @@ -188,6 +188,7 @@ typedef struct * prepare time */ char *origin; /* Only publish data originating from the * specified origin */ + bool messages; /* Logical messages */ } logical; } proto; } WalRcvStreamOptions; diff --git a/src/test/modules/Makefile b/src/test/modules/Makefile index 098bb8142ae..e8a7f50eb38 100644 --- a/src/test/modules/Makefile +++ b/src/test/modules/Makefile @@ -36,6 +36,7 @@ SUBDIRS = \ test_integerset \ test_json_parser \ test_lfind \ + test_logicalmsg_hooks \ test_lwlock_tranches \ test_misc \ test_oat_hooks \ diff --git a/src/test/modules/meson.build b/src/test/modules/meson.build index 4bca42bb370..5f6c0cae77a 100644 --- a/src/test/modules/meson.build +++ b/src/test/modules/meson.build @@ -37,6 +37,7 @@ subdir('test_int128') subdir('test_integerset') subdir('test_json_parser') subdir('test_lfind') +subdir('test_logicalmsg_hooks') subdir('test_lwlock_tranches') subdir('test_misc') subdir('test_oat_hooks') diff --git a/src/test/modules/test_logicalmsg_hooks/.gitignore b/src/test/modules/test_logicalmsg_hooks/.gitignore new file mode 100644 index 00000000000..5dcb3ff9723 --- /dev/null +++ b/src/test/modules/test_logicalmsg_hooks/.gitignore @@ -0,0 +1,4 @@ +# Generated subdirectories +/log/ +/results/ +/tmp_check/ diff --git a/src/test/modules/test_logicalmsg_hooks/Makefile b/src/test/modules/test_logicalmsg_hooks/Makefile new file mode 100644 index 00000000000..af129009b8a --- /dev/null +++ b/src/test/modules/test_logicalmsg_hooks/Makefile @@ -0,0 +1,20 @@ +# src/test/modules/test_logicalmsg_hooks/Makefile + +MODULE_big = test_logicalmsg_hooks +OBJS = \ + $(WIN32RES) \ + test_logicalmsg_hooks.o +PGFILEDESC = "test_logicalmsg_hooks - test logical replication message hooks" + +TAP_TESTS = 1 + +ifdef USE_PGXS +PG_CONFIG = pg_config +PGXS := $(shell $(PG_CONFIG) --pgxs) +include $(PGXS) +else +subdir = src/test/modules/test_logicalmsg_hooks +top_builddir = ../../../.. +include $(top_builddir)/src/Makefile.global +include $(top_srcdir)/contrib/contrib-global.mk +endif diff --git a/src/test/modules/test_logicalmsg_hooks/meson.build b/src/test/modules/test_logicalmsg_hooks/meson.build new file mode 100644 index 00000000000..8084d82e47b --- /dev/null +++ b/src/test/modules/test_logicalmsg_hooks/meson.build @@ -0,0 +1,28 @@ +# Copyright (c) 2022-2026, PostgreSQL Global Development Group + +test_logicalmsg_hooks_sources = files( + 'test_logicalmsg_hooks.c', +) + +if host_system == 'windows' + test_logicalmsg_hooks_sources += rc_lib_gen.process(win32ver_rc, extra_args: [ + '--NAME', 'test_logicalmsg_hooks', + '--FILEDESC', 'test_logicalmsg_hooks - test logical message hooks',]) +endif + +test_logicalmsg_hooks = shared_module('test_logicalmsg_hooks', + test_logicalmsg_hooks_sources, + kwargs: pg_test_mod_args, +) +test_install_libs += test_logicalmsg_hooks + +tests += { + 'name': 'test_logicalmsg_hooks', + 'sd': meson.current_source_dir(), + 'bd': meson.current_build_dir(), + 'tap': { + 'tests': [ + 't/001_basic.pl', + ], + }, +} diff --git a/src/test/modules/test_logicalmsg_hooks/t/001_basic.pl b/src/test/modules/test_logicalmsg_hooks/t/001_basic.pl new file mode 100644 index 00000000000..a9ff6171ad6 --- /dev/null +++ b/src/test/modules/test_logicalmsg_hooks/t/001_basic.pl @@ -0,0 +1,207 @@ +# Copyright (c) 2026, PostgreSQL Global Development Group + +# Test LogicalRepMessageHandle_hook. The test module records every message it +# receives into public.test_logicalmsg_log on the subscriber, so most checks +# here are plain queries against that table. + +use strict; +use warnings FATAL => 'all'; + +use PostgreSQL::Test::Cluster; +use PostgreSQL::Test::Utils; +use Test::More; + +my $pub = PostgreSQL::Test::Cluster->new('publisher'); +$pub->init(allows_streaming => 'logical'); + +# Keep the memory limit low so that the transactions below are streamed. +$pub->append_conf('postgresql.conf', "logical_decoding_work_mem = 64kB\n"); +$pub->start; + +my $sub = PostgreSQL::Test::Cluster->new('subscriber'); +$sub->init(allows_streaming => 'logical'); +$sub->append_conf( + 'postgresql.conf', q{ +shared_preload_libraries = 'test_logicalmsg_hooks' +max_parallel_apply_workers_per_subscription = 2 +}); +$sub->start; + +$pub->safe_psql( + 'postgres', qq{ +CREATE TABLE test (a int); +INSERT INTO test VALUES (1); +CREATE PUBLICATION pub FOR ALL TABLES; +}); + +my $pub_connstr = $pub->connstr . ' dbname=postgres'; + +$sub->safe_psql( + 'postgres', qq{ +CREATE TABLE test (a int primary key); +CREATE SUBSCRIPTION sub CONNECTION '$pub_connstr' PUBLICATION pub WITH (messages = true, disable_on_error = true) +}); + +$sub->wait_for_subscription_sync($pub, 'sub'); + +# Number of messages recorded under the given prefix. +sub logged_count +{ + my ($prefix) = @_; + + return $sub->safe_psql('postgres', + "SELECT count(*) FROM test_logicalmsg_log WHERE prefix = '$prefix'"); +} + +# Verify that a transactional message is passed to the hook, and that the +# message data given to the hook is correct. +$pub->safe_psql('postgres', + q{SELECT pg_logical_emit_message(true, 'basic', 'transactional message')} +); +$pub->wait_for_catchup('sub'); + +is( $sub->safe_psql( + 'postgres', + q{SELECT transactional, message_size, message FROM test_logicalmsg_log + WHERE prefix = 'basic'}), + 't|21|transactional message', + "transactional message is passed to the hook"); + +# Verify that a non-transactional message is applied even if the transaction +# that emitted it rolls back. +$pub->safe_psql( + 'postgres', q{ +BEGIN; +SELECT pg_logical_emit_message(false, 'nontransactional', 'emitted then rolled back'); +ROLLBACK; +}); + +# Commit another message to flush WAL, as ROLLBACK doesn't flush. +$pub->safe_psql('postgres', + q{SELECT pg_logical_emit_message(true, 'flush', 'flush')}); +$pub->wait_for_catchup('sub'); + +is( $sub->safe_psql( + 'postgres', + q{SELECT transactional, message FROM test_logicalmsg_log + WHERE prefix = 'nontransactional'}), + 'f|emitted then rolled back', + "non-transactional message of an aborted transaction is applied"); + +# Verify that a transactional message emitted in a subtransaction that is +# rolled back is not passed to the hook. +$pub->safe_psql( + 'postgres', q{ +BEGIN; +SAVEPOINT sp; +SELECT pg_logical_emit_message(true, 'subxact_aborted', 'not delivered'); +ROLLBACK TO SAVEPOINT sp; +SELECT pg_logical_emit_message(true, 'subxact_committed', 'delivered'); +COMMIT; +}); +$pub->wait_for_catchup('sub'); + +is(logged_count('subxact_aborted'), + 0, "message of an aborted subtransaction is not applied"); +is(logged_count('subxact_committed'), + 1, "message of a committed subtransaction is applied"); + +# Verify that transactional messages are applied in streaming transactions, +# both when the leader apply worker serializes the transaction to a file and +# when it hands the transaction to a parallel apply worker. +foreach my $mode ('on', 'parallel') +{ + $sub->safe_psql('postgres', + "ALTER SUBSCRIPTION sub SET (streaming = $mode)"); + + # Reconnect so that the new streaming mode takes effect immediately. + $sub->safe_psql('postgres', 'ALTER SUBSCRIPTION sub DISABLE'); + $sub->safe_psql('postgres', 'ALTER SUBSCRIPTION sub ENABLE'); + + $pub->safe_psql('postgres', + q{SELECT pg_stat_reset_replication_slot('sub')}); + + $pub->safe_psql( + 'postgres', qq{ +BEGIN; +INSERT INTO test SELECT generate_series(1000, 6000); +SELECT pg_logical_emit_message(true, 'streaming_$mode', 'streamed message'); +COMMIT; +}); + $pub->wait_for_catchup('sub'); + + # Make sure the transaction really was streamed, so that this test cannot + # silently degrade into a non-streaming one. + $pub->poll_query_until('postgres', + q{SELECT stream_txns > 0 FROM pg_stat_replication_slots WHERE slot_name = 'sub'} + ) or die "transaction was not streamed"; + + is(logged_count("streaming_$mode"), + 1, + "message in a streamed transaction is applied (streaming = $mode)"); + + $pub->safe_psql('postgres', 'TRUNCATE test'); + $pub->wait_for_catchup('sub'); +} + +$sub->safe_psql('postgres', 'ALTER SUBSCRIPTION sub SET (streaming = off)'); + +# Verify that ALTER SUBSCRIPTION ... SKIP skips a transactional message along +# with the rest of its transaction. The message is emitted before the +# conflicting change, so the hook runs and records the message on the first +# attempt; that work must be rolled back with the failed transaction, and must +# not be redone once the transaction is skipped. +$pub->safe_psql('postgres', 'INSERT INTO test VALUES (1)'); +$pub->wait_for_catchup('sub'); + +my $log_location = -s $sub->logfile; + +$pub->safe_psql( + 'postgres', q{ +BEGIN; +SELECT pg_logical_emit_message(true, 'skipped', 'rolled back, then skipped'); +INSERT INTO test VALUES (1); +COMMIT; +}); + +# Wait until the conflict disables the subscription. +$sub->poll_query_until('postgres', + q{SELECT subenabled = FALSE FROM pg_subscription WHERE subname = 'sub'}); + +# The hook did run before the transaction failed. +ok( $sub->log_contains( + qr/LOG[^\n]+received message: [^\n]+prefix: skipped/, + $log_location), + "hook is called for a message in a transaction that later fails"); + +# ... but its work was rolled back with the transaction. +is(logged_count('skipped'), 0, + "work done by the hook is rolled back with the remote transaction"); + +# Get the finish LSN of the failed transaction. +my $contents = slurp_file($sub->logfile, $log_location); +$contents =~ + qr/conflict detected on relation "public.test".*\n.*DETAIL:.*Could not apply remote change.*\n.*Key already exists in unique index "test_pkey", modified in transaction \d+: key .*, local row .*\n.*CONTEXT:.* for replication target relation "public.test" in transaction \d+, finished at ([[:xdigit:]]+\/[[:xdigit:]]+)/m + or die "could not get error-LSN"; +my $lsn = $1; + +$log_location = -s $sub->logfile; + +# Set skip LSN and re-enable the subscription. +$sub->safe_psql('postgres', qq{ALTER SUBSCRIPTION sub SKIP (lsn = '$lsn')}); +$sub->safe_psql('postgres', 'ALTER SUBSCRIPTION sub ENABLE'); + +# Wait for the failed transaction to be skipped. +$sub->poll_query_until('postgres', + q{SELECT subskiplsn = '0/0' FROM pg_subscription WHERE subname = 'sub'}); + +ok( !$sub->log_contains( + qr/LOG[^\n]+received message: [^\n]+prefix: skipped/, + $log_location), + "hook is not called for a message in a skipped transaction"); +is(logged_count('skipped'), 0, + "message in a skipped transaction is not applied"); + +$sub->stop; +$pub->stop; +done_testing(); diff --git a/src/test/modules/test_logicalmsg_hooks/test_logicalmsg_hooks.c b/src/test/modules/test_logicalmsg_hooks/test_logicalmsg_hooks.c new file mode 100644 index 00000000000..d8d6fc7c619 --- /dev/null +++ b/src/test/modules/test_logicalmsg_hooks/test_logicalmsg_hooks.c @@ -0,0 +1,140 @@ +/*-------------------------------------------------------------------------- + * + * test_logicalmsg_hooks.c + * Code for testing LogicalRepMessageHandle_hook + * + * The handler records every logical decoding message it receives into the + * table public.test_logicalmsg_log, so that tests can assert on the contents + * of a table rather than on the server log. Recording the messages this way + * also exercises the transactional behavior of the hook: a transactional + * message is recorded as part of the remote transaction that emitted it, and + * therefore disappears together with it if that transaction is rolled back or + * skipped, while a non-transactional message is recorded independently. + * + * The handler creates that table itself on the first message, so a test does + * not have to set it up. For a transactional message the creation is part of + * the remote transaction too, and is rolled back with it, so the table does + * not exist until some message has been applied successfully; a test must not + * query it before then. + * + * The messages are also logged, which lets a test tell "the handler ran but + * its work was rolled back" apart from "the handler was never called". + * + * Portions Copyright (c) 1996-2026, PostgreSQL Global Development Group + * Portions Copyright (c) 1994, Regents of the University of California + * + * IDENTIFICATION + * src/test/modules/test_logicalmsg_hooks/test_logicalmsg_hooks.c + * + * ------------------------------------------------------------------------- + */ + +#include "postgres.h" + +#include "catalog/namespace.h" +#include "catalog/pg_type.h" +#include "executor/spi.h" +#include "replication/logicalproto.h" +#include "replication/logicalworker.h" +#include "utils/builtins.h" +#include "utils/lsyscache.h" +#include "utils/pg_lsn.h" + +PG_MODULE_MAGIC; + +#define LOG_SCHEMA "public" +#define LOG_TABLE "test_logicalmsg_log" + +static LogicalRepMessageHandle_hook_type prev_logical_message_handler = NULL; +static void test_logical_message_handler(const LogicalRepMessageData *msg); + +/* + * Module load callback + */ +void +_PG_init(void) +{ + prev_logical_message_handler = LogicalRepMessageHandle_hook; + LogicalRepMessageHandle_hook = &test_logical_message_handler; +} + +/* + * Create the log table if not exists. + */ +static void +ensure_log_table_exists(void) +{ + Oid relid; + int ret; + + relid = get_relname_relid(LOG_TABLE, get_namespace_oid(LOG_SCHEMA, true)); + if (OidIsValid(relid)) + return; + + SPI_connect(); + + ret = SPI_execute("CREATE TABLE " LOG_SCHEMA "." LOG_TABLE + "(lsn pg_lsn, transactional boolean, prefix text, message_size int, message text)", + false, 0); + if (ret != SPI_OK_UTILITY) + elog(ERROR, "could not create " LOG_SCHEMA "." LOG_TABLE); + + SPI_finish(); +} + +static void +test_logical_message_handler(const LogicalRepMessageData *msg) +{ + Oid argtypes[5] = {LSNOID, BOOLOID, TEXTOID, INT4OID, TEXTOID}; + Datum values[5]; + int ret; + + /* + * Give a handler installed before ours its turn, so that several + * extensions can act on the same message. + */ + if (prev_logical_message_handler) + (*prev_logical_message_handler) (msg); + + ereport(LOG, + (errmsg("received message: LSN %X/%08X, prefix: %s, size: %zu, transactional: %d", + LSN_FORMAT_ARGS(msg->lsn), msg->prefix, + msg->message_size, msg->transactional))); + + ensure_log_table_exists(); + + /* + * Pass the message as query parameters rather than interpolating it into + * the query text. Message contents come from the publisher and are not to + * be trusted, and the payload may contain characters that would need + * quoting. Note also that message_size, not strlen(), is authoritative + * for the payload length. + * + * The payload is recorded as text because the accompanying test only ever + * emits text messages. A handler must not assume that in general: a + * payload is an arbitrary string of bytes, which may contain embedded + * nulls and bytes that are not valid in the subscriber's encoding. See + * the comments on LogicalRepMessageHandle_hook_type. + */ + values[0] = LSNGetDatum(msg->lsn); + values[1] = BoolGetDatum(msg->transactional); + values[2] = CStringGetTextDatum(msg->prefix); + values[3] = Int32GetDatum((int32) msg->message_size); + values[4] = PointerGetDatum(cstring_to_text_with_len(msg->message, + msg->message_size)); + + SPI_connect(); + + /* + * The apply worker runs with an empty search_path, so the table name must + * be schema-qualified. + */ + ret = SPI_execute_with_args("INSERT INTO " LOG_SCHEMA "." LOG_TABLE + " (lsn, transactional, prefix, message_size, message)" + " VALUES ($1, $2, $3, $4, $5)", + 5, argtypes, values, NULL, false, 0); + if (ret != SPI_OK_INSERT) + elog(ERROR, "could not record logical message: SPI returned %d", ret); + + SPI_finish(); +} diff --git a/src/test/regress/expected/subscription.out b/src/test/regress/expected/subscription.out index 259db747334..81d436edb8d 100644 --- a/src/test/regress/expected/subscription.out +++ b/src/test/regress/expected/subscription.out @@ -143,18 +143,18 @@ CREATE SUBSCRIPTION regress_testsub4 CONNECTION 'dbname=regress_doesnotexist' PU WARNING: subscription was created, but is not connected HINT: To initiate replication, you must manually create the replication slot, enable the subscription, and alter the subscription to refresh publications. \dRs+ regress_testsub4 - List of subscriptions - Name | Owner | Enabled | Publication | Binary | Streaming | Two-phase commit | Disable on error | Origin | Password required | Run as owner? | Failover | Server | Retain dead tuples | Max retention duration | Retention active | Synchronous commit | Conninfo | Receiver timeout | Skip LSN | Description -------------------+---------------------------+---------+-------------+--------+-----------+------------------+------------------+--------+-------------------+---------------+----------+--------+--------------------+------------------------+------------------+--------------------+-----------------------------+------------------+------------+------------- - regress_testsub4 | regress_subscription_user | f | {testpub} | f | parallel | d | f | none | t | f | f | | f | 0 | f | off | dbname=regress_doesnotexist | -1 | 0/00000000 | + List of subscriptions + Name | Owner | Enabled | Publication | Binary | Streaming | Two-phase commit | Disable on error | Origin | Password required | Run as owner? | Failover | Server | Retain dead tuples | Max retention duration | Retention active | Messages | Synchronous commit | Conninfo | Receiver timeout | Skip LSN | Description +------------------+---------------------------+---------+-------------+--------+-----------+------------------+------------------+--------+-------------------+---------------+----------+--------+--------------------+------------------------+------------------+----------+--------------------+-----------------------------+------------------+------------+------------- + regress_testsub4 | regress_subscription_user | f | {testpub} | f | parallel | d | f | none | t | f | f | | f | 0 | f | f | off | dbname=regress_doesnotexist | -1 | 0/00000000 | (1 row) ALTER SUBSCRIPTION regress_testsub4 SET (origin = any); \dRs+ regress_testsub4 - List of subscriptions - Name | Owner | Enabled | Publication | Binary | Streaming | Two-phase commit | Disable on error | Origin | Password required | Run as owner? | Failover | Server | Retain dead tuples | Max retention duration | Retention active | Synchronous commit | Conninfo | Receiver timeout | Skip LSN | Description -------------------+---------------------------+---------+-------------+--------+-----------+------------------+------------------+--------+-------------------+---------------+----------+--------+--------------------+------------------------+------------------+--------------------+-----------------------------+------------------+------------+------------- - regress_testsub4 | regress_subscription_user | f | {testpub} | f | parallel | d | f | any | t | f | f | | f | 0 | f | off | dbname=regress_doesnotexist | -1 | 0/00000000 | + List of subscriptions + Name | Owner | Enabled | Publication | Binary | Streaming | Two-phase commit | Disable on error | Origin | Password required | Run as owner? | Failover | Server | Retain dead tuples | Max retention duration | Retention active | Messages | Synchronous commit | Conninfo | Receiver timeout | Skip LSN | Description +------------------+---------------------------+---------+-------------+--------+-----------+------------------+------------------+--------+-------------------+---------------+----------+--------+--------------------+------------------------+------------------+----------+--------------------+-----------------------------+------------------+------------+------------- + regress_testsub4 | regress_subscription_user | f | {testpub} | f | parallel | d | f | any | t | f | f | | f | 0 | f | f | off | dbname=regress_doesnotexist | -1 | 0/00000000 | (1 row) DROP SUBSCRIPTION regress_testsub3; @@ -258,10 +258,10 @@ ALTER SUBSCRIPTION regress_testsub CONNECTION 'foobar'; ERROR: invalid connection string syntax: missing "=" after "foobar" in connection info string \dRs+ - List of subscriptions - Name | Owner | Enabled | Publication | Binary | Streaming | Two-phase commit | Disable on error | Origin | Password required | Run as owner? | Failover | Server | Retain dead tuples | Max retention duration | Retention active | Synchronous commit | Conninfo | Receiver timeout | Skip LSN | Description ------------------+---------------------------+---------+-------------+--------+-----------+------------------+------------------+--------+-------------------+---------------+----------+--------+--------------------+------------------------+------------------+--------------------+-----------------------------+------------------+------------+------------------- - regress_testsub | regress_subscription_user | f | {testpub} | f | parallel | d | f | any | t | f | f | | f | 0 | f | off | dbname=regress_doesnotexist | -1 | 0/00000000 | test subscription + List of subscriptions + Name | Owner | Enabled | Publication | Binary | Streaming | Two-phase commit | Disable on error | Origin | Password required | Run as owner? | Failover | Server | Retain dead tuples | Max retention duration | Retention active | Messages | Synchronous commit | Conninfo | Receiver timeout | Skip LSN | Description +-----------------+---------------------------+---------+-------------+--------+-----------+------------------+------------------+--------+-------------------+---------------+----------+--------+--------------------+------------------------+------------------+----------+--------------------+-----------------------------+------------------+------------+------------------- + regress_testsub | regress_subscription_user | f | {testpub} | f | parallel | d | f | any | t | f | f | | f | 0 | f | f | off | dbname=regress_doesnotexist | -1 | 0/00000000 | test subscription (1 row) ALTER SUBSCRIPTION regress_testsub SET PUBLICATION testpub2, testpub3 WITH (refresh = false); @@ -270,10 +270,10 @@ ALTER SUBSCRIPTION regress_testsub SET (slot_name = 'newname'); ALTER SUBSCRIPTION regress_testsub SET (password_required = false); ALTER SUBSCRIPTION regress_testsub SET (run_as_owner = true); \dRs+ - List of subscriptions - Name | Owner | Enabled | Publication | Binary | Streaming | Two-phase commit | Disable on error | Origin | Password required | Run as owner? | Failover | Server | Retain dead tuples | Max retention duration | Retention active | Synchronous commit | Conninfo | Receiver timeout | Skip LSN | Description ------------------+---------------------------+---------+---------------------+--------+-----------+------------------+------------------+--------+-------------------+---------------+----------+--------+--------------------+------------------------+------------------+--------------------+------------------------------+------------------+------------+------------------- - regress_testsub | regress_subscription_user | f | {testpub2,testpub3} | f | parallel | d | f | any | f | t | f | | f | 0 | f | off | dbname=regress_doesnotexist2 | -1 | 0/00000000 | test subscription + List of subscriptions + Name | Owner | Enabled | Publication | Binary | Streaming | Two-phase commit | Disable on error | Origin | Password required | Run as owner? | Failover | Server | Retain dead tuples | Max retention duration | Retention active | Messages | Synchronous commit | Conninfo | Receiver timeout | Skip LSN | Description +-----------------+---------------------------+---------+---------------------+--------+-----------+------------------+------------------+--------+-------------------+---------------+----------+--------+--------------------+------------------------+------------------+----------+--------------------+------------------------------+------------------+------------+------------------- + regress_testsub | regress_subscription_user | f | {testpub2,testpub3} | f | parallel | d | f | any | f | t | f | | f | 0 | f | f | off | dbname=regress_doesnotexist2 | -1 | 0/00000000 | test subscription (1 row) ALTER SUBSCRIPTION regress_testsub SET (password_required = true); @@ -289,10 +289,10 @@ ERROR: unrecognized subscription parameter: "create_slot" -- ok ALTER SUBSCRIPTION regress_testsub SKIP (lsn = '0/12345'); \dRs+ - List of subscriptions - Name | Owner | Enabled | Publication | Binary | Streaming | Two-phase commit | Disable on error | Origin | Password required | Run as owner? | Failover | Server | Retain dead tuples | Max retention duration | Retention active | Synchronous commit | Conninfo | Receiver timeout | Skip LSN | Description ------------------+---------------------------+---------+---------------------+--------+-----------+------------------+------------------+--------+-------------------+---------------+----------+--------+--------------------+------------------------+------------------+--------------------+------------------------------+------------------+------------+------------------- - regress_testsub | regress_subscription_user | f | {testpub2,testpub3} | f | parallel | d | f | any | t | f | f | | f | 0 | f | off | dbname=regress_doesnotexist2 | -1 | 0/00012345 | test subscription + List of subscriptions + Name | Owner | Enabled | Publication | Binary | Streaming | Two-phase commit | Disable on error | Origin | Password required | Run as owner? | Failover | Server | Retain dead tuples | Max retention duration | Retention active | Messages | Synchronous commit | Conninfo | Receiver timeout | Skip LSN | Description +-----------------+---------------------------+---------+---------------------+--------+-----------+------------------+------------------+--------+-------------------+---------------+----------+--------+--------------------+------------------------+------------------+----------+--------------------+------------------------------+------------------+------------+------------------- + regress_testsub | regress_subscription_user | f | {testpub2,testpub3} | f | parallel | d | f | any | t | f | f | | f | 0 | f | f | off | dbname=regress_doesnotexist2 | -1 | 0/00012345 | test subscription (1 row) -- ok - with lsn = NONE @@ -301,10 +301,10 @@ ALTER SUBSCRIPTION regress_testsub SKIP (lsn = NONE); ALTER SUBSCRIPTION regress_testsub SKIP (lsn = '0/0'); ERROR: invalid WAL location (LSN): 0/0 \dRs+ - List of subscriptions - Name | Owner | Enabled | Publication | Binary | Streaming | Two-phase commit | Disable on error | Origin | Password required | Run as owner? | Failover | Server | Retain dead tuples | Max retention duration | Retention active | Synchronous commit | Conninfo | Receiver timeout | Skip LSN | Description ------------------+---------------------------+---------+---------------------+--------+-----------+------------------+------------------+--------+-------------------+---------------+----------+--------+--------------------+------------------------+------------------+--------------------+------------------------------+------------------+------------+------------------- - regress_testsub | regress_subscription_user | f | {testpub2,testpub3} | f | parallel | d | f | any | t | f | f | | f | 0 | f | off | dbname=regress_doesnotexist2 | -1 | 0/00000000 | test subscription + List of subscriptions + Name | Owner | Enabled | Publication | Binary | Streaming | Two-phase commit | Disable on error | Origin | Password required | Run as owner? | Failover | Server | Retain dead tuples | Max retention duration | Retention active | Messages | Synchronous commit | Conninfo | Receiver timeout | Skip LSN | Description +-----------------+---------------------------+---------+---------------------+--------+-----------+------------------+------------------+--------+-------------------+---------------+----------+--------+--------------------+------------------------+------------------+----------+--------------------+------------------------------+------------------+------------+------------------- + regress_testsub | regress_subscription_user | f | {testpub2,testpub3} | f | parallel | d | f | any | t | f | f | | f | 0 | f | f | off | dbname=regress_doesnotexist2 | -1 | 0/00000000 | test subscription (1 row) BEGIN; @@ -340,10 +340,10 @@ ALTER SUBSCRIPTION regress_testsub_foo SET (wal_receiver_timeout = '80s'); ALTER SUBSCRIPTION regress_testsub_foo SET (wal_receiver_timeout = 'foobar'); ERROR: invalid value for parameter "wal_receiver_timeout": "foobar" \dRs+ - List of subscriptions - Name | Owner | Enabled | Publication | Binary | Streaming | Two-phase commit | Disable on error | Origin | Password required | Run as owner? | Failover | Server | Retain dead tuples | Max retention duration | Retention active | Synchronous commit | Conninfo | Receiver timeout | Skip LSN | Description ----------------------+---------------------------+---------+---------------------+--------+-----------+------------------+------------------+--------+-------------------+---------------+----------+--------+--------------------+------------------------+------------------+--------------------+------------------------------+------------------+------------+------------------- - regress_testsub_foo | regress_subscription_user | f | {testpub2,testpub3} | f | parallel | d | f | any | t | f | f | | f | 0 | f | local | dbname=regress_doesnotexist2 | 80s | 0/00000000 | test subscription + List of subscriptions + Name | Owner | Enabled | Publication | Binary | Streaming | Two-phase commit | Disable on error | Origin | Password required | Run as owner? | Failover | Server | Retain dead tuples | Max retention duration | Retention active | Messages | Synchronous commit | Conninfo | Receiver timeout | Skip LSN | Description +---------------------+---------------------------+---------+---------------------+--------+-----------+------------------+------------------+--------+-------------------+---------------+----------+--------+--------------------+------------------------+------------------+----------+--------------------+------------------------------+------------------+------------+------------------- + regress_testsub_foo | regress_subscription_user | f | {testpub2,testpub3} | f | parallel | d | f | any | t | f | f | | f | 0 | f | f | local | dbname=regress_doesnotexist2 | 80s | 0/00000000 | test subscription (1 row) -- rename back to keep the rest simple @@ -372,19 +372,19 @@ CREATE SUBSCRIPTION regress_testsub CONNECTION 'dbname=regress_doesnotexist' PUB WARNING: subscription was created, but is not connected HINT: To initiate replication, you must manually create the replication slot, enable the subscription, and alter the subscription to refresh publications. \dRs+ - List of subscriptions - Name | Owner | Enabled | Publication | Binary | Streaming | Two-phase commit | Disable on error | Origin | Password required | Run as owner? | Failover | Server | Retain dead tuples | Max retention duration | Retention active | Synchronous commit | Conninfo | Receiver timeout | Skip LSN | Description ------------------+---------------------------+---------+-------------+--------+-----------+------------------+------------------+--------+-------------------+---------------+----------+--------+--------------------+------------------------+------------------+--------------------+-----------------------------+------------------+------------+------------- - regress_testsub | regress_subscription_user | f | {testpub} | t | parallel | d | f | any | t | f | f | | f | 0 | f | off | dbname=regress_doesnotexist | -1 | 0/00000000 | + List of subscriptions + Name | Owner | Enabled | Publication | Binary | Streaming | Two-phase commit | Disable on error | Origin | Password required | Run as owner? | Failover | Server | Retain dead tuples | Max retention duration | Retention active | Messages | Synchronous commit | Conninfo | Receiver timeout | Skip LSN | Description +-----------------+---------------------------+---------+-------------+--------+-----------+------------------+------------------+--------+-------------------+---------------+----------+--------+--------------------+------------------------+------------------+----------+--------------------+-----------------------------+------------------+------------+------------- + regress_testsub | regress_subscription_user | f | {testpub} | t | parallel | d | f | any | t | f | f | | f | 0 | f | f | off | dbname=regress_doesnotexist | -1 | 0/00000000 | (1 row) ALTER SUBSCRIPTION regress_testsub SET (binary = false); ALTER SUBSCRIPTION regress_testsub SET (slot_name = NONE); \dRs+ - List of subscriptions - Name | Owner | Enabled | Publication | Binary | Streaming | Two-phase commit | Disable on error | Origin | Password required | Run as owner? | Failover | Server | Retain dead tuples | Max retention duration | Retention active | Synchronous commit | Conninfo | Receiver timeout | Skip LSN | Description ------------------+---------------------------+---------+-------------+--------+-----------+------------------+------------------+--------+-------------------+---------------+----------+--------+--------------------+------------------------+------------------+--------------------+-----------------------------+------------------+------------+------------- - regress_testsub | regress_subscription_user | f | {testpub} | f | parallel | d | f | any | t | f | f | | f | 0 | f | off | dbname=regress_doesnotexist | -1 | 0/00000000 | + List of subscriptions + Name | Owner | Enabled | Publication | Binary | Streaming | Two-phase commit | Disable on error | Origin | Password required | Run as owner? | Failover | Server | Retain dead tuples | Max retention duration | Retention active | Messages | Synchronous commit | Conninfo | Receiver timeout | Skip LSN | Description +-----------------+---------------------------+---------+-------------+--------+-----------+------------------+------------------+--------+-------------------+---------------+----------+--------+--------------------+------------------------+------------------+----------+--------------------+-----------------------------+------------------+------------+------------- + regress_testsub | regress_subscription_user | f | {testpub} | f | parallel | d | f | any | t | f | f | | f | 0 | f | f | off | dbname=regress_doesnotexist | -1 | 0/00000000 | (1 row) DROP SUBSCRIPTION regress_testsub; @@ -396,27 +396,27 @@ CREATE SUBSCRIPTION regress_testsub CONNECTION 'dbname=regress_doesnotexist' PUB WARNING: subscription was created, but is not connected HINT: To initiate replication, you must manually create the replication slot, enable the subscription, and alter the subscription to refresh publications. \dRs+ - List of subscriptions - Name | Owner | Enabled | Publication | Binary | Streaming | Two-phase commit | Disable on error | Origin | Password required | Run as owner? | Failover | Server | Retain dead tuples | Max retention duration | Retention active | Synchronous commit | Conninfo | Receiver timeout | Skip LSN | Description ------------------+---------------------------+---------+-------------+--------+-----------+------------------+------------------+--------+-------------------+---------------+----------+--------+--------------------+------------------------+------------------+--------------------+-----------------------------+------------------+------------+------------- - regress_testsub | regress_subscription_user | f | {testpub} | f | on | d | f | any | t | f | f | | f | 0 | f | off | dbname=regress_doesnotexist | -1 | 0/00000000 | + List of subscriptions + Name | Owner | Enabled | Publication | Binary | Streaming | Two-phase commit | Disable on error | Origin | Password required | Run as owner? | Failover | Server | Retain dead tuples | Max retention duration | Retention active | Messages | Synchronous commit | Conninfo | Receiver timeout | Skip LSN | Description +-----------------+---------------------------+---------+-------------+--------+-----------+------------------+------------------+--------+-------------------+---------------+----------+--------+--------------------+------------------------+------------------+----------+--------------------+-----------------------------+------------------+------------+------------- + regress_testsub | regress_subscription_user | f | {testpub} | f | on | d | f | any | t | f | f | | f | 0 | f | f | off | dbname=regress_doesnotexist | -1 | 0/00000000 | (1 row) ALTER SUBSCRIPTION regress_testsub SET (streaming = parallel); \dRs+ - List of subscriptions - Name | Owner | Enabled | Publication | Binary | Streaming | Two-phase commit | Disable on error | Origin | Password required | Run as owner? | Failover | Server | Retain dead tuples | Max retention duration | Retention active | Synchronous commit | Conninfo | Receiver timeout | Skip LSN | Description ------------------+---------------------------+---------+-------------+--------+-----------+------------------+------------------+--------+-------------------+---------------+----------+--------+--------------------+------------------------+------------------+--------------------+-----------------------------+------------------+------------+------------- - regress_testsub | regress_subscription_user | f | {testpub} | f | parallel | d | f | any | t | f | f | | f | 0 | f | off | dbname=regress_doesnotexist | -1 | 0/00000000 | + List of subscriptions + Name | Owner | Enabled | Publication | Binary | Streaming | Two-phase commit | Disable on error | Origin | Password required | Run as owner? | Failover | Server | Retain dead tuples | Max retention duration | Retention active | Messages | Synchronous commit | Conninfo | Receiver timeout | Skip LSN | Description +-----------------+---------------------------+---------+-------------+--------+-----------+------------------+------------------+--------+-------------------+---------------+----------+--------+--------------------+------------------------+------------------+----------+--------------------+-----------------------------+------------------+------------+------------- + regress_testsub | regress_subscription_user | f | {testpub} | f | parallel | d | f | any | t | f | f | | f | 0 | f | f | off | dbname=regress_doesnotexist | -1 | 0/00000000 | (1 row) ALTER SUBSCRIPTION regress_testsub SET (streaming = false); ALTER SUBSCRIPTION regress_testsub SET (slot_name = NONE); \dRs+ - List of subscriptions - Name | Owner | Enabled | Publication | Binary | Streaming | Two-phase commit | Disable on error | Origin | Password required | Run as owner? | Failover | Server | Retain dead tuples | Max retention duration | Retention active | Synchronous commit | Conninfo | Receiver timeout | Skip LSN | Description ------------------+---------------------------+---------+-------------+--------+-----------+------------------+------------------+--------+-------------------+---------------+----------+--------+--------------------+------------------------+------------------+--------------------+-----------------------------+------------------+------------+------------- - regress_testsub | regress_subscription_user | f | {testpub} | f | off | d | f | any | t | f | f | | f | 0 | f | off | dbname=regress_doesnotexist | -1 | 0/00000000 | + List of subscriptions + Name | Owner | Enabled | Publication | Binary | Streaming | Two-phase commit | Disable on error | Origin | Password required | Run as owner? | Failover | Server | Retain dead tuples | Max retention duration | Retention active | Messages | Synchronous commit | Conninfo | Receiver timeout | Skip LSN | Description +-----------------+---------------------------+---------+-------------+--------+-----------+------------------+------------------+--------+-------------------+---------------+----------+--------+--------------------+------------------------+------------------+----------+--------------------+-----------------------------+------------------+------------+------------- + regress_testsub | regress_subscription_user | f | {testpub} | f | off | d | f | any | t | f | f | | f | 0 | f | f | off | dbname=regress_doesnotexist | -1 | 0/00000000 | (1 row) -- fail - publication already exists @@ -431,10 +431,10 @@ ALTER SUBSCRIPTION regress_testsub ADD PUBLICATION testpub1, testpub2 WITH (refr ALTER SUBSCRIPTION regress_testsub ADD PUBLICATION testpub1, testpub2 WITH (refresh = false); ERROR: publication "testpub1" is already in subscription "regress_testsub" \dRs+ - List of subscriptions - Name | Owner | Enabled | Publication | Binary | Streaming | Two-phase commit | Disable on error | Origin | Password required | Run as owner? | Failover | Server | Retain dead tuples | Max retention duration | Retention active | Synchronous commit | Conninfo | Receiver timeout | Skip LSN | Description ------------------+---------------------------+---------+-----------------------------+--------+-----------+------------------+------------------+--------+-------------------+---------------+----------+--------+--------------------+------------------------+------------------+--------------------+-----------------------------+------------------+------------+------------- - regress_testsub | regress_subscription_user | f | {testpub,testpub1,testpub2} | f | off | d | f | any | t | f | f | | f | 0 | f | off | dbname=regress_doesnotexist | -1 | 0/00000000 | + List of subscriptions + Name | Owner | Enabled | Publication | Binary | Streaming | Two-phase commit | Disable on error | Origin | Password required | Run as owner? | Failover | Server | Retain dead tuples | Max retention duration | Retention active | Messages | Synchronous commit | Conninfo | Receiver timeout | Skip LSN | Description +-----------------+---------------------------+---------+-----------------------------+--------+-----------+------------------+------------------+--------+-------------------+---------------+----------+--------+--------------------+------------------------+------------------+----------+--------------------+-----------------------------+------------------+------------+------------- + regress_testsub | regress_subscription_user | f | {testpub,testpub1,testpub2} | f | off | d | f | any | t | f | f | | f | 0 | f | f | off | dbname=regress_doesnotexist | -1 | 0/00000000 | (1 row) -- fail - publication used more than once @@ -449,10 +449,10 @@ ERROR: publication "testpub3" is not in subscription "regress_testsub" -- ok - delete publications ALTER SUBSCRIPTION regress_testsub DROP PUBLICATION testpub1, testpub2 WITH (refresh = false); \dRs+ - List of subscriptions - Name | Owner | Enabled | Publication | Binary | Streaming | Two-phase commit | Disable on error | Origin | Password required | Run as owner? | Failover | Server | Retain dead tuples | Max retention duration | Retention active | Synchronous commit | Conninfo | Receiver timeout | Skip LSN | Description ------------------+---------------------------+---------+-------------+--------+-----------+------------------+------------------+--------+-------------------+---------------+----------+--------+--------------------+------------------------+------------------+--------------------+-----------------------------+------------------+------------+------------- - regress_testsub | regress_subscription_user | f | {testpub} | f | off | d | f | any | t | f | f | | f | 0 | f | off | dbname=regress_doesnotexist | -1 | 0/00000000 | + List of subscriptions + Name | Owner | Enabled | Publication | Binary | Streaming | Two-phase commit | Disable on error | Origin | Password required | Run as owner? | Failover | Server | Retain dead tuples | Max retention duration | Retention active | Messages | Synchronous commit | Conninfo | Receiver timeout | Skip LSN | Description +-----------------+---------------------------+---------+-------------+--------+-----------+------------------+------------------+--------+-------------------+---------------+----------+--------+--------------------+------------------------+------------------+----------+--------------------+-----------------------------+------------------+------------+------------- + regress_testsub | regress_subscription_user | f | {testpub} | f | off | d | f | any | t | f | f | | f | 0 | f | f | off | dbname=regress_doesnotexist | -1 | 0/00000000 | (1 row) DROP SUBSCRIPTION regress_testsub; @@ -488,19 +488,19 @@ CREATE SUBSCRIPTION regress_testsub CONNECTION 'dbname=regress_doesnotexist' PUB WARNING: subscription was created, but is not connected HINT: To initiate replication, you must manually create the replication slot, enable the subscription, and alter the subscription to refresh publications. \dRs+ - List of subscriptions - Name | Owner | Enabled | Publication | Binary | Streaming | Two-phase commit | Disable on error | Origin | Password required | Run as owner? | Failover | Server | Retain dead tuples | Max retention duration | Retention active | Synchronous commit | Conninfo | Receiver timeout | Skip LSN | Description ------------------+---------------------------+---------+-------------+--------+-----------+------------------+------------------+--------+-------------------+---------------+----------+--------+--------------------+------------------------+------------------+--------------------+-----------------------------+------------------+------------+------------- - regress_testsub | regress_subscription_user | f | {testpub} | f | parallel | p | f | any | t | f | f | | f | 0 | f | off | dbname=regress_doesnotexist | -1 | 0/00000000 | + List of subscriptions + Name | Owner | Enabled | Publication | Binary | Streaming | Two-phase commit | Disable on error | Origin | Password required | Run as owner? | Failover | Server | Retain dead tuples | Max retention duration | Retention active | Messages | Synchronous commit | Conninfo | Receiver timeout | Skip LSN | Description +-----------------+---------------------------+---------+-------------+--------+-----------+------------------+------------------+--------+-------------------+---------------+----------+--------+--------------------+------------------------+------------------+----------+--------------------+-----------------------------+------------------+------------+------------- + regress_testsub | regress_subscription_user | f | {testpub} | f | parallel | p | f | any | t | f | f | | f | 0 | f | f | off | dbname=regress_doesnotexist | -1 | 0/00000000 | (1 row) -- we can alter streaming when two_phase enabled ALTER SUBSCRIPTION regress_testsub SET (streaming = true); \dRs+ - List of subscriptions - Name | Owner | Enabled | Publication | Binary | Streaming | Two-phase commit | Disable on error | Origin | Password required | Run as owner? | Failover | Server | Retain dead tuples | Max retention duration | Retention active | Synchronous commit | Conninfo | Receiver timeout | Skip LSN | Description ------------------+---------------------------+---------+-------------+--------+-----------+------------------+------------------+--------+-------------------+---------------+----------+--------+--------------------+------------------------+------------------+--------------------+-----------------------------+------------------+------------+------------- - regress_testsub | regress_subscription_user | f | {testpub} | f | on | p | f | any | t | f | f | | f | 0 | f | off | dbname=regress_doesnotexist | -1 | 0/00000000 | + List of subscriptions + Name | Owner | Enabled | Publication | Binary | Streaming | Two-phase commit | Disable on error | Origin | Password required | Run as owner? | Failover | Server | Retain dead tuples | Max retention duration | Retention active | Messages | Synchronous commit | Conninfo | Receiver timeout | Skip LSN | Description +-----------------+---------------------------+---------+-------------+--------+-----------+------------------+------------------+--------+-------------------+---------------+----------+--------+--------------------+------------------------+------------------+----------+--------------------+-----------------------------+------------------+------------+------------- + regress_testsub | regress_subscription_user | f | {testpub} | f | on | p | f | any | t | f | f | | f | 0 | f | f | off | dbname=regress_doesnotexist | -1 | 0/00000000 | (1 row) ALTER SUBSCRIPTION regress_testsub SET (slot_name = NONE); @@ -510,10 +510,10 @@ CREATE SUBSCRIPTION regress_testsub CONNECTION 'dbname=regress_doesnotexist' PUB WARNING: subscription was created, but is not connected HINT: To initiate replication, you must manually create the replication slot, enable the subscription, and alter the subscription to refresh publications. \dRs+ - List of subscriptions - Name | Owner | Enabled | Publication | Binary | Streaming | Two-phase commit | Disable on error | Origin | Password required | Run as owner? | Failover | Server | Retain dead tuples | Max retention duration | Retention active | Synchronous commit | Conninfo | Receiver timeout | Skip LSN | Description ------------------+---------------------------+---------+-------------+--------+-----------+------------------+------------------+--------+-------------------+---------------+----------+--------+--------------------+------------------------+------------------+--------------------+-----------------------------+------------------+------------+------------- - regress_testsub | regress_subscription_user | f | {testpub} | f | on | p | f | any | t | f | f | | f | 0 | f | off | dbname=regress_doesnotexist | -1 | 0/00000000 | + List of subscriptions + Name | Owner | Enabled | Publication | Binary | Streaming | Two-phase commit | Disable on error | Origin | Password required | Run as owner? | Failover | Server | Retain dead tuples | Max retention duration | Retention active | Messages | Synchronous commit | Conninfo | Receiver timeout | Skip LSN | Description +-----------------+---------------------------+---------+-------------+--------+-----------+------------------+------------------+--------+-------------------+---------------+----------+--------+--------------------+------------------------+------------------+----------+--------------------+-----------------------------+------------------+------------+------------- + regress_testsub | regress_subscription_user | f | {testpub} | f | on | p | f | any | t | f | f | | f | 0 | f | f | off | dbname=regress_doesnotexist | -1 | 0/00000000 | (1 row) ALTER SUBSCRIPTION regress_testsub SET (slot_name = NONE); @@ -526,18 +526,18 @@ CREATE SUBSCRIPTION regress_testsub CONNECTION 'dbname=regress_doesnotexist' PUB WARNING: subscription was created, but is not connected HINT: To initiate replication, you must manually create the replication slot, enable the subscription, and alter the subscription to refresh publications. \dRs+ - List of subscriptions - Name | Owner | Enabled | Publication | Binary | Streaming | Two-phase commit | Disable on error | Origin | Password required | Run as owner? | Failover | Server | Retain dead tuples | Max retention duration | Retention active | Synchronous commit | Conninfo | Receiver timeout | Skip LSN | Description ------------------+---------------------------+---------+-------------+--------+-----------+------------------+------------------+--------+-------------------+---------------+----------+--------+--------------------+------------------------+------------------+--------------------+-----------------------------+------------------+------------+------------- - regress_testsub | regress_subscription_user | f | {testpub} | f | parallel | d | f | any | t | f | f | | f | 0 | f | off | dbname=regress_doesnotexist | -1 | 0/00000000 | + List of subscriptions + Name | Owner | Enabled | Publication | Binary | Streaming | Two-phase commit | Disable on error | Origin | Password required | Run as owner? | Failover | Server | Retain dead tuples | Max retention duration | Retention active | Messages | Synchronous commit | Conninfo | Receiver timeout | Skip LSN | Description +-----------------+---------------------------+---------+-------------+--------+-----------+------------------+------------------+--------+-------------------+---------------+----------+--------+--------------------+------------------------+------------------+----------+--------------------+-----------------------------+------------------+------------+------------- + regress_testsub | regress_subscription_user | f | {testpub} | f | parallel | d | f | any | t | f | f | | f | 0 | f | f | off | dbname=regress_doesnotexist | -1 | 0/00000000 | (1 row) ALTER SUBSCRIPTION regress_testsub SET (disable_on_error = true); \dRs+ - List of subscriptions - Name | Owner | Enabled | Publication | Binary | Streaming | Two-phase commit | Disable on error | Origin | Password required | Run as owner? | Failover | Server | Retain dead tuples | Max retention duration | Retention active | Synchronous commit | Conninfo | Receiver timeout | Skip LSN | Description ------------------+---------------------------+---------+-------------+--------+-----------+------------------+------------------+--------+-------------------+---------------+----------+--------+--------------------+------------------------+------------------+--------------------+-----------------------------+------------------+------------+------------- - regress_testsub | regress_subscription_user | f | {testpub} | f | parallel | d | t | any | t | f | f | | f | 0 | f | off | dbname=regress_doesnotexist | -1 | 0/00000000 | + List of subscriptions + Name | Owner | Enabled | Publication | Binary | Streaming | Two-phase commit | Disable on error | Origin | Password required | Run as owner? | Failover | Server | Retain dead tuples | Max retention duration | Retention active | Messages | Synchronous commit | Conninfo | Receiver timeout | Skip LSN | Description +-----------------+---------------------------+---------+-------------+--------+-----------+------------------+------------------+--------+-------------------+---------------+----------+--------+--------------------+------------------------+------------------+----------+--------------------+-----------------------------+------------------+------------+------------- + regress_testsub | regress_subscription_user | f | {testpub} | f | parallel | d | t | any | t | f | f | | f | 0 | f | f | off | dbname=regress_doesnotexist | -1 | 0/00000000 | (1 row) ALTER SUBSCRIPTION regress_testsub SET (slot_name = NONE); @@ -550,10 +550,10 @@ CREATE SUBSCRIPTION regress_testsub CONNECTION 'dbname=regress_doesnotexist' PUB WARNING: subscription was created, but is not connected HINT: To initiate replication, you must manually create the replication slot, enable the subscription, and alter the subscription to refresh publications. \dRs+ - List of subscriptions - Name | Owner | Enabled | Publication | Binary | Streaming | Two-phase commit | Disable on error | Origin | Password required | Run as owner? | Failover | Server | Retain dead tuples | Max retention duration | Retention active | Synchronous commit | Conninfo | Receiver timeout | Skip LSN | Description ------------------+---------------------------+---------+-------------+--------+-----------+------------------+------------------+--------+-------------------+---------------+----------+--------+--------------------+------------------------+------------------+--------------------+-----------------------------+------------------+------------+------------- - regress_testsub | regress_subscription_user | f | {testpub} | f | parallel | d | f | any | t | f | f | | f | 0 | f | off | dbname=regress_doesnotexist | -1 | 0/00000000 | + List of subscriptions + Name | Owner | Enabled | Publication | Binary | Streaming | Two-phase commit | Disable on error | Origin | Password required | Run as owner? | Failover | Server | Retain dead tuples | Max retention duration | Retention active | Messages | Synchronous commit | Conninfo | Receiver timeout | Skip LSN | Description +-----------------+---------------------------+---------+-------------+--------+-----------+------------------+------------------+--------+-------------------+---------------+----------+--------+--------------------+------------------------+------------------+----------+--------------------+-----------------------------+------------------+------------+------------- + regress_testsub | regress_subscription_user | f | {testpub} | f | parallel | d | f | any | t | f | f | | f | 0 | f | f | off | dbname=regress_doesnotexist | -1 | 0/00000000 | (1 row) ALTER SUBSCRIPTION regress_testsub SET (slot_name = NONE); @@ -570,10 +570,10 @@ NOTICE: max_retention_duration is ineffective when retain_dead_tuples is disabl WARNING: subscription was created, but is not connected HINT: To initiate replication, you must manually create the replication slot, enable the subscription, and alter the subscription to refresh publications. \dRs+ - List of subscriptions - Name | Owner | Enabled | Publication | Binary | Streaming | Two-phase commit | Disable on error | Origin | Password required | Run as owner? | Failover | Server | Retain dead tuples | Max retention duration | Retention active | Synchronous commit | Conninfo | Receiver timeout | Skip LSN | Description ------------------+---------------------------+---------+-------------+--------+-----------+------------------+------------------+--------+-------------------+---------------+----------+--------+--------------------+------------------------+------------------+--------------------+-----------------------------+------------------+------------+------------- - regress_testsub | regress_subscription_user | f | {testpub} | f | parallel | d | f | any | t | f | f | | f | 1000 | f | off | dbname=regress_doesnotexist | -1 | 0/00000000 | + List of subscriptions + Name | Owner | Enabled | Publication | Binary | Streaming | Two-phase commit | Disable on error | Origin | Password required | Run as owner? | Failover | Server | Retain dead tuples | Max retention duration | Retention active | Messages | Synchronous commit | Conninfo | Receiver timeout | Skip LSN | Description +-----------------+---------------------------+---------+-------------+--------+-----------+------------------+------------------+--------+-------------------+---------------+----------+--------+--------------------+------------------------+------------------+----------+--------------------+-----------------------------+------------------+------------+------------- + regress_testsub | regress_subscription_user | f | {testpub} | f | parallel | d | f | any | t | f | f | | f | 1000 | f | f | off | dbname=regress_doesnotexist | -1 | 0/00000000 | (1 row) -- fail - max_retention_duration must be non-negative @@ -582,12 +582,38 @@ ERROR: max_retention_duration cannot be negative -- ok ALTER SUBSCRIPTION regress_testsub SET (max_retention_duration = 0); \dRs+ - List of subscriptions - Name | Owner | Enabled | Publication | Binary | Streaming | Two-phase commit | Disable on error | Origin | Password required | Run as owner? | Failover | Server | Retain dead tuples | Max retention duration | Retention active | Synchronous commit | Conninfo | Receiver timeout | Skip LSN | Description ------------------+---------------------------+---------+-------------+--------+-----------+------------------+------------------+--------+-------------------+---------------+----------+--------+--------------------+------------------------+------------------+--------------------+-----------------------------+------------------+------------+------------- - regress_testsub | regress_subscription_user | f | {testpub} | f | parallel | d | f | any | t | f | f | | f | 0 | f | off | dbname=regress_doesnotexist | -1 | 0/00000000 | + List of subscriptions + Name | Owner | Enabled | Publication | Binary | Streaming | Two-phase commit | Disable on error | Origin | Password required | Run as owner? | Failover | Server | Retain dead tuples | Max retention duration | Retention active | Messages | Synchronous commit | Conninfo | Receiver timeout | Skip LSN | Description +-----------------+---------------------------+---------+-------------+--------+-----------+------------------+------------------+--------+-------------------+---------------+----------+--------+--------------------+------------------------+------------------+----------+--------------------+-----------------------------+------------------+------------+------------- + regress_testsub | regress_subscription_user | f | {testpub} | f | parallel | d | f | any | t | f | f | | f | 0 | f | f | off | dbname=regress_doesnotexist | -1 | 0/00000000 | +(1 row) + +ALTER SUBSCRIPTION regress_testsub SET (slot_name = NONE); +DROP SUBSCRIPTION regress_testsub; +-- fail - messages must be boolean +CREATE SUBSCRIPTION regress_testsub CONNECTION 'dbname=regress_doesnotexist' PUBLICATION testpub WITH (connect = false, messages = foo); +ERROR: messages requires a Boolean value +-- ok +CREATE SUBSCRIPTION regress_testsub CONNECTION 'dbname=regress_doesnotexist' PUBLICATION testpub WITH (connect = false, messages = true); +WARNING: subscription was created, but is not connected +HINT: To initiate replication, you must manually create the replication slot, enable the subscription, and alter the subscription to refresh publications. +\dRs+ + List of subscriptions + Name | Owner | Enabled | Publication | Binary | Streaming | Two-phase commit | Disable on error | Origin | Password required | Run as owner? | Failover | Server | Retain dead tuples | Max retention duration | Retention active | Messages | Synchronous commit | Conninfo | Receiver timeout | Skip LSN | Description +-----------------+---------------------------+---------+-------------+--------+-----------+------------------+------------------+--------+-------------------+---------------+----------+--------+--------------------+------------------------+------------------+----------+--------------------+-----------------------------+------------------+------------+------------- + regress_testsub | regress_subscription_user | f | {testpub} | f | parallel | d | f | any | t | f | f | | f | 0 | f | t | off | dbname=regress_doesnotexist | -1 | 0/00000000 | +(1 row) + +-- ok +ALTER SUBSCRIPTION regress_testsub SET (messages = false); +\dRs+ + List of subscriptions + Name | Owner | Enabled | Publication | Binary | Streaming | Two-phase commit | Disable on error | Origin | Password required | Run as owner? | Failover | Server | Retain dead tuples | Max retention duration | Retention active | Messages | Synchronous commit | Conninfo | Receiver timeout | Skip LSN | Description +-----------------+---------------------------+---------+-------------+--------+-----------+------------------+------------------+--------+-------------------+---------------+----------+--------+--------------------+------------------------+------------------+----------+--------------------+-----------------------------+------------------+------------+------------- + regress_testsub | regress_subscription_user | f | {testpub} | f | parallel | d | f | any | t | f | f | | f | 0 | f | f | off | dbname=regress_doesnotexist | -1 | 0/00000000 | (1 row) +-- cleanup ALTER SUBSCRIPTION regress_testsub SET (slot_name = NONE); DROP SUBSCRIPTION regress_testsub; -- let's do some tests with pg_create_subscription rather than superuser diff --git a/src/test/regress/sql/subscription.sql b/src/test/regress/sql/subscription.sql index 7718c742974..4fecd0f87db 100644 --- a/src/test/regress/sql/subscription.sql +++ b/src/test/regress/sql/subscription.sql @@ -441,6 +441,23 @@ ALTER SUBSCRIPTION regress_testsub SET (max_retention_duration = 0); ALTER SUBSCRIPTION regress_testsub SET (slot_name = NONE); DROP SUBSCRIPTION regress_testsub; +-- fail - messages must be boolean +CREATE SUBSCRIPTION regress_testsub CONNECTION 'dbname=regress_doesnotexist' PUBLICATION testpub WITH (connect = false, messages = foo); + +-- ok +CREATE SUBSCRIPTION regress_testsub CONNECTION 'dbname=regress_doesnotexist' PUBLICATION testpub WITH (connect = false, messages = true); + +\dRs+ + +-- ok +ALTER SUBSCRIPTION regress_testsub SET (messages = false); + +\dRs+ + +-- cleanup +ALTER SUBSCRIPTION regress_testsub SET (slot_name = NONE); +DROP SUBSCRIPTION regress_testsub; + -- let's do some tests with pg_create_subscription rather than superuser SET SESSION AUTHORIZATION regress_subscription_user3; diff --git a/src/tools/pgindent/typedefs.list b/src/tools/pgindent/typedefs.list index 85d989f395d..6b452f8c267 100644 --- a/src/tools/pgindent/typedefs.list +++ b/src/tools/pgindent/typedefs.list @@ -1707,6 +1707,8 @@ LogicalRepBeginData LogicalRepCommitData LogicalRepCommitPreparedTxnData LogicalRepCtxStruct +LogicalRepMessageData +LogicalRepMessageHandle_hook_type LogicalRepMsgType LogicalRepPartMapEntry LogicalRepPreparedTxnData -- 2.55.0
^ permalink raw reply [nested|flat] 44+ messages in thread

* Re: Add a hook for handling logical decoding messages on subscribers.
@ 2026-08-06 17:41  Bharath Rupireddy <bharath.rupireddyforpostgres@gmail.com>
  parent: Masahiko Sawada <sawada.mshk@gmail.com>
  0 siblings, 1 reply; 44+ messages in thread

From: Bharath Rupireddy @ 2026-08-06 17:41 UTC (permalink / raw)
  To: Masahiko Sawada <sawada.mshk@gmail.com>; +Cc: Chao Li <li.evan.chao@gmail.com>; Fujii Masao <masao.fujii@gmail.com>; Amit Kapila <amit.kapila16@gmail.com>; pgsql-hackers

Hi,

On Thu, Aug 6, 2026 at 9:59 AM Masahiko Sawada <sawada.mshk@gmail.com> wrote:
>
> > 2/
> > + /*
> > + * The message doesn't belong to any remote transaction, so there is
> > + * no remote commit LSN nor timestamp to record. Clear the state left
> > + * over by the previously applied transaction so that this commit
> > + * doesn't inherit it.
> > + */
> > + replorigin_xact_clear(false);
> >
> > Why is this a problem if we let the non-transactional message inherit it?
>
> IIUC non-transactional messages would have the same commit timestamp
> as the previously applied transaction, which is wrong to me.

Having replorigin_xact_clear there looks fine to me. The next
transaction commit would anyway set the origin LSN and timestamp.

> > Wrapping the hook with begin and end replication step is nice. This
> > lets the hook see the correct command ID, snapshot, and memory
> > context. There are callers that do the begin first and read message
> > next (insert), but it seems okay this way because read message doesn't
> > do any catalog or table accesses, so it should be fine.
> >
>
> begin_replication_step() switches the memory context to
> ApplyMessageContext. Given logicalrep_read_message() palloc's for
> messages, it should be called after begin_replication_step(). Fixed it.

Right. I verified other places and wherever the read does a palloc, it
is wrapped within begin and end replication step.

> I've attached the updated patch.

Thanks. The v4 patch looks good to me. pgindent and tests are happy. I
have no further comments. I marked the CF entry RfC
(https://commitfest.postgresql.org/patch/7092/). FWIW, the CF bot
complains with "needs rebase":
https://cfbot.cputube.org/patch_7092.log.

--
Bharath Rupireddy
Amazon Web Services: https://aws.amazon.com






^ permalink  raw  reply  [nested|flat] 44+ messages in thread

* Re: Add a hook for handling logical decoding messages on subscribers.
@ 2026-09-18 00:42  Masahiko Sawada <sawada.mshk@gmail.com>
  parent: Bharath Rupireddy <bharath.rupireddyforpostgres@gmail.com>
  0 siblings, 2 replies; 44+ messages in thread

From: Masahiko Sawada @ 2026-09-18 00:42 UTC (permalink / raw)
  To: Bharath Rupireddy <bharath.rupireddyforpostgres@gmail.com>; +Cc: Chao Li <li.evan.chao@gmail.com>; Fujii Masao <masao.fujii@gmail.com>; Amit Kapila <amit.kapila16@gmail.com>; pgsql-hackers

On Thu, Aug 6, 2026 at 10:42 AM Bharath Rupireddy
<bharath.rupireddyforpostgres@gmail.com> wrote:
>
> Hi,
>
> On Thu, Aug 6, 2026 at 9:59 AM Masahiko Sawada <sawada.mshk@gmail.com> wrote:
> >
> > > 2/
> > > + /*
> > > + * The message doesn't belong to any remote transaction, so there is
> > > + * no remote commit LSN nor timestamp to record. Clear the state left
> > > + * over by the previously applied transaction so that this commit
> > > + * doesn't inherit it.
> > > + */
> > > + replorigin_xact_clear(false);
> > >
> > > Why is this a problem if we let the non-transactional message inherit it?
> >
> > IIUC non-transactional messages would have the same commit timestamp
> > as the previously applied transaction, which is wrong to me.
>
> Having replorigin_xact_clear there looks fine to me. The next
> transaction commit would anyway set the origin LSN and timestamp.
>
> > > Wrapping the hook with begin and end replication step is nice. This
> > > lets the hook see the correct command ID, snapshot, and memory
> > > context. There are callers that do the begin first and read message
> > > next (insert), but it seems okay this way because read message doesn't
> > > do any catalog or table accesses, so it should be fine.
> > >
> >
> > begin_replication_step() switches the memory context to
> > ApplyMessageContext. Given logicalrep_read_message() palloc's for
> > messages, it should be called after begin_replication_step(). Fixed it.
>
> Right. I verified other places and wherever the read does a palloc, it
> is wrapped within begin and end replication step.
>
> > I've attached the updated patch.
>
> Thanks. The v4 patch looks good to me. pgindent and tests are happy. I
> have no further comments. I marked the CF entry RfC
> (https://commitfest.postgresql.org/patch/7092/). FWIW, the CF bot
> complains with "needs rebase":
> https://cfbot.cputube.org/patch_7092.log.

Thanks for reviewing the patch!

While I also think the patch is in good shape, I'd like to raise a
security risk this feature might introduce, particularly around
pg_logical_emit_message():

EXECUTE on that function is granted to PUBLIC, so any role that can
connect to the publisher database can emit a logical decoding message.
With this patch the apply worker hands the message to a handler that
runs with the privileges of the subscription owner, and unlike
insert/update/delete there is no table owner to switch to. One use
case I have in mind is DDL replication. If an extension implements it
on top of this hook and the subscription is owned by a superuser, any
role on the publisher can choose what the handler is given and have it
executed with superuser privileges on the subscriber. So extensions
should carefully consider this case. I think the same applies to other
extensions that might use this feature. The real problem is that there
is no reliable way for the subscriber to tell whether a message came
from the source it expects.

A practical solution is to revoke EXECUTE on pg_logical_emit_message()
from PUBLIC on the publisher and grant it to a role created for that
purpose. I think that covers most cases I did consider having the
server record the emitting role in the message so that the subscriber
could check it, but I'm not sure this feature alone justifies it. So
my current thought is to document these risks and add nothing special
for these cases.

I've added the documentation changes and rebased the patch. Any ideas
and feedback is very welcome.

Regards,

--
Masahiko Sawada
Amazon Web Services: https://aws.amazon.com

Attachments:

v5-0001-Add-a-hook-for-handling-logical-messages-on-subsc.patch (text/x-patch, 114.7K) download From 283569252e34b0da2db397532b671e9a11258a1d Mon Sep 17 00:00:00 2001 From: Masahiko Sawada <sawada.mshk@gmail.com> Date: Mon, 8 Jun 2026 11:25:47 -0700 Subject: [PATCH v5] Add a hook for handling logical messages on subscribers. Previously, pgoutput could emit logical decoding messages, those written with pg_logical_emit_message(), but the built-in subscriber never requested them from the publisher, and apply_handle_message() was only a placeholder that discarded the MESSAGE message type. Acting on such messages therefore meant writing a separate logical decoding client, and giving up the apply worker's streaming, conflict handling and progress tracking. Logical messages are a general-purpose side channel carried in band with the data, useful for replicating deparsed DDL, for carrying change-data-capture context that the row data does not have, and for cache invalidation delivered in commit order relative to the rows it refers to. This commit adds a new "messages" subscription option, which asks the publisher for the pgoutput "messages" option, and dispatches each received message to a new LogicalRepMessageHandle_hook. Behavior is unchanged when the option is off or when no handler is installed. A transactional message is applied as a step of the remote transaction it belongs to, so the handler's work commits atomically with that transaction, and transaction streaming and ALTER SUBSCRIPTION ... SKIP apply to it as they do to any other change. A non-transactional message belongs to no remote transaction and is committed on its own. A message may still reach the handler more than once, for example after an apply worker restart, so handlers must be idempotent. Note also that messages are not scoped by publication: every message emitted in the publisher database is sent, subject only to origin filtering, and pg_logical_emit_message() requires no special privileges. A handler must therefore treat the payload as untrusted input, and as an arbitrary string of bytes rather than as text in the subscriber's encoding. The publisher must be running PostgreSQL 14 or later, where pgoutput gained the "messages" option; against an older publisher the option is accepted but no message is ever sent. Bump catalog version. Reviewed-by: Bharath Rupireddy <bharath.rupireddyforpostgres@gmail.com> Reviewed-by: Fujii Masao <masao.fujii@gmail.com> Reviewed-by: Chao Li <li.evan.chao@gmail.com> Discussion: https://postgr.es/m/CAD21AoCTNGiddikkUcDKj5QLnsg-51bpr-o6L-GTHWZL4ZFYtQ@mail.gmail.com --- doc/src/sgml/catalogs.sgml | 11 + doc/src/sgml/func/func-admin.sgml | 7 +- doc/src/sgml/logical-replication.sgml | 27 +++ doc/src/sgml/ref/alter_subscription.sgml | 5 +- doc/src/sgml/ref/create_subscription.sgml | 43 ++++ src/backend/catalog/pg_subscription.c | 1 + src/backend/catalog/system_views.sql | 5 +- src/backend/commands/subscriptioncmds.c | 25 ++- .../libpqwalreceiver/libpqwalreceiver.c | 4 + src/backend/replication/logical/proto.c | 37 ++++ src/backend/replication/logical/worker.c | 87 +++++++- src/bin/pg_dump/pg_dump.c | 16 +- src/bin/pg_dump/pg_dump.h | 1 + src/bin/pg_dump/t/002_pg_dump.pl | 4 +- src/bin/psql/describe.c | 7 +- src/include/catalog/pg_subscription.h | 6 + src/include/replication/logicalproto.h | 13 ++ src/include/replication/logicalworker.h | 42 ++++ src/include/replication/walreceiver.h | 1 + src/test/modules/Makefile | 1 + src/test/modules/meson.build | 1 + .../modules/test_logicalmsg_hooks/.gitignore | 4 + .../modules/test_logicalmsg_hooks/Makefile | 20 ++ .../modules/test_logicalmsg_hooks/meson.build | 28 +++ .../test_logicalmsg_hooks/t/001_basic.pl | 207 ++++++++++++++++++ .../test_logicalmsg_hooks.c | 140 ++++++++++++ src/test/regress/expected/subscription.out | 202 +++++++++-------- src/test/regress/sql/subscription.sql | 17 ++ src/tools/pgindent/typedefs.list | 2 + 29 files changed, 857 insertions(+), 107 deletions(-) create mode 100644 src/test/modules/test_logicalmsg_hooks/.gitignore create mode 100644 src/test/modules/test_logicalmsg_hooks/Makefile create mode 100644 src/test/modules/test_logicalmsg_hooks/meson.build create mode 100644 src/test/modules/test_logicalmsg_hooks/t/001_basic.pl create mode 100644 src/test/modules/test_logicalmsg_hooks/test_logicalmsg_hooks.c diff --git a/doc/src/sgml/catalogs.sgml b/doc/src/sgml/catalogs.sgml index 78cce7e370d..6180b60eeb7 100644 --- a/doc/src/sgml/catalogs.sgml +++ b/doc/src/sgml/catalogs.sgml @@ -8231,6 +8231,17 @@ SCRAM-SHA-256$<replaceable>&lt;iteration count&gt;</replaceable>:<replaceable>&l </para></entry> </row> + <row> + <entry role="catalog_table_entry"><para role="column_definition"> + <structfield>submessages</structfield> <type>bool</type> + </para> + <para> + If true, the subscription requests the publisher to send logical + decoding messages, which are passed to a handler registered by an + extension on the subscriber. + </para></entry> + </row> + <row> <entry role="catalog_table_entry"><para role="column_definition"> <structfield>subconninfo</structfield> <type>text</type> diff --git a/doc/src/sgml/func/func-admin.sgml b/doc/src/sgml/func/func-admin.sgml index 64b0e7bb972..d64a8cca536 100644 --- a/doc/src/sgml/func/func-admin.sgml +++ b/doc/src/sgml/func/func-admin.sgml @@ -1479,7 +1479,12 @@ postgres=# SELECT '0/0'::pg_lsn + pd.segment_number * ps.setting::int + :offset <literal>false</literal>) controls if the message is immediately flushed to WAL or not. <parameter>flush</parameter> has no effect with <parameter>transactional</parameter>, as the message's WAL - record is flushed along with its transaction. + record is flushed along with its transaction. In addition to logical + decoding clients, messages emitted by this function are delivered to + subscribers whose subscriptions have + <link linkend="sql-createsubscription-params-with-messages"><literal>messages</literal></link> + enabled, where they are passed to a handler registered by an extension + on the subscriber. </para></entry> </row> diff --git a/doc/src/sgml/logical-replication.sgml b/doc/src/sgml/logical-replication.sgml index 4701a3d9d18..6783ae03e38 100644 --- a/doc/src/sgml/logical-replication.sgml +++ b/doc/src/sgml/logical-replication.sgml @@ -2661,6 +2661,33 @@ CONTEXT: processing remote data for replication origin "pg_16395" during "INSER not need privileges to <literal>SET ROLE</literal> to the sequence owner. </para> + <para> + Logical decoding messages are an exception to the user switching described + above. If the subscription has <link linkend="sql-createsubscription-params-with-messages"><literal>messages</literal></link> + enabled, the handler registered for them runs with the privileges of the + subscription owner. Messages are not restricted to the tables or schemas + of the subscribed publications either, and by default any role may call + <link linkend="pg-logical-emit-message"><function>pg_logcal_emit_message()</function></link>. + Any role that can connect to the publisher database can therefore choose + what the handler is given, and a handler that acts on what it receives does + so with the subscription owner's privileges. Restrict who can emit messages + on the publisher. Revoke the default privilege and grant it to a role created + for this purpose: +<programlisting> +REVOKE EXECUTE ON FUNCTION pg_logical_emit_message(boolean, text, text, boolean) FROM PUBLIC; +REVOKE EXECUTE ON FUNCTION pg_logical_emit_message(boolean, text, bytea, boolean) FROM PUBLIC; +GRANT EXECUTE ON FUNCTION pg_logical_emit_message(boolean, text, text, boolean) TO message_emitter; +GRANT EXECUTE ON FUNCTION pg_logical_emit_message(boolean, text, bytea, boolean) TO message_emitter; +</programlisting> + Function privileges are per database, so this has to be done in each database + that such a subscription replicates from. No role that is not trusted to drive + the handler should be a member of the new role, since a role that can + <literal>SET ROLE</literal> to it can emit whatever it likes. Where a message + has to be emitted on behalf of an ordinary user, for example from an event + trigger that captures that user's DDL, use a <literal>SECURITY DEFINER</literal> + function owned by that role rather than granting the privilege more widely. + </para> + <para> On the publisher, privileges are only checked once at the start of a replication connection and are not re-checked as each change record is read. diff --git a/doc/src/sgml/ref/alter_subscription.sgml b/doc/src/sgml/ref/alter_subscription.sgml index 545264e8a0a..72b1b0fcd83 100644 --- a/doc/src/sgml/ref/alter_subscription.sgml +++ b/doc/src/sgml/ref/alter_subscription.sgml @@ -302,8 +302,9 @@ ALTER SUBSCRIPTION <replaceable class="parameter">name</replaceable> RENAME TO < <link linkend="sql-createsubscription-params-with-two-phase"><literal>two_phase</literal></link>, <link linkend="sql-createsubscription-params-with-retain-dead-tuples"><literal>retain_dead_tuples</literal></link>, <link linkend="sql-createsubscription-params-with-max-retention-duration"><literal>max_retention_duration</literal></link>, - <link linkend="sql-createsubscription-params-with-wal-receiver-timeout"><literal>wal_receiver_timeout</literal></link>, and - <link linkend="sql-createsubscription-params-with-conflict-log-destination"><literal>conflict_log_destination</literal></link>. + <link linkend="sql-createsubscription-params-with-wal-receiver-timeout"><literal>wal_receiver_timeout</literal></link>, + <link linkend="sql-createsubscription-params-with-conflict-log-destination"><literal>conflict_log_destination</literal></link>, and + <link linkend="sql-createsubscription-params-with-messages"><literal>messages</literal></link>. Only a superuser can set <literal>password_required = false</literal>. </para> diff --git a/doc/src/sgml/ref/create_subscription.sgml b/doc/src/sgml/ref/create_subscription.sgml index 6248c228a64..1c38daaf434 100644 --- a/doc/src/sgml/ref/create_subscription.sgml +++ b/doc/src/sgml/ref/create_subscription.sgml @@ -478,6 +478,49 @@ CREATE SUBSCRIPTION <replaceable class="parameter">subscription_name</replaceabl </listitem> </varlistentry> + <varlistentry id="sql-createsubscription-params-with-messages"> + <term><literal>messages</literal> (<type>boolean</type>)</term> + <listitem> + <para> + Specifies whether the subscription requests the publisher to send + logical decoding messages, that is, those emitted with + <link linkend="pg-logical-emit-message"><function>pg_logical_emit_message</function></link>. + The default is <literal>false</literal>. + </para> + <para> + Received messages are only acted upon if an extension installed on + the subscriber has registered a handler for them; otherwise they are + received and discarded. A message may be passed to the handler more + than once, for example after an apply worker restart, so a handler + must be prepared to process the same message repeatedly. + </para> + <para> + A transactional message is applied as part of the remote transaction + that emitted it, and is therefore delivered in commit order with + respect to other transactions, not in the order in which it was written. + Note that + <link linkend="sql-createsubscription-params-with-streaming"><literal>streaming</literal></link> + and <link linkend="sql-altersubscription-params-skip"><literal>SKIP</literal></link> + apply to transactional messages just as they do to other changes. A + non-transactional message is delivered as soon as it is decoded, even + if the transaction that emitted it later aborted. + </para> + <para> + Messages are not restricted to the tables or schemas of the + subscribed publications: every message emitted in the publisher + database is sent, subject only to + <link linkend="sql-createsubscription-params-with-origin"><literal>origin</literal></link> + filtering. Message contents are supplied by whoever calls + <function>pg_logical_emit_message</function> on the publisher, which + by default any role may do, so a handler should treat them as + untrusted input. A message is an arbitrary string of bytes, and is + transmitted without character set conversion; it may therefore + contain embedded nulls, and need not be valid text in the encoding + of either database. + </para> + </listitem> + </varlistentry> + <varlistentry id="sql-createsubscription-params-with-origin"> <term><literal>origin</literal> (<type>string</type>)</term> <listitem> diff --git a/src/backend/catalog/pg_subscription.c b/src/backend/catalog/pg_subscription.c index 60c15b22194..138cdecd436 100644 --- a/src/backend/catalog/pg_subscription.c +++ b/src/backend/catalog/pg_subscription.c @@ -133,6 +133,7 @@ GetSubscription(Oid subid, bool missing_ok) sub->maxretention = subform->submaxretention; sub->retentionactive = subform->subretentionactive; sub->conflictlogrelid = subform->subconflictlogrelid; + sub->messages = subform->submessages; /* Get slotname */ datum = SysCacheGetAttr(SUBSCRIPTIONOID, diff --git a/src/backend/catalog/system_views.sql b/src/backend/catalog/system_views.sql index ad340887f54..728eacae112 100644 --- a/src/backend/catalog/system_views.sql +++ b/src/backend/catalog/system_views.sql @@ -1539,8 +1539,9 @@ GRANT SELECT (oid, subdbid, subskiplsn, subname, subowner, subenabled, subbinary, substream, subtwophasestate, subdisableonerr, subpasswordrequired, subrunasowner, subfailover, subretaindeadtuples, submaxretention, subretentionactive, - subserver, subconflictlogrelid, subconflictlogdest, subslotname, - subsynccommit, subwalrcvtimeout, subpublications, suborigin) + subserver, subconflictlogrelid, subconflictlogdest, submessages, + subslotname, subsynccommit, subwalrcvtimeout, subpublications, + suborigin) ON pg_subscription TO public; CREATE VIEW pg_stat_subscription_stats AS diff --git a/src/backend/commands/subscriptioncmds.c b/src/backend/commands/subscriptioncmds.c index 22a61dca65d..fb5b3b71831 100644 --- a/src/backend/commands/subscriptioncmds.c +++ b/src/backend/commands/subscriptioncmds.c @@ -84,6 +84,7 @@ #define SUBOPT_LSN 0x00020000 #define SUBOPT_ORIGIN 0x00040000 #define SUBOPT_CONFLICT_LOG_DEST 0x00080000 +#define SUBOPT_MESSAGES 0x00100000 /* check if the 'val' has 'bits' set */ #define IsSet(val, bits) (((val) & (bits)) == (bits)) @@ -110,6 +111,7 @@ typedef struct SubOpts bool runasowner; bool failover; bool retaindeadtuples; + bool messages; int32 maxretention; char *origin; ConflictLogDest conflictlogdest; @@ -208,6 +210,8 @@ parse_subscription_options(ParseState *pstate, List *stmt_options, opts->origin = pstrdup(LOGICALREP_ORIGIN_ANY); if (IsSet(supported_opts, SUBOPT_CONFLICT_LOG_DEST)) opts->conflictlogdest = CONFLICT_LOG_DEST_LOG; + if (IsSet(supported_opts, SUBOPT_MESSAGES)) + opts->messages = false; /* Parse options */ foreach(lc, stmt_options) @@ -460,6 +464,15 @@ parse_subscription_options(ParseState *pstate, List *stmt_options, opts->conflictlogdest = GetConflictLogDest(val); opts->specified_opts |= SUBOPT_CONFLICT_LOG_DEST; } + else if (IsSet(supported_opts, SUBOPT_MESSAGES) && + strcmp(defel->defname, "messages") == 0) + { + if (IsSet(opts->specified_opts, SUBOPT_MESSAGES)) + errorConflictingDefElem(defel, pstate); + + opts->specified_opts |= SUBOPT_MESSAGES; + opts->messages = defGetBoolean(defel); + } else ereport(ERROR, (errcode(ERRCODE_SYNTAX_ERROR), @@ -700,7 +713,7 @@ CreateSubscription(ParseState *pstate, CreateSubscriptionStmt *stmt, SUBOPT_RETAIN_DEAD_TUPLES | SUBOPT_MAX_RETENTION_DURATION | SUBOPT_WAL_RECEIVER_TIMEOUT | SUBOPT_ORIGIN | - SUBOPT_CONFLICT_LOG_DEST); + SUBOPT_CONFLICT_LOG_DEST | SUBOPT_MESSAGES); parse_subscription_options(pstate, stmt->options, supported_opts, &opts); /* @@ -872,6 +885,7 @@ CreateSubscription(ParseState *pstate, CreateSubscriptionStmt *stmt, values[Anum_pg_subscription_subretentionactive - 1] = BoolGetDatum(opts.retaindeadtuples); values[Anum_pg_subscription_subserver - 1] = ObjectIdGetDatum(serverid); + values[Anum_pg_subscription_submessages - 1] = BoolGetDatum(opts.messages); if (stmt->conninfo) { Assert(stmt->conninfo == conninfo && !OidIsValid(serverid)); @@ -1708,7 +1722,8 @@ AlterSubscription(ParseState *pstate, AlterSubscriptionStmt *stmt, SUBOPT_MAX_RETENTION_DURATION | SUBOPT_WAL_RECEIVER_TIMEOUT | SUBOPT_ORIGIN | - SUBOPT_CONFLICT_LOG_DEST); + SUBOPT_CONFLICT_LOG_DEST | + SUBOPT_MESSAGES); break; case ALTER_SUBSCRIPTION_ENABLED: @@ -2115,6 +2130,12 @@ AlterSubscription(ParseState *pstate, AlterSubscriptionStmt *stmt, } } + if (IsSet(opts.specified_opts, SUBOPT_MESSAGES)) + { + values[Anum_pg_subscription_submessages - 1] = BoolGetDatum(opts.messages); + replaces[Anum_pg_subscription_submessages - 1] = true; + } + update_tuple = true; break; } diff --git a/src/backend/replication/libpqwalreceiver/libpqwalreceiver.c b/src/backend/replication/libpqwalreceiver/libpqwalreceiver.c index 029990d9fce..e345b80690f 100644 --- a/src/backend/replication/libpqwalreceiver/libpqwalreceiver.c +++ b/src/backend/replication/libpqwalreceiver/libpqwalreceiver.c @@ -631,6 +631,10 @@ libpqrcv_startstreaming(WalReceiverConn *conn, appendQuotedLiteral(&cmd, options->proto.logical.origin); } + if (options->proto.logical.messages && + PQserverVersion(conn->streamConn) >= 140000) + appendStringInfo(&cmd, ", messages 'on'"); + pubnames = options->proto.logical.publication_names; pubnames_str = stringlist_to_identifierstr(pubnames); appendStringInfoString(&cmd, ", publication_names "); diff --git a/src/backend/replication/logical/proto.c b/src/backend/replication/logical/proto.c index 86ad97cd937..6461361d68d 100644 --- a/src/backend/replication/logical/proto.c +++ b/src/backend/replication/logical/proto.c @@ -660,6 +660,43 @@ logicalrep_write_message(StringInfo out, TransactionId xid, XLogRecPtr lsn, pq_sendbytes(out, message, sz); } +/* + * Read MESSAGE from stream. + */ +void +logicalrep_read_message(StringInfo in, LogicalRepMessageData *msg_data) +{ + Size len; + char *msg; + uint8 flags; + + /* read and decode flags */ + flags = pq_getmsgint(in, 1); + msg_data->transactional = (flags & MESSAGE_TRANSACTIONAL) != 0; + + msg_data->lsn = pq_getmsgint64(in); + msg_data->prefix = pstrdup(pq_getmsgstring(in)); + + /* read message length */ + len = pq_getmsgint(in, 4); + msg_data->message_size = len; + + /* and data */ + msg = palloc(len + 1); + pq_copymsgbytes(in, msg, len); + + /* + * The payload is an arbitrary string of bytes, so message_size, not + * strlen(), is what gives its length. NULL-terminate it anyway, as + * logicalrep_read_tuple() does for column values: that way it can be + * handed to code expecting a C string without reading past the end, which + * is convenient for logging and debugging. Such code will of course stop + * at the first embedded null, if any. + */ + msg[len] = '\0'; + msg_data->message = msg; +} + /* * Write relation description to the output stream. */ diff --git a/src/backend/replication/logical/worker.c b/src/backend/replication/logical/worker.c index 7781bb1c168..43a069c6ca0 100644 --- a/src/backend/replication/logical/worker.c +++ b/src/backend/replication/logical/worker.c @@ -560,6 +560,9 @@ typedef struct ApplySubXactData static ApplySubXactData subxact_data = {0, 0, InvalidTransactionId, NULL}; +/* Hook for plugins to get control in handling logical decoding messages */ +LogicalRepMessageHandle_hook_type LogicalRepMessageHandle_hook = NULL; + static inline void subxact_filename(char *path, Oid subid, TransactionId xid); static inline void changes_filename(char *path, Oid subid, TransactionId xid); @@ -1710,6 +1713,71 @@ apply_handle_origin(StringInfo s) errmsg_internal("ORIGIN message sent out of order"))); } +/* + * Handle MESSAGE message. + * + * Invoke a hook function if set. + */ +static void +apply_handle_message(StringInfo s) +{ + LogicalRepMessageData msg; + + /* Tablesync worker should never receive MESSAGE */ + if (am_tablesync_worker()) + ereport(ERROR, + errcode(ERRCODE_PROTOCOL_VIOLATION), + errmsg_internal("tablesync worker received a MESSAGE message")); + + if (!LogicalRepMessageHandle_hook) + return; + + if (is_skipping_changes() || + handle_streamed_transaction(LOGICAL_REP_MSG_MESSAGE, s)) + return; + + begin_replication_step(); + + logicalrep_read_message(s, &msg); + + (*LogicalRepMessageHandle_hook) (&msg); + + end_replication_step(); + + /* + * A transactional message is applied as a step of the remote transaction + * that emitted it, and is committed together with it when applying the + * commit message. A non-transactional message belongs to no remote + * transaction, so commit it here. + */ + if (!msg.transactional) + { + Assert(!in_remote_transaction); + Assert(!in_streamed_transaction); + + /* + * The message doesn't belong to any remote transaction, so there is + * no remote commit LSN nor timestamp to record. Clear the state left + * over by the previously applied transaction so that this commit + * doesn't inherit it. + */ + replorigin_xact_clear(false); + + CommitTransactionCommand(); + + pgstat_report_stat(false); + + /* + * Report the flush position only after the local commit is durable. + * Otherwise send_feedback() would report everything received so far + * as flushed, the publisher would advance confirmed_flush_lsn past + * this message, and a crash before the commit reaches disk would lose + * the handler's work without the message ever being sent again. + */ + store_flush_position(msg.lsn, XactLastCommitEnd); + } +} + /* * Initialize fileset (if not already done). * @@ -3872,12 +3940,7 @@ apply_dispatch(StringInfo s) break; case LOGICAL_REP_MSG_MESSAGE: - - /* - * Logical replication does not use generic logical messages yet. - * Although, it could be used by other applications that use this - * output plugin. - */ + apply_handle_message(s); break; case LOGICAL_REP_MSG_STREAM_START: @@ -5163,7 +5226,8 @@ maybe_reread_subscription(void) newsub->passwordrequired != MySubscription->passwordrequired || strcmp(newsub->origin, MySubscription->origin) != 0 || newsub->owner != MySubscription->owner || - !equal(newsub->publications, MySubscription->publications)) + !equal(newsub->publications, MySubscription->publications) || + newsub->messages != MySubscription->messages) { if (am_parallel_apply_worker()) ereport(LOG, @@ -5654,6 +5718,15 @@ set_stream_options(WalRcvStreamOptions *options, options->proto.logical.twophase = false; options->proto.logical.origin = pstrdup(MySubscription->origin); + + /* + * Logical messages are relation-agnostic, so they don't map cleanly onto + * the tablesync worker of a particular relation, and there would be no + * well-defined ordering between a message and the initial copy. Leave + * them to the (parallel) apply workers. + */ + options->proto.logical.messages = (MySubscription->messages && + !am_tablesync_worker()); } /* diff --git a/src/bin/pg_dump/pg_dump.c b/src/bin/pg_dump/pg_dump.c index 388c3b9c346..f3615d66ec2 100644 --- a/src/bin/pg_dump/pg_dump.c +++ b/src/bin/pg_dump/pg_dump.c @@ -5136,6 +5136,7 @@ getSubscriptions(Archive *fout) int i_subfailover; int i_subretaindeadtuples; int i_submaxretention; + int i_submessages; int i, ntups; @@ -5234,9 +5235,14 @@ getSubscriptions(Archive *fout) " '-1' AS subwalrcvtimeout,\n"); if (fout->remoteVersion >= 190000) - appendPQExpBufferStr(query, " fs.srvname AS subservername\n"); + appendPQExpBufferStr(query, " fs.srvname AS subservername,\n"); else - appendPQExpBufferStr(query, " NULL AS subservername\n"); + appendPQExpBufferStr(query, " NULL AS subservername,\n"); + + if (fout->remoteVersion >= 200000) + appendPQExpBufferStr(query, " s.submessages\n"); + else + appendPQExpBufferStr(query, " false AS submessages\n"); appendPQExpBufferStr(query, "FROM pg_subscription s\n"); @@ -5285,6 +5291,7 @@ getSubscriptions(Archive *fout) i_subpublications = PQfnumber(res, "subpublications"); i_suborigin = PQfnumber(res, "suborigin"); i_suboriginremotelsn = PQfnumber(res, "suboriginremotelsn"); + i_submessages = PQfnumber(res, "submessages"); subinfo = pg_malloc_array(SubscriptionInfo, ntups); @@ -5342,6 +5349,8 @@ getSubscriptions(Archive *fout) else subinfo[i].suboriginremotelsn = pg_strdup(PQgetvalue(res, i, i_suboriginremotelsn)); + subinfo[i].submessages = + (strcmp(PQgetvalue(res, i, i_submessages), "t") == 0); /* Decide whether we want to dump it */ selectDumpableObject(&(subinfo[i].dobj), fout); @@ -5594,6 +5603,9 @@ dumpSubscription(Archive *fout, const SubscriptionInfo *subinfo) if (subinfo->subretaindeadtuples) appendPQExpBufferStr(query, ", retain_dead_tuples = true"); + if (subinfo->submessages) + appendPQExpBufferStr(query, ", messages = true"); + if (subinfo->submaxretention) appendPQExpBuffer(query, ", max_retention_duration = %d", subinfo->submaxretention); diff --git a/src/bin/pg_dump/pg_dump.h b/src/bin/pg_dump/pg_dump.h index 2bbb5d5773b..4f703e40133 100644 --- a/src/bin/pg_dump/pg_dump.h +++ b/src/bin/pg_dump/pg_dump.h @@ -719,6 +719,7 @@ typedef struct _SubscriptionInfo bool subrunasowner; bool subfailover; bool subretaindeadtuples; + bool submessages; int submaxretention; char *subservername; char *subconninfo; diff --git a/src/bin/pg_dump/t/002_pg_dump.pl b/src/bin/pg_dump/t/002_pg_dump.pl index 1299c837063..780f6415141 100644 --- a/src/bin/pg_dump/t/002_pg_dump.pl +++ b/src/bin/pg_dump/t/002_pg_dump.pl @@ -3305,9 +3305,9 @@ my %tests = ( create_order => 50, create_sql => 'CREATE SUBSCRIPTION sub3 CONNECTION \'dbname=doesnotexist\' PUBLICATION pub1 - WITH (connect = false, origin = any, streaming = on);', + WITH (connect = false, origin = any, streaming = on, messages = on);', regexp => qr/^ - \QCREATE SUBSCRIPTION sub3 CONNECTION 'dbname=doesnotexist' PUBLICATION pub1 WITH (connect = false, slot_name = 'sub3', streaming = on);\E + \QCREATE SUBSCRIPTION sub3 CONNECTION 'dbname=doesnotexist' PUBLICATION pub1 WITH (connect = false, slot_name = 'sub3', streaming = on, messages = true);\E /xm, like => { %full_runs, section_post_data => 1, }, unlike => { diff --git a/src/bin/psql/describe.c b/src/bin/psql/describe.c index dc9b3841592..171c5c189e3 100644 --- a/src/bin/psql/describe.c +++ b/src/bin/psql/describe.c @@ -6804,7 +6804,7 @@ describeSubscriptions(const char *pattern, bool verbose) printQueryOpt myopt = pset.popt; static const bool translate_columns[] = {false, false, false, false, false, false, false, false, false, false, false, false, false, false, - false, false, false, false, false, false, false}; + false, false, false, false, false, false, false, false}; initPQExpBuffer(&buf); @@ -6882,6 +6882,11 @@ describeSubscriptions(const char *pattern, bool verbose) gettext_noop("Retention active")); } + if (pset.sversion >= 200000) + appendPQExpBuffer(&buf, + ", submessages AS \"%s\"\n", + gettext_noop("Messages")); + appendPQExpBuffer(&buf, ", subsynccommit AS \"%s\"\n" ", subconninfo AS \"%s\"\n", diff --git a/src/include/catalog/pg_subscription.h b/src/include/catalog/pg_subscription.h index d2781a0b837..6465d79bee5 100644 --- a/src/include/catalog/pg_subscription.h +++ b/src/include/catalog/pg_subscription.h @@ -96,6 +96,9 @@ CATALOG(pg_subscription,6100,SubscriptionRelationId) BKI_SHARED_RELATION BKI_ROW * server */ Oid subconflictlogrelid; /* Relid of the conflict log table. */ + + bool submessages; /* True if the subscription wants to receive + * logical messages. */ #ifdef CATALOG_VARLEN /* variable-length fields start here */ /* @@ -173,6 +176,9 @@ typedef struct Subscription * exceeded max_retention_duration, when * defined */ Oid conflictlogrelid; /* conflict log table Oid */ + bool messages; /* True if the subscription wants to receive + * logical messages */ + char *conninfo; /* Connection string to the publisher */ char *slotname; /* Name of the replication slot */ char *synccommit; /* Synchronous commit setting for worker */ char *walrcvtimeout; /* wal_receiver_timeout setting for worker */ diff --git a/src/include/replication/logicalproto.h b/src/include/replication/logicalproto.h index 058a955e20c..83741a2080d 100644 --- a/src/include/replication/logicalproto.h +++ b/src/include/replication/logicalproto.h @@ -191,6 +191,18 @@ typedef struct LogicalRepStreamAbortData TimestampTz abort_time; } LogicalRepStreamAbortData; +/* + * Logical decoding message information + */ +typedef struct LogicalRepMessageData +{ + XLogRecPtr lsn; + bool transactional; + const char *prefix; + Size message_size; /* length of message in bytes */ + const char *message; /* payload; may contain embedded nulls */ +} LogicalRepMessageData; + extern void logicalrep_write_begin(StringInfo out, ReorderBufferTXN *txn); extern void logicalrep_read_begin(StringInfo in, LogicalRepBeginData *begin_data); @@ -248,6 +260,7 @@ extern List *logicalrep_read_truncate(StringInfo in, bool *cascade, bool *restart_seqs); extern void logicalrep_write_message(StringInfo out, TransactionId xid, XLogRecPtr lsn, bool transactional, const char *prefix, Size sz, const char *message); +extern void logicalrep_read_message(StringInfo in, LogicalRepMessageData *msg_data); extern void logicalrep_write_rel(StringInfo out, TransactionId xid, Relation rel, Bitmapset *columns, PublishGencolsType include_gencols_type); diff --git a/src/include/replication/logicalworker.h b/src/include/replication/logicalworker.h index 7d748a28da8..c1f0c81c0df 100644 --- a/src/include/replication/logicalworker.h +++ b/src/include/replication/logicalworker.h @@ -16,6 +16,48 @@ extern PGDLLIMPORT volatile sig_atomic_t ParallelApplyMessagePending; +/* + * Forward-declared so that this header, which is included by several files + * unrelated to logical replication, does not have to include logicalproto.h. + * A handler that dereferences the struct must include that header itself. + */ +struct LogicalRepMessageData; + +/* + * Hook for extensions to handle logical decoding messages (see + * pg_logical_emit_message()) received from the publisher. Called only when + * the subscription has the "messages" option enabled. + * + * The handler runs in the apply worker, inside a transaction, with an active + * snapshot pushed. For a transactional message the handler's work is part of + * the remote transaction and commits with it; a non-transactional message is + * committed as an independent unit. The same message may be delivered more + * than once, so the handler must be idempotent. + * + * The payload is an arbitrary string of bytes. The WAL record makes no + * distinction between the text and the bytea form of + * pg_logical_emit_message(), and the bytes are transmitted without character + * set conversion, so the payload may contain embedded nulls and bytes that + * are not valid in the subscriber's encoding. Its length is given by + * msg->message_size, not by strlen(); the buffer is null-terminated only for + * convenience, and that null is not part of the payload. + * + * The handler always runs with the privileges of the subscription owner. + * The payload is chosen by whoever called pg_logical_emit_message() on the + * publisher, which by default any role may do, so a handler that acts on the + * payload gives every role on the publisher the subscription owner's + * privileges. A handler that needs less than that can lower them itself + * with SwitchToUntrustedUser(). + * + * Note that the apply worker runs with an empty search_path. An ERROR thrown + * by the handler restarts the apply worker, or disables the subscription if + * disable_on_error is set. Because the message is delivered again after the + * restart, a handler cannot reject a message by throwing an error; that only + * loops. A message the handler does not want must simply be ignored. + */ +typedef void (*LogicalRepMessageHandle_hook_type) (const struct LogicalRepMessageData *msg); +extern PGDLLIMPORT LogicalRepMessageHandle_hook_type LogicalRepMessageHandle_hook; + extern void ApplyWorkerMain(Datum main_arg); extern void ParallelApplyWorkerMain(Datum main_arg); extern void TableSyncWorkerMain(Datum main_arg); diff --git a/src/include/replication/walreceiver.h b/src/include/replication/walreceiver.h index 5cfe90f9989..6831bd1f751 100644 --- a/src/include/replication/walreceiver.h +++ b/src/include/replication/walreceiver.h @@ -188,6 +188,7 @@ typedef struct * prepare time */ char *origin; /* Only publish data originating from the * specified origin */ + bool messages; /* Logical messages */ } logical; } proto; } WalRcvStreamOptions; diff --git a/src/test/modules/Makefile b/src/test/modules/Makefile index 71a2e65ad70..c58d53fdf79 100644 --- a/src/test/modules/Makefile +++ b/src/test/modules/Makefile @@ -37,6 +37,7 @@ SUBDIRS = \ test_integerset \ test_json_parser \ test_lfind \ + test_logicalmsg_hooks \ test_lwlock_tranches \ test_misc \ test_oat_hooks \ diff --git a/src/test/modules/meson.build b/src/test/modules/meson.build index 77e1a2810e5..e7a0d351292 100644 --- a/src/test/modules/meson.build +++ b/src/test/modules/meson.build @@ -38,6 +38,7 @@ subdir('test_int128') subdir('test_integerset') subdir('test_json_parser') subdir('test_lfind') +subdir('test_logicalmsg_hooks') subdir('test_lwlock_tranches') subdir('test_misc') subdir('test_oat_hooks') diff --git a/src/test/modules/test_logicalmsg_hooks/.gitignore b/src/test/modules/test_logicalmsg_hooks/.gitignore new file mode 100644 index 00000000000..5dcb3ff9723 --- /dev/null +++ b/src/test/modules/test_logicalmsg_hooks/.gitignore @@ -0,0 +1,4 @@ +# Generated subdirectories +/log/ +/results/ +/tmp_check/ diff --git a/src/test/modules/test_logicalmsg_hooks/Makefile b/src/test/modules/test_logicalmsg_hooks/Makefile new file mode 100644 index 00000000000..af129009b8a --- /dev/null +++ b/src/test/modules/test_logicalmsg_hooks/Makefile @@ -0,0 +1,20 @@ +# src/test/modules/test_logicalmsg_hooks/Makefile + +MODULE_big = test_logicalmsg_hooks +OBJS = \ + $(WIN32RES) \ + test_logicalmsg_hooks.o +PGFILEDESC = "test_logicalmsg_hooks - test logical replication message hooks" + +TAP_TESTS = 1 + +ifdef USE_PGXS +PG_CONFIG = pg_config +PGXS := $(shell $(PG_CONFIG) --pgxs) +include $(PGXS) +else +subdir = src/test/modules/test_logicalmsg_hooks +top_builddir = ../../../.. +include $(top_builddir)/src/Makefile.global +include $(top_srcdir)/contrib/contrib-global.mk +endif diff --git a/src/test/modules/test_logicalmsg_hooks/meson.build b/src/test/modules/test_logicalmsg_hooks/meson.build new file mode 100644 index 00000000000..8084d82e47b --- /dev/null +++ b/src/test/modules/test_logicalmsg_hooks/meson.build @@ -0,0 +1,28 @@ +# Copyright (c) 2022-2026, PostgreSQL Global Development Group + +test_logicalmsg_hooks_sources = files( + 'test_logicalmsg_hooks.c', +) + +if host_system == 'windows' + test_logicalmsg_hooks_sources += rc_lib_gen.process(win32ver_rc, extra_args: [ + '--NAME', 'test_logicalmsg_hooks', + '--FILEDESC', 'test_logicalmsg_hooks - test logical message hooks',]) +endif + +test_logicalmsg_hooks = shared_module('test_logicalmsg_hooks', + test_logicalmsg_hooks_sources, + kwargs: pg_test_mod_args, +) +test_install_libs += test_logicalmsg_hooks + +tests += { + 'name': 'test_logicalmsg_hooks', + 'sd': meson.current_source_dir(), + 'bd': meson.current_build_dir(), + 'tap': { + 'tests': [ + 't/001_basic.pl', + ], + }, +} diff --git a/src/test/modules/test_logicalmsg_hooks/t/001_basic.pl b/src/test/modules/test_logicalmsg_hooks/t/001_basic.pl new file mode 100644 index 00000000000..a9ff6171ad6 --- /dev/null +++ b/src/test/modules/test_logicalmsg_hooks/t/001_basic.pl @@ -0,0 +1,207 @@ +# Copyright (c) 2026, PostgreSQL Global Development Group + +# Test LogicalRepMessageHandle_hook. The test module records every message it +# receives into public.test_logicalmsg_log on the subscriber, so most checks +# here are plain queries against that table. + +use strict; +use warnings FATAL => 'all'; + +use PostgreSQL::Test::Cluster; +use PostgreSQL::Test::Utils; +use Test::More; + +my $pub = PostgreSQL::Test::Cluster->new('publisher'); +$pub->init(allows_streaming => 'logical'); + +# Keep the memory limit low so that the transactions below are streamed. +$pub->append_conf('postgresql.conf', "logical_decoding_work_mem = 64kB\n"); +$pub->start; + +my $sub = PostgreSQL::Test::Cluster->new('subscriber'); +$sub->init(allows_streaming => 'logical'); +$sub->append_conf( + 'postgresql.conf', q{ +shared_preload_libraries = 'test_logicalmsg_hooks' +max_parallel_apply_workers_per_subscription = 2 +}); +$sub->start; + +$pub->safe_psql( + 'postgres', qq{ +CREATE TABLE test (a int); +INSERT INTO test VALUES (1); +CREATE PUBLICATION pub FOR ALL TABLES; +}); + +my $pub_connstr = $pub->connstr . ' dbname=postgres'; + +$sub->safe_psql( + 'postgres', qq{ +CREATE TABLE test (a int primary key); +CREATE SUBSCRIPTION sub CONNECTION '$pub_connstr' PUBLICATION pub WITH (messages = true, disable_on_error = true) +}); + +$sub->wait_for_subscription_sync($pub, 'sub'); + +# Number of messages recorded under the given prefix. +sub logged_count +{ + my ($prefix) = @_; + + return $sub->safe_psql('postgres', + "SELECT count(*) FROM test_logicalmsg_log WHERE prefix = '$prefix'"); +} + +# Verify that a transactional message is passed to the hook, and that the +# message data given to the hook is correct. +$pub->safe_psql('postgres', + q{SELECT pg_logical_emit_message(true, 'basic', 'transactional message')} +); +$pub->wait_for_catchup('sub'); + +is( $sub->safe_psql( + 'postgres', + q{SELECT transactional, message_size, message FROM test_logicalmsg_log + WHERE prefix = 'basic'}), + 't|21|transactional message', + "transactional message is passed to the hook"); + +# Verify that a non-transactional message is applied even if the transaction +# that emitted it rolls back. +$pub->safe_psql( + 'postgres', q{ +BEGIN; +SELECT pg_logical_emit_message(false, 'nontransactional', 'emitted then rolled back'); +ROLLBACK; +}); + +# Commit another message to flush WAL, as ROLLBACK doesn't flush. +$pub->safe_psql('postgres', + q{SELECT pg_logical_emit_message(true, 'flush', 'flush')}); +$pub->wait_for_catchup('sub'); + +is( $sub->safe_psql( + 'postgres', + q{SELECT transactional, message FROM test_logicalmsg_log + WHERE prefix = 'nontransactional'}), + 'f|emitted then rolled back', + "non-transactional message of an aborted transaction is applied"); + +# Verify that a transactional message emitted in a subtransaction that is +# rolled back is not passed to the hook. +$pub->safe_psql( + 'postgres', q{ +BEGIN; +SAVEPOINT sp; +SELECT pg_logical_emit_message(true, 'subxact_aborted', 'not delivered'); +ROLLBACK TO SAVEPOINT sp; +SELECT pg_logical_emit_message(true, 'subxact_committed', 'delivered'); +COMMIT; +}); +$pub->wait_for_catchup('sub'); + +is(logged_count('subxact_aborted'), + 0, "message of an aborted subtransaction is not applied"); +is(logged_count('subxact_committed'), + 1, "message of a committed subtransaction is applied"); + +# Verify that transactional messages are applied in streaming transactions, +# both when the leader apply worker serializes the transaction to a file and +# when it hands the transaction to a parallel apply worker. +foreach my $mode ('on', 'parallel') +{ + $sub->safe_psql('postgres', + "ALTER SUBSCRIPTION sub SET (streaming = $mode)"); + + # Reconnect so that the new streaming mode takes effect immediately. + $sub->safe_psql('postgres', 'ALTER SUBSCRIPTION sub DISABLE'); + $sub->safe_psql('postgres', 'ALTER SUBSCRIPTION sub ENABLE'); + + $pub->safe_psql('postgres', + q{SELECT pg_stat_reset_replication_slot('sub')}); + + $pub->safe_psql( + 'postgres', qq{ +BEGIN; +INSERT INTO test SELECT generate_series(1000, 6000); +SELECT pg_logical_emit_message(true, 'streaming_$mode', 'streamed message'); +COMMIT; +}); + $pub->wait_for_catchup('sub'); + + # Make sure the transaction really was streamed, so that this test cannot + # silently degrade into a non-streaming one. + $pub->poll_query_until('postgres', + q{SELECT stream_txns > 0 FROM pg_stat_replication_slots WHERE slot_name = 'sub'} + ) or die "transaction was not streamed"; + + is(logged_count("streaming_$mode"), + 1, + "message in a streamed transaction is applied (streaming = $mode)"); + + $pub->safe_psql('postgres', 'TRUNCATE test'); + $pub->wait_for_catchup('sub'); +} + +$sub->safe_psql('postgres', 'ALTER SUBSCRIPTION sub SET (streaming = off)'); + +# Verify that ALTER SUBSCRIPTION ... SKIP skips a transactional message along +# with the rest of its transaction. The message is emitted before the +# conflicting change, so the hook runs and records the message on the first +# attempt; that work must be rolled back with the failed transaction, and must +# not be redone once the transaction is skipped. +$pub->safe_psql('postgres', 'INSERT INTO test VALUES (1)'); +$pub->wait_for_catchup('sub'); + +my $log_location = -s $sub->logfile; + +$pub->safe_psql( + 'postgres', q{ +BEGIN; +SELECT pg_logical_emit_message(true, 'skipped', 'rolled back, then skipped'); +INSERT INTO test VALUES (1); +COMMIT; +}); + +# Wait until the conflict disables the subscription. +$sub->poll_query_until('postgres', + q{SELECT subenabled = FALSE FROM pg_subscription WHERE subname = 'sub'}); + +# The hook did run before the transaction failed. +ok( $sub->log_contains( + qr/LOG[^\n]+received message: [^\n]+prefix: skipped/, + $log_location), + "hook is called for a message in a transaction that later fails"); + +# ... but its work was rolled back with the transaction. +is(logged_count('skipped'), 0, + "work done by the hook is rolled back with the remote transaction"); + +# Get the finish LSN of the failed transaction. +my $contents = slurp_file($sub->logfile, $log_location); +$contents =~ + qr/conflict detected on relation "public.test".*\n.*DETAIL:.*Could not apply remote change.*\n.*Key already exists in unique index "test_pkey", modified in transaction \d+: key .*, local row .*\n.*CONTEXT:.* for replication target relation "public.test" in transaction \d+, finished at ([[:xdigit:]]+\/[[:xdigit:]]+)/m + or die "could not get error-LSN"; +my $lsn = $1; + +$log_location = -s $sub->logfile; + +# Set skip LSN and re-enable the subscription. +$sub->safe_psql('postgres', qq{ALTER SUBSCRIPTION sub SKIP (lsn = '$lsn')}); +$sub->safe_psql('postgres', 'ALTER SUBSCRIPTION sub ENABLE'); + +# Wait for the failed transaction to be skipped. +$sub->poll_query_until('postgres', + q{SELECT subskiplsn = '0/0' FROM pg_subscription WHERE subname = 'sub'}); + +ok( !$sub->log_contains( + qr/LOG[^\n]+received message: [^\n]+prefix: skipped/, + $log_location), + "hook is not called for a message in a skipped transaction"); +is(logged_count('skipped'), 0, + "message in a skipped transaction is not applied"); + +$sub->stop; +$pub->stop; +done_testing(); diff --git a/src/test/modules/test_logicalmsg_hooks/test_logicalmsg_hooks.c b/src/test/modules/test_logicalmsg_hooks/test_logicalmsg_hooks.c new file mode 100644 index 00000000000..d8d6fc7c619 --- /dev/null +++ b/src/test/modules/test_logicalmsg_hooks/test_logicalmsg_hooks.c @@ -0,0 +1,140 @@ +/*-------------------------------------------------------------------------- + * + * test_logicalmsg_hooks.c + * Code for testing LogicalRepMessageHandle_hook + * + * The handler records every logical decoding message it receives into the + * table public.test_logicalmsg_log, so that tests can assert on the contents + * of a table rather than on the server log. Recording the messages this way + * also exercises the transactional behavior of the hook: a transactional + * message is recorded as part of the remote transaction that emitted it, and + * therefore disappears together with it if that transaction is rolled back or + * skipped, while a non-transactional message is recorded independently. + * + * The handler creates that table itself on the first message, so a test does + * not have to set it up. For a transactional message the creation is part of + * the remote transaction too, and is rolled back with it, so the table does + * not exist until some message has been applied successfully; a test must not + * query it before then. + * + * The messages are also logged, which lets a test tell "the handler ran but + * its work was rolled back" apart from "the handler was never called". + * + * Portions Copyright (c) 1996-2026, PostgreSQL Global Development Group + * Portions Copyright (c) 1994, Regents of the University of California + * + * IDENTIFICATION + * src/test/modules/test_logicalmsg_hooks/test_logicalmsg_hooks.c + * + * ------------------------------------------------------------------------- + */ + +#include "postgres.h" + +#include "catalog/namespace.h" +#include "catalog/pg_type.h" +#include "executor/spi.h" +#include "replication/logicalproto.h" +#include "replication/logicalworker.h" +#include "utils/builtins.h" +#include "utils/lsyscache.h" +#include "utils/pg_lsn.h" + +PG_MODULE_MAGIC; + +#define LOG_SCHEMA "public" +#define LOG_TABLE "test_logicalmsg_log" + +static LogicalRepMessageHandle_hook_type prev_logical_message_handler = NULL; +static void test_logical_message_handler(const LogicalRepMessageData *msg); + +/* + * Module load callback + */ +void +_PG_init(void) +{ + prev_logical_message_handler = LogicalRepMessageHandle_hook; + LogicalRepMessageHandle_hook = &test_logical_message_handler; +} + +/* + * Create the log table if not exists. + */ +static void +ensure_log_table_exists(void) +{ + Oid relid; + int ret; + + relid = get_relname_relid(LOG_TABLE, get_namespace_oid(LOG_SCHEMA, true)); + if (OidIsValid(relid)) + return; + + SPI_connect(); + + ret = SPI_execute("CREATE TABLE " LOG_SCHEMA "." LOG_TABLE + "(lsn pg_lsn, transactional boolean, prefix text, message_size int, message text)", + false, 0); + if (ret != SPI_OK_UTILITY) + elog(ERROR, "could not create " LOG_SCHEMA "." LOG_TABLE); + + SPI_finish(); +} + +static void +test_logical_message_handler(const LogicalRepMessageData *msg) +{ + Oid argtypes[5] = {LSNOID, BOOLOID, TEXTOID, INT4OID, TEXTOID}; + Datum values[5]; + int ret; + + /* + * Give a handler installed before ours its turn, so that several + * extensions can act on the same message. + */ + if (prev_logical_message_handler) + (*prev_logical_message_handler) (msg); + + ereport(LOG, + (errmsg("received message: LSN %X/%08X, prefix: %s, size: %zu, transactional: %d", + LSN_FORMAT_ARGS(msg->lsn), msg->prefix, + msg->message_size, msg->transactional))); + + ensure_log_table_exists(); + + /* + * Pass the message as query parameters rather than interpolating it into + * the query text. Message contents come from the publisher and are not to + * be trusted, and the payload may contain characters that would need + * quoting. Note also that message_size, not strlen(), is authoritative + * for the payload length. + * + * The payload is recorded as text because the accompanying test only ever + * emits text messages. A handler must not assume that in general: a + * payload is an arbitrary string of bytes, which may contain embedded + * nulls and bytes that are not valid in the subscriber's encoding. See + * the comments on LogicalRepMessageHandle_hook_type. + */ + values[0] = LSNGetDatum(msg->lsn); + values[1] = BoolGetDatum(msg->transactional); + values[2] = CStringGetTextDatum(msg->prefix); + values[3] = Int32GetDatum((int32) msg->message_size); + values[4] = PointerGetDatum(cstring_to_text_with_len(msg->message, + msg->message_size)); + + SPI_connect(); + + /* + * The apply worker runs with an empty search_path, so the table name must + * be schema-qualified. + */ + ret = SPI_execute_with_args("INSERT INTO " LOG_SCHEMA "." LOG_TABLE + " (lsn, transactional, prefix, message_size, message)" + " VALUES ($1, $2, $3, $4, $5)", + 5, argtypes, values, NULL, false, 0); + if (ret != SPI_OK_INSERT) + elog(ERROR, "could not record logical message: SPI returned %d", ret); + + SPI_finish(); +} diff --git a/src/test/regress/expected/subscription.out b/src/test/regress/expected/subscription.out index 681dc66dca6..52487149a81 100644 --- a/src/test/regress/expected/subscription.out +++ b/src/test/regress/expected/subscription.out @@ -139,18 +139,18 @@ CREATE SUBSCRIPTION regress_testsub4 CONNECTION 'dbname=regress_doesnotexist' PU WARNING: subscription was created, but is not connected HINT: To initiate replication, you must manually create the replication slot, enable the subscription, and alter the subscription to refresh publications. \dRs+ regress_testsub4 - List of subscriptions - Name | Owner | Enabled | Publication | Binary | Streaming | Two-phase commit | Disable on error | Origin | Password required | Run as owner? | Failover | Server | Retain dead tuples | Max retention duration | Retention active | Synchronous commit | Conninfo | Receiver timeout | Skip LSN | Description -------------------+---------------------------+---------+-------------+--------+-----------+------------------+------------------+--------+-------------------+---------------+----------+--------+--------------------+------------------------+------------------+--------------------+-----------------------------+------------------+------------+------------- - regress_testsub4 | regress_subscription_user | f | {testpub} | f | parallel | d | f | none | t | f | f | | f | 0 | f | off | dbname=regress_doesnotexist | -1 | 0/00000000 | + List of subscriptions + Name | Owner | Enabled | Publication | Binary | Streaming | Two-phase commit | Disable on error | Origin | Password required | Run as owner? | Failover | Server | Retain dead tuples | Max retention duration | Retention active | Messages | Synchronous commit | Conninfo | Receiver timeout | Skip LSN | Description +------------------+---------------------------+---------+-------------+--------+-----------+------------------+------------------+--------+-------------------+---------------+----------+--------+--------------------+------------------------+------------------+----------+--------------------+-----------------------------+------------------+------------+------------- + regress_testsub4 | regress_subscription_user | f | {testpub} | f | parallel | d | f | none | t | f | f | | f | 0 | f | f | off | dbname=regress_doesnotexist | -1 | 0/00000000 | (1 row) ALTER SUBSCRIPTION regress_testsub4 SET (origin = any); \dRs+ regress_testsub4 - List of subscriptions - Name | Owner | Enabled | Publication | Binary | Streaming | Two-phase commit | Disable on error | Origin | Password required | Run as owner? | Failover | Server | Retain dead tuples | Max retention duration | Retention active | Synchronous commit | Conninfo | Receiver timeout | Skip LSN | Description -------------------+---------------------------+---------+-------------+--------+-----------+------------------+------------------+--------+-------------------+---------------+----------+--------+--------------------+------------------------+------------------+--------------------+-----------------------------+------------------+------------+------------- - regress_testsub4 | regress_subscription_user | f | {testpub} | f | parallel | d | f | any | t | f | f | | f | 0 | f | off | dbname=regress_doesnotexist | -1 | 0/00000000 | + List of subscriptions + Name | Owner | Enabled | Publication | Binary | Streaming | Two-phase commit | Disable on error | Origin | Password required | Run as owner? | Failover | Server | Retain dead tuples | Max retention duration | Retention active | Messages | Synchronous commit | Conninfo | Receiver timeout | Skip LSN | Description +------------------+---------------------------+---------+-------------+--------+-----------+------------------+------------------+--------+-------------------+---------------+----------+--------+--------------------+------------------------+------------------+----------+--------------------+-----------------------------+------------------+------------+------------- + regress_testsub4 | regress_subscription_user | f | {testpub} | f | parallel | d | f | any | t | f | f | | f | 0 | f | f | off | dbname=regress_doesnotexist | -1 | 0/00000000 | (1 row) DROP SUBSCRIPTION regress_testsub3; @@ -255,10 +255,10 @@ ALTER SUBSCRIPTION regress_testsub CONNECTION 'foobar'; ERROR: invalid connection string syntax: missing "=" after "foobar" in connection info string \dRs+ - List of subscriptions - Name | Owner | Enabled | Publication | Binary | Streaming | Two-phase commit | Disable on error | Origin | Password required | Run as owner? | Failover | Server | Retain dead tuples | Max retention duration | Retention active | Synchronous commit | Conninfo | Receiver timeout | Skip LSN | Description ------------------+---------------------------+---------+-------------+--------+-----------+------------------+------------------+--------+-------------------+---------------+----------+--------+--------------------+------------------------+------------------+--------------------+-----------------------------+------------------+------------+------------------- - regress_testsub | regress_subscription_user | f | {testpub} | f | parallel | d | f | any | t | f | f | | f | 0 | f | off | dbname=regress_doesnotexist | -1 | 0/00000000 | test subscription + List of subscriptions + Name | Owner | Enabled | Publication | Binary | Streaming | Two-phase commit | Disable on error | Origin | Password required | Run as owner? | Failover | Server | Retain dead tuples | Max retention duration | Retention active | Messages | Synchronous commit | Conninfo | Receiver timeout | Skip LSN | Description +-----------------+---------------------------+---------+-------------+--------+-----------+------------------+------------------+--------+-------------------+---------------+----------+--------+--------------------+------------------------+------------------+----------+--------------------+-----------------------------+------------------+------------+------------------- + regress_testsub | regress_subscription_user | f | {testpub} | f | parallel | d | f | any | t | f | f | | f | 0 | f | f | off | dbname=regress_doesnotexist | -1 | 0/00000000 | test subscription (1 row) ALTER SUBSCRIPTION regress_testsub SET PUBLICATION testpub2, testpub3 WITH (refresh = false); @@ -267,10 +267,10 @@ ALTER SUBSCRIPTION regress_testsub SET (slot_name = 'newname'); ALTER SUBSCRIPTION regress_testsub SET (password_required = false); ALTER SUBSCRIPTION regress_testsub SET (run_as_owner = true); \dRs+ - List of subscriptions - Name | Owner | Enabled | Publication | Binary | Streaming | Two-phase commit | Disable on error | Origin | Password required | Run as owner? | Failover | Server | Retain dead tuples | Max retention duration | Retention active | Synchronous commit | Conninfo | Receiver timeout | Skip LSN | Description ------------------+---------------------------+---------+---------------------+--------+-----------+------------------+------------------+--------+-------------------+---------------+----------+--------+--------------------+------------------------+------------------+--------------------+------------------------------+------------------+------------+------------------- - regress_testsub | regress_subscription_user | f | {testpub2,testpub3} | f | parallel | d | f | any | f | t | f | | f | 0 | f | off | dbname=regress_doesnotexist2 | -1 | 0/00000000 | test subscription + List of subscriptions + Name | Owner | Enabled | Publication | Binary | Streaming | Two-phase commit | Disable on error | Origin | Password required | Run as owner? | Failover | Server | Retain dead tuples | Max retention duration | Retention active | Messages | Synchronous commit | Conninfo | Receiver timeout | Skip LSN | Description +-----------------+---------------------------+---------+---------------------+--------+-----------+------------------+------------------+--------+-------------------+---------------+----------+--------+--------------------+------------------------+------------------+----------+--------------------+------------------------------+------------------+------------+------------------- + regress_testsub | regress_subscription_user | f | {testpub2,testpub3} | f | parallel | d | f | any | f | t | f | | f | 0 | f | f | off | dbname=regress_doesnotexist2 | -1 | 0/00000000 | test subscription (1 row) ALTER SUBSCRIPTION regress_testsub SET (password_required = true); @@ -286,10 +286,10 @@ ERROR: unrecognized subscription parameter: "create_slot" -- ok ALTER SUBSCRIPTION regress_testsub SKIP (lsn = '0/12345'); \dRs+ - List of subscriptions - Name | Owner | Enabled | Publication | Binary | Streaming | Two-phase commit | Disable on error | Origin | Password required | Run as owner? | Failover | Server | Retain dead tuples | Max retention duration | Retention active | Synchronous commit | Conninfo | Receiver timeout | Skip LSN | Description ------------------+---------------------------+---------+---------------------+--------+-----------+------------------+------------------+--------+-------------------+---------------+----------+--------+--------------------+------------------------+------------------+--------------------+------------------------------+------------------+------------+------------------- - regress_testsub | regress_subscription_user | f | {testpub2,testpub3} | f | parallel | d | f | any | t | f | f | | f | 0 | f | off | dbname=regress_doesnotexist2 | -1 | 0/00012345 | test subscription + List of subscriptions + Name | Owner | Enabled | Publication | Binary | Streaming | Two-phase commit | Disable on error | Origin | Password required | Run as owner? | Failover | Server | Retain dead tuples | Max retention duration | Retention active | Messages | Synchronous commit | Conninfo | Receiver timeout | Skip LSN | Description +-----------------+---------------------------+---------+---------------------+--------+-----------+------------------+------------------+--------+-------------------+---------------+----------+--------+--------------------+------------------------+------------------+----------+--------------------+------------------------------+------------------+------------+------------------- + regress_testsub | regress_subscription_user | f | {testpub2,testpub3} | f | parallel | d | f | any | t | f | f | | f | 0 | f | f | off | dbname=regress_doesnotexist2 | -1 | 0/00012345 | test subscription (1 row) -- ok - with lsn = NONE @@ -298,10 +298,10 @@ ALTER SUBSCRIPTION regress_testsub SKIP (lsn = NONE); ALTER SUBSCRIPTION regress_testsub SKIP (lsn = '0/0'); ERROR: invalid WAL location (LSN): 0/0 \dRs+ - List of subscriptions - Name | Owner | Enabled | Publication | Binary | Streaming | Two-phase commit | Disable on error | Origin | Password required | Run as owner? | Failover | Server | Retain dead tuples | Max retention duration | Retention active | Synchronous commit | Conninfo | Receiver timeout | Skip LSN | Description ------------------+---------------------------+---------+---------------------+--------+-----------+------------------+------------------+--------+-------------------+---------------+----------+--------+--------------------+------------------------+------------------+--------------------+------------------------------+------------------+------------+------------------- - regress_testsub | regress_subscription_user | f | {testpub2,testpub3} | f | parallel | d | f | any | t | f | f | | f | 0 | f | off | dbname=regress_doesnotexist2 | -1 | 0/00000000 | test subscription + List of subscriptions + Name | Owner | Enabled | Publication | Binary | Streaming | Two-phase commit | Disable on error | Origin | Password required | Run as owner? | Failover | Server | Retain dead tuples | Max retention duration | Retention active | Messages | Synchronous commit | Conninfo | Receiver timeout | Skip LSN | Description +-----------------+---------------------------+---------+---------------------+--------+-----------+------------------+------------------+--------+-------------------+---------------+----------+--------+--------------------+------------------------+------------------+----------+--------------------+------------------------------+------------------+------------+------------------- + regress_testsub | regress_subscription_user | f | {testpub2,testpub3} | f | parallel | d | f | any | t | f | f | | f | 0 | f | f | off | dbname=regress_doesnotexist2 | -1 | 0/00000000 | test subscription (1 row) BEGIN; @@ -337,10 +337,10 @@ ALTER SUBSCRIPTION regress_testsub_foo SET (wal_receiver_timeout = '80s'); ALTER SUBSCRIPTION regress_testsub_foo SET (wal_receiver_timeout = 'foobar'); ERROR: invalid value for parameter "wal_receiver_timeout": "foobar" \dRs+ - List of subscriptions - Name | Owner | Enabled | Publication | Binary | Streaming | Two-phase commit | Disable on error | Origin | Password required | Run as owner? | Failover | Server | Retain dead tuples | Max retention duration | Retention active | Synchronous commit | Conninfo | Receiver timeout | Skip LSN | Description ----------------------+---------------------------+---------+---------------------+--------+-----------+------------------+------------------+--------+-------------------+---------------+----------+--------+--------------------+------------------------+------------------+--------------------+------------------------------+------------------+------------+------------------- - regress_testsub_foo | regress_subscription_user | f | {testpub2,testpub3} | f | parallel | d | f | any | t | f | f | | f | 0 | f | local | dbname=regress_doesnotexist2 | 80s | 0/00000000 | test subscription + List of subscriptions + Name | Owner | Enabled | Publication | Binary | Streaming | Two-phase commit | Disable on error | Origin | Password required | Run as owner? | Failover | Server | Retain dead tuples | Max retention duration | Retention active | Messages | Synchronous commit | Conninfo | Receiver timeout | Skip LSN | Description +---------------------+---------------------------+---------+---------------------+--------+-----------+------------------+------------------+--------+-------------------+---------------+----------+--------+--------------------+------------------------+------------------+----------+--------------------+------------------------------+------------------+------------+------------------- + regress_testsub_foo | regress_subscription_user | f | {testpub2,testpub3} | f | parallel | d | f | any | t | f | f | | f | 0 | f | f | local | dbname=regress_doesnotexist2 | 80s | 0/00000000 | test subscription (1 row) -- rename back to keep the rest simple @@ -369,19 +369,19 @@ CREATE SUBSCRIPTION regress_testsub CONNECTION 'dbname=regress_doesnotexist' PUB WARNING: subscription was created, but is not connected HINT: To initiate replication, you must manually create the replication slot, enable the subscription, and alter the subscription to refresh publications. \dRs+ - List of subscriptions - Name | Owner | Enabled | Publication | Binary | Streaming | Two-phase commit | Disable on error | Origin | Password required | Run as owner? | Failover | Server | Retain dead tuples | Max retention duration | Retention active | Synchronous commit | Conninfo | Receiver timeout | Skip LSN | Description ------------------+---------------------------+---------+-------------+--------+-----------+------------------+------------------+--------+-------------------+---------------+----------+--------+--------------------+------------------------+------------------+--------------------+-----------------------------+------------------+------------+------------- - regress_testsub | regress_subscription_user | f | {testpub} | t | parallel | d | f | any | t | f | f | | f | 0 | f | off | dbname=regress_doesnotexist | -1 | 0/00000000 | + List of subscriptions + Name | Owner | Enabled | Publication | Binary | Streaming | Two-phase commit | Disable on error | Origin | Password required | Run as owner? | Failover | Server | Retain dead tuples | Max retention duration | Retention active | Messages | Synchronous commit | Conninfo | Receiver timeout | Skip LSN | Description +-----------------+---------------------------+---------+-------------+--------+-----------+------------------+------------------+--------+-------------------+---------------+----------+--------+--------------------+------------------------+------------------+----------+--------------------+-----------------------------+------------------+------------+------------- + regress_testsub | regress_subscription_user | f | {testpub} | t | parallel | d | f | any | t | f | f | | f | 0 | f | f | off | dbname=regress_doesnotexist | -1 | 0/00000000 | (1 row) ALTER SUBSCRIPTION regress_testsub SET (binary = false); ALTER SUBSCRIPTION regress_testsub SET (slot_name = NONE); \dRs+ - List of subscriptions - Name | Owner | Enabled | Publication | Binary | Streaming | Two-phase commit | Disable on error | Origin | Password required | Run as owner? | Failover | Server | Retain dead tuples | Max retention duration | Retention active | Synchronous commit | Conninfo | Receiver timeout | Skip LSN | Description ------------------+---------------------------+---------+-------------+--------+-----------+------------------+------------------+--------+-------------------+---------------+----------+--------+--------------------+------------------------+------------------+--------------------+-----------------------------+------------------+------------+------------- - regress_testsub | regress_subscription_user | f | {testpub} | f | parallel | d | f | any | t | f | f | | f | 0 | f | off | dbname=regress_doesnotexist | -1 | 0/00000000 | + List of subscriptions + Name | Owner | Enabled | Publication | Binary | Streaming | Two-phase commit | Disable on error | Origin | Password required | Run as owner? | Failover | Server | Retain dead tuples | Max retention duration | Retention active | Messages | Synchronous commit | Conninfo | Receiver timeout | Skip LSN | Description +-----------------+---------------------------+---------+-------------+--------+-----------+------------------+------------------+--------+-------------------+---------------+----------+--------+--------------------+------------------------+------------------+----------+--------------------+-----------------------------+------------------+------------+------------- + regress_testsub | regress_subscription_user | f | {testpub} | f | parallel | d | f | any | t | f | f | | f | 0 | f | f | off | dbname=regress_doesnotexist | -1 | 0/00000000 | (1 row) DROP SUBSCRIPTION regress_testsub; @@ -393,27 +393,27 @@ CREATE SUBSCRIPTION regress_testsub CONNECTION 'dbname=regress_doesnotexist' PUB WARNING: subscription was created, but is not connected HINT: To initiate replication, you must manually create the replication slot, enable the subscription, and alter the subscription to refresh publications. \dRs+ - List of subscriptions - Name | Owner | Enabled | Publication | Binary | Streaming | Two-phase commit | Disable on error | Origin | Password required | Run as owner? | Failover | Server | Retain dead tuples | Max retention duration | Retention active | Synchronous commit | Conninfo | Receiver timeout | Skip LSN | Description ------------------+---------------------------+---------+-------------+--------+-----------+------------------+------------------+--------+-------------------+---------------+----------+--------+--------------------+------------------------+------------------+--------------------+-----------------------------+------------------+------------+------------- - regress_testsub | regress_subscription_user | f | {testpub} | f | on | d | f | any | t | f | f | | f | 0 | f | off | dbname=regress_doesnotexist | -1 | 0/00000000 | + List of subscriptions + Name | Owner | Enabled | Publication | Binary | Streaming | Two-phase commit | Disable on error | Origin | Password required | Run as owner? | Failover | Server | Retain dead tuples | Max retention duration | Retention active | Messages | Synchronous commit | Conninfo | Receiver timeout | Skip LSN | Description +-----------------+---------------------------+---------+-------------+--------+-----------+------------------+------------------+--------+-------------------+---------------+----------+--------+--------------------+------------------------+------------------+----------+--------------------+-----------------------------+------------------+------------+------------- + regress_testsub | regress_subscription_user | f | {testpub} | f | on | d | f | any | t | f | f | | f | 0 | f | f | off | dbname=regress_doesnotexist | -1 | 0/00000000 | (1 row) ALTER SUBSCRIPTION regress_testsub SET (streaming = parallel); \dRs+ - List of subscriptions - Name | Owner | Enabled | Publication | Binary | Streaming | Two-phase commit | Disable on error | Origin | Password required | Run as owner? | Failover | Server | Retain dead tuples | Max retention duration | Retention active | Synchronous commit | Conninfo | Receiver timeout | Skip LSN | Description ------------------+---------------------------+---------+-------------+--------+-----------+------------------+------------------+--------+-------------------+---------------+----------+--------+--------------------+------------------------+------------------+--------------------+-----------------------------+------------------+------------+------------- - regress_testsub | regress_subscription_user | f | {testpub} | f | parallel | d | f | any | t | f | f | | f | 0 | f | off | dbname=regress_doesnotexist | -1 | 0/00000000 | + List of subscriptions + Name | Owner | Enabled | Publication | Binary | Streaming | Two-phase commit | Disable on error | Origin | Password required | Run as owner? | Failover | Server | Retain dead tuples | Max retention duration | Retention active | Messages | Synchronous commit | Conninfo | Receiver timeout | Skip LSN | Description +-----------------+---------------------------+---------+-------------+--------+-----------+------------------+------------------+--------+-------------------+---------------+----------+--------+--------------------+------------------------+------------------+----------+--------------------+-----------------------------+------------------+------------+------------- + regress_testsub | regress_subscription_user | f | {testpub} | f | parallel | d | f | any | t | f | f | | f | 0 | f | f | off | dbname=regress_doesnotexist | -1 | 0/00000000 | (1 row) ALTER SUBSCRIPTION regress_testsub SET (streaming = false); ALTER SUBSCRIPTION regress_testsub SET (slot_name = NONE); \dRs+ - List of subscriptions - Name | Owner | Enabled | Publication | Binary | Streaming | Two-phase commit | Disable on error | Origin | Password required | Run as owner? | Failover | Server | Retain dead tuples | Max retention duration | Retention active | Synchronous commit | Conninfo | Receiver timeout | Skip LSN | Description ------------------+---------------------------+---------+-------------+--------+-----------+------------------+------------------+--------+-------------------+---------------+----------+--------+--------------------+------------------------+------------------+--------------------+-----------------------------+------------------+------------+------------- - regress_testsub | regress_subscription_user | f | {testpub} | f | off | d | f | any | t | f | f | | f | 0 | f | off | dbname=regress_doesnotexist | -1 | 0/00000000 | + List of subscriptions + Name | Owner | Enabled | Publication | Binary | Streaming | Two-phase commit | Disable on error | Origin | Password required | Run as owner? | Failover | Server | Retain dead tuples | Max retention duration | Retention active | Messages | Synchronous commit | Conninfo | Receiver timeout | Skip LSN | Description +-----------------+---------------------------+---------+-------------+--------+-----------+------------------+------------------+--------+-------------------+---------------+----------+--------+--------------------+------------------------+------------------+----------+--------------------+-----------------------------+------------------+------------+------------- + regress_testsub | regress_subscription_user | f | {testpub} | f | off | d | f | any | t | f | f | | f | 0 | f | f | off | dbname=regress_doesnotexist | -1 | 0/00000000 | (1 row) -- fail - publication already exists @@ -428,10 +428,10 @@ ALTER SUBSCRIPTION regress_testsub ADD PUBLICATION testpub1, testpub2 WITH (refr ALTER SUBSCRIPTION regress_testsub ADD PUBLICATION testpub1, testpub2 WITH (refresh = false); ERROR: publication "testpub1" is already in subscription "regress_testsub" \dRs+ - List of subscriptions - Name | Owner | Enabled | Publication | Binary | Streaming | Two-phase commit | Disable on error | Origin | Password required | Run as owner? | Failover | Server | Retain dead tuples | Max retention duration | Retention active | Synchronous commit | Conninfo | Receiver timeout | Skip LSN | Description ------------------+---------------------------+---------+-----------------------------+--------+-----------+------------------+------------------+--------+-------------------+---------------+----------+--------+--------------------+------------------------+------------------+--------------------+-----------------------------+------------------+------------+------------- - regress_testsub | regress_subscription_user | f | {testpub,testpub1,testpub2} | f | off | d | f | any | t | f | f | | f | 0 | f | off | dbname=regress_doesnotexist | -1 | 0/00000000 | + List of subscriptions + Name | Owner | Enabled | Publication | Binary | Streaming | Two-phase commit | Disable on error | Origin | Password required | Run as owner? | Failover | Server | Retain dead tuples | Max retention duration | Retention active | Messages | Synchronous commit | Conninfo | Receiver timeout | Skip LSN | Description +-----------------+---------------------------+---------+-----------------------------+--------+-----------+------------------+------------------+--------+-------------------+---------------+----------+--------+--------------------+------------------------+------------------+----------+--------------------+-----------------------------+------------------+------------+------------- + regress_testsub | regress_subscription_user | f | {testpub,testpub1,testpub2} | f | off | d | f | any | t | f | f | | f | 0 | f | f | off | dbname=regress_doesnotexist | -1 | 0/00000000 | (1 row) -- fail - publication used more than once @@ -446,10 +446,10 @@ ERROR: publication "testpub3" is not in subscription "regress_testsub" -- ok - delete publications ALTER SUBSCRIPTION regress_testsub DROP PUBLICATION testpub1, testpub2 WITH (refresh = false); \dRs+ - List of subscriptions - Name | Owner | Enabled | Publication | Binary | Streaming | Two-phase commit | Disable on error | Origin | Password required | Run as owner? | Failover | Server | Retain dead tuples | Max retention duration | Retention active | Synchronous commit | Conninfo | Receiver timeout | Skip LSN | Description ------------------+---------------------------+---------+-------------+--------+-----------+------------------+------------------+--------+-------------------+---------------+----------+--------+--------------------+------------------------+------------------+--------------------+-----------------------------+------------------+------------+------------- - regress_testsub | regress_subscription_user | f | {testpub} | f | off | d | f | any | t | f | f | | f | 0 | f | off | dbname=regress_doesnotexist | -1 | 0/00000000 | + List of subscriptions + Name | Owner | Enabled | Publication | Binary | Streaming | Two-phase commit | Disable on error | Origin | Password required | Run as owner? | Failover | Server | Retain dead tuples | Max retention duration | Retention active | Messages | Synchronous commit | Conninfo | Receiver timeout | Skip LSN | Description +-----------------+---------------------------+---------+-------------+--------+-----------+------------------+------------------+--------+-------------------+---------------+----------+--------+--------------------+------------------------+------------------+----------+--------------------+-----------------------------+------------------+------------+------------- + regress_testsub | regress_subscription_user | f | {testpub} | f | off | d | f | any | t | f | f | | f | 0 | f | f | off | dbname=regress_doesnotexist | -1 | 0/00000000 | (1 row) DROP SUBSCRIPTION regress_testsub; @@ -485,19 +485,19 @@ CREATE SUBSCRIPTION regress_testsub CONNECTION 'dbname=regress_doesnotexist' PUB WARNING: subscription was created, but is not connected HINT: To initiate replication, you must manually create the replication slot, enable the subscription, and alter the subscription to refresh publications. \dRs+ - List of subscriptions - Name | Owner | Enabled | Publication | Binary | Streaming | Two-phase commit | Disable on error | Origin | Password required | Run as owner? | Failover | Server | Retain dead tuples | Max retention duration | Retention active | Synchronous commit | Conninfo | Receiver timeout | Skip LSN | Description ------------------+---------------------------+---------+-------------+--------+-----------+------------------+------------------+--------+-------------------+---------------+----------+--------+--------------------+------------------------+------------------+--------------------+-----------------------------+------------------+------------+------------- - regress_testsub | regress_subscription_user | f | {testpub} | f | parallel | p | f | any | t | f | f | | f | 0 | f | off | dbname=regress_doesnotexist | -1 | 0/00000000 | + List of subscriptions + Name | Owner | Enabled | Publication | Binary | Streaming | Two-phase commit | Disable on error | Origin | Password required | Run as owner? | Failover | Server | Retain dead tuples | Max retention duration | Retention active | Messages | Synchronous commit | Conninfo | Receiver timeout | Skip LSN | Description +-----------------+---------------------------+---------+-------------+--------+-----------+------------------+------------------+--------+-------------------+---------------+----------+--------+--------------------+------------------------+------------------+----------+--------------------+-----------------------------+------------------+------------+------------- + regress_testsub | regress_subscription_user | f | {testpub} | f | parallel | p | f | any | t | f | f | | f | 0 | f | f | off | dbname=regress_doesnotexist | -1 | 0/00000000 | (1 row) -- we can alter streaming when two_phase enabled ALTER SUBSCRIPTION regress_testsub SET (streaming = true); \dRs+ - List of subscriptions - Name | Owner | Enabled | Publication | Binary | Streaming | Two-phase commit | Disable on error | Origin | Password required | Run as owner? | Failover | Server | Retain dead tuples | Max retention duration | Retention active | Synchronous commit | Conninfo | Receiver timeout | Skip LSN | Description ------------------+---------------------------+---------+-------------+--------+-----------+------------------+------------------+--------+-------------------+---------------+----------+--------+--------------------+------------------------+------------------+--------------------+-----------------------------+------------------+------------+------------- - regress_testsub | regress_subscription_user | f | {testpub} | f | on | p | f | any | t | f | f | | f | 0 | f | off | dbname=regress_doesnotexist | -1 | 0/00000000 | + List of subscriptions + Name | Owner | Enabled | Publication | Binary | Streaming | Two-phase commit | Disable on error | Origin | Password required | Run as owner? | Failover | Server | Retain dead tuples | Max retention duration | Retention active | Messages | Synchronous commit | Conninfo | Receiver timeout | Skip LSN | Description +-----------------+---------------------------+---------+-------------+--------+-----------+------------------+------------------+--------+-------------------+---------------+----------+--------+--------------------+------------------------+------------------+----------+--------------------+-----------------------------+------------------+------------+------------- + regress_testsub | regress_subscription_user | f | {testpub} | f | on | p | f | any | t | f | f | | f | 0 | f | f | off | dbname=regress_doesnotexist | -1 | 0/00000000 | (1 row) ALTER SUBSCRIPTION regress_testsub SET (slot_name = NONE); @@ -507,10 +507,10 @@ CREATE SUBSCRIPTION regress_testsub CONNECTION 'dbname=regress_doesnotexist' PUB WARNING: subscription was created, but is not connected HINT: To initiate replication, you must manually create the replication slot, enable the subscription, and alter the subscription to refresh publications. \dRs+ - List of subscriptions - Name | Owner | Enabled | Publication | Binary | Streaming | Two-phase commit | Disable on error | Origin | Password required | Run as owner? | Failover | Server | Retain dead tuples | Max retention duration | Retention active | Synchronous commit | Conninfo | Receiver timeout | Skip LSN | Description ------------------+---------------------------+---------+-------------+--------+-----------+------------------+------------------+--------+-------------------+---------------+----------+--------+--------------------+------------------------+------------------+--------------------+-----------------------------+------------------+------------+------------- - regress_testsub | regress_subscription_user | f | {testpub} | f | on | p | f | any | t | f | f | | f | 0 | f | off | dbname=regress_doesnotexist | -1 | 0/00000000 | + List of subscriptions + Name | Owner | Enabled | Publication | Binary | Streaming | Two-phase commit | Disable on error | Origin | Password required | Run as owner? | Failover | Server | Retain dead tuples | Max retention duration | Retention active | Messages | Synchronous commit | Conninfo | Receiver timeout | Skip LSN | Description +-----------------+---------------------------+---------+-------------+--------+-----------+------------------+------------------+--------+-------------------+---------------+----------+--------+--------------------+------------------------+------------------+----------+--------------------+-----------------------------+------------------+------------+------------- + regress_testsub | regress_subscription_user | f | {testpub} | f | on | p | f | any | t | f | f | | f | 0 | f | f | off | dbname=regress_doesnotexist | -1 | 0/00000000 | (1 row) ALTER SUBSCRIPTION regress_testsub SET (slot_name = NONE); @@ -523,18 +523,18 @@ CREATE SUBSCRIPTION regress_testsub CONNECTION 'dbname=regress_doesnotexist' PUB WARNING: subscription was created, but is not connected HINT: To initiate replication, you must manually create the replication slot, enable the subscription, and alter the subscription to refresh publications. \dRs+ - List of subscriptions - Name | Owner | Enabled | Publication | Binary | Streaming | Two-phase commit | Disable on error | Origin | Password required | Run as owner? | Failover | Server | Retain dead tuples | Max retention duration | Retention active | Synchronous commit | Conninfo | Receiver timeout | Skip LSN | Description ------------------+---------------------------+---------+-------------+--------+-----------+------------------+------------------+--------+-------------------+---------------+----------+--------+--------------------+------------------------+------------------+--------------------+-----------------------------+------------------+------------+------------- - regress_testsub | regress_subscription_user | f | {testpub} | f | parallel | d | f | any | t | f | f | | f | 0 | f | off | dbname=regress_doesnotexist | -1 | 0/00000000 | + List of subscriptions + Name | Owner | Enabled | Publication | Binary | Streaming | Two-phase commit | Disable on error | Origin | Password required | Run as owner? | Failover | Server | Retain dead tuples | Max retention duration | Retention active | Messages | Synchronous commit | Conninfo | Receiver timeout | Skip LSN | Description +-----------------+---------------------------+---------+-------------+--------+-----------+------------------+------------------+--------+-------------------+---------------+----------+--------+--------------------+------------------------+------------------+----------+--------------------+-----------------------------+------------------+------------+------------- + regress_testsub | regress_subscription_user | f | {testpub} | f | parallel | d | f | any | t | f | f | | f | 0 | f | f | off | dbname=regress_doesnotexist | -1 | 0/00000000 | (1 row) ALTER SUBSCRIPTION regress_testsub SET (disable_on_error = true); \dRs+ - List of subscriptions - Name | Owner | Enabled | Publication | Binary | Streaming | Two-phase commit | Disable on error | Origin | Password required | Run as owner? | Failover | Server | Retain dead tuples | Max retention duration | Retention active | Synchronous commit | Conninfo | Receiver timeout | Skip LSN | Description ------------------+---------------------------+---------+-------------+--------+-----------+------------------+------------------+--------+-------------------+---------------+----------+--------+--------------------+------------------------+------------------+--------------------+-----------------------------+------------------+------------+------------- - regress_testsub | regress_subscription_user | f | {testpub} | f | parallel | d | t | any | t | f | f | | f | 0 | f | off | dbname=regress_doesnotexist | -1 | 0/00000000 | + List of subscriptions + Name | Owner | Enabled | Publication | Binary | Streaming | Two-phase commit | Disable on error | Origin | Password required | Run as owner? | Failover | Server | Retain dead tuples | Max retention duration | Retention active | Messages | Synchronous commit | Conninfo | Receiver timeout | Skip LSN | Description +-----------------+---------------------------+---------+-------------+--------+-----------+------------------+------------------+--------+-------------------+---------------+----------+--------+--------------------+------------------------+------------------+----------+--------------------+-----------------------------+------------------+------------+------------- + regress_testsub | regress_subscription_user | f | {testpub} | f | parallel | d | t | any | t | f | f | | f | 0 | f | f | off | dbname=regress_doesnotexist | -1 | 0/00000000 | (1 row) ALTER SUBSCRIPTION regress_testsub SET (slot_name = NONE); @@ -547,10 +547,10 @@ CREATE SUBSCRIPTION regress_testsub CONNECTION 'dbname=regress_doesnotexist' PUB WARNING: subscription was created, but is not connected HINT: To initiate replication, you must manually create the replication slot, enable the subscription, and alter the subscription to refresh publications. \dRs+ - List of subscriptions - Name | Owner | Enabled | Publication | Binary | Streaming | Two-phase commit | Disable on error | Origin | Password required | Run as owner? | Failover | Server | Retain dead tuples | Max retention duration | Retention active | Synchronous commit | Conninfo | Receiver timeout | Skip LSN | Description ------------------+---------------------------+---------+-------------+--------+-----------+------------------+------------------+--------+-------------------+---------------+----------+--------+--------------------+------------------------+------------------+--------------------+-----------------------------+------------------+------------+------------- - regress_testsub | regress_subscription_user | f | {testpub} | f | parallel | d | f | any | t | f | f | | f | 0 | f | off | dbname=regress_doesnotexist | -1 | 0/00000000 | + List of subscriptions + Name | Owner | Enabled | Publication | Binary | Streaming | Two-phase commit | Disable on error | Origin | Password required | Run as owner? | Failover | Server | Retain dead tuples | Max retention duration | Retention active | Messages | Synchronous commit | Conninfo | Receiver timeout | Skip LSN | Description +-----------------+---------------------------+---------+-------------+--------+-----------+------------------+------------------+--------+-------------------+---------------+----------+--------+--------------------+------------------------+------------------+----------+--------------------+-----------------------------+------------------+------------+------------- + regress_testsub | regress_subscription_user | f | {testpub} | f | parallel | d | f | any | t | f | f | | f | 0 | f | f | off | dbname=regress_doesnotexist | -1 | 0/00000000 | (1 row) ALTER SUBSCRIPTION regress_testsub SET (slot_name = NONE); @@ -567,10 +567,10 @@ NOTICE: max_retention_duration is ineffective when retain_dead_tuples is disabl WARNING: subscription was created, but is not connected HINT: To initiate replication, you must manually create the replication slot, enable the subscription, and alter the subscription to refresh publications. \dRs+ - List of subscriptions - Name | Owner | Enabled | Publication | Binary | Streaming | Two-phase commit | Disable on error | Origin | Password required | Run as owner? | Failover | Server | Retain dead tuples | Max retention duration | Retention active | Synchronous commit | Conninfo | Receiver timeout | Skip LSN | Description ------------------+---------------------------+---------+-------------+--------+-----------+------------------+------------------+--------+-------------------+---------------+----------+--------+--------------------+------------------------+------------------+--------------------+-----------------------------+------------------+------------+------------- - regress_testsub | regress_subscription_user | f | {testpub} | f | parallel | d | f | any | t | f | f | | f | 1000 | f | off | dbname=regress_doesnotexist | -1 | 0/00000000 | + List of subscriptions + Name | Owner | Enabled | Publication | Binary | Streaming | Two-phase commit | Disable on error | Origin | Password required | Run as owner? | Failover | Server | Retain dead tuples | Max retention duration | Retention active | Messages | Synchronous commit | Conninfo | Receiver timeout | Skip LSN | Description +-----------------+---------------------------+---------+-------------+--------+-----------+------------------+------------------+--------+-------------------+---------------+----------+--------+--------------------+------------------------+------------------+----------+--------------------+-----------------------------+------------------+------------+------------- + regress_testsub | regress_subscription_user | f | {testpub} | f | parallel | d | f | any | t | f | f | | f | 1000 | f | f | off | dbname=regress_doesnotexist | -1 | 0/00000000 | (1 row) -- fail - max_retention_duration must be non-negative @@ -579,12 +579,38 @@ ERROR: option "max_retention_duration" cannot be negative -- ok ALTER SUBSCRIPTION regress_testsub SET (max_retention_duration = 0); \dRs+ - List of subscriptions - Name | Owner | Enabled | Publication | Binary | Streaming | Two-phase commit | Disable on error | Origin | Password required | Run as owner? | Failover | Server | Retain dead tuples | Max retention duration | Retention active | Synchronous commit | Conninfo | Receiver timeout | Skip LSN | Description ------------------+---------------------------+---------+-------------+--------+-----------+------------------+------------------+--------+-------------------+---------------+----------+--------+--------------------+------------------------+------------------+--------------------+-----------------------------+------------------+------------+------------- - regress_testsub | regress_subscription_user | f | {testpub} | f | parallel | d | f | any | t | f | f | | f | 0 | f | off | dbname=regress_doesnotexist | -1 | 0/00000000 | + List of subscriptions + Name | Owner | Enabled | Publication | Binary | Streaming | Two-phase commit | Disable on error | Origin | Password required | Run as owner? | Failover | Server | Retain dead tuples | Max retention duration | Retention active | Messages | Synchronous commit | Conninfo | Receiver timeout | Skip LSN | Description +-----------------+---------------------------+---------+-------------+--------+-----------+------------------+------------------+--------+-------------------+---------------+----------+--------+--------------------+------------------------+------------------+----------+--------------------+-----------------------------+------------------+------------+------------- + regress_testsub | regress_subscription_user | f | {testpub} | f | parallel | d | f | any | t | f | f | | f | 0 | f | f | off | dbname=regress_doesnotexist | -1 | 0/00000000 | +(1 row) + +ALTER SUBSCRIPTION regress_testsub SET (slot_name = NONE); +DROP SUBSCRIPTION regress_testsub; +-- fail - messages must be boolean +CREATE SUBSCRIPTION regress_testsub CONNECTION 'dbname=regress_doesnotexist' PUBLICATION testpub WITH (connect = false, messages = foo); +ERROR: messages requires a Boolean value +-- ok +CREATE SUBSCRIPTION regress_testsub CONNECTION 'dbname=regress_doesnotexist' PUBLICATION testpub WITH (connect = false, messages = true); +WARNING: subscription was created, but is not connected +HINT: To initiate replication, you must manually create the replication slot, enable the subscription, and alter the subscription to refresh publications. +\dRs+ + List of subscriptions + Name | Owner | Enabled | Publication | Binary | Streaming | Two-phase commit | Disable on error | Origin | Password required | Run as owner? | Failover | Server | Retain dead tuples | Max retention duration | Retention active | Messages | Synchronous commit | Conninfo | Receiver timeout | Skip LSN | Description +-----------------+---------------------------+---------+-------------+--------+-----------+------------------+------------------+--------+-------------------+---------------+----------+--------+--------------------+------------------------+------------------+----------+--------------------+-----------------------------+------------------+------------+------------- + regress_testsub | regress_subscription_user | f | {testpub} | f | parallel | d | f | any | t | f | f | | f | 0 | f | t | off | dbname=regress_doesnotexist | -1 | 0/00000000 | +(1 row) + +-- ok +ALTER SUBSCRIPTION regress_testsub SET (messages = false); +\dRs+ + List of subscriptions + Name | Owner | Enabled | Publication | Binary | Streaming | Two-phase commit | Disable on error | Origin | Password required | Run as owner? | Failover | Server | Retain dead tuples | Max retention duration | Retention active | Messages | Synchronous commit | Conninfo | Receiver timeout | Skip LSN | Description +-----------------+---------------------------+---------+-------------+--------+-----------+------------------+------------------+--------+-------------------+---------------+----------+--------+--------------------+------------------------+------------------+----------+--------------------+-----------------------------+------------------+------------+------------- + regress_testsub | regress_subscription_user | f | {testpub} | f | parallel | d | f | any | t | f | f | | f | 0 | f | f | off | dbname=regress_doesnotexist | -1 | 0/00000000 | (1 row) +-- cleanup ALTER SUBSCRIPTION regress_testsub SET (slot_name = NONE); DROP SUBSCRIPTION regress_testsub; -- let's do some tests with pg_create_subscription rather than superuser diff --git a/src/test/regress/sql/subscription.sql b/src/test/regress/sql/subscription.sql index cfeebaf9302..c7d5476fe7c 100644 --- a/src/test/regress/sql/subscription.sql +++ b/src/test/regress/sql/subscription.sql @@ -439,6 +439,23 @@ ALTER SUBSCRIPTION regress_testsub SET (max_retention_duration = 0); ALTER SUBSCRIPTION regress_testsub SET (slot_name = NONE); DROP SUBSCRIPTION regress_testsub; +-- fail - messages must be boolean +CREATE SUBSCRIPTION regress_testsub CONNECTION 'dbname=regress_doesnotexist' PUBLICATION testpub WITH (connect = false, messages = foo); + +-- ok +CREATE SUBSCRIPTION regress_testsub CONNECTION 'dbname=regress_doesnotexist' PUBLICATION testpub WITH (connect = false, messages = true); + +\dRs+ + +-- ok +ALTER SUBSCRIPTION regress_testsub SET (messages = false); + +\dRs+ + +-- cleanup +ALTER SUBSCRIPTION regress_testsub SET (slot_name = NONE); +DROP SUBSCRIPTION regress_testsub; + -- let's do some tests with pg_create_subscription rather than superuser SET SESSION AUTHORIZATION regress_subscription_user3; diff --git a/src/tools/pgindent/typedefs.list b/src/tools/pgindent/typedefs.list index 5d432074c2c..2a53289e9f6 100644 --- a/src/tools/pgindent/typedefs.list +++ b/src/tools/pgindent/typedefs.list @@ -1684,6 +1684,8 @@ LogicalRepBeginData LogicalRepCommitData LogicalRepCommitPreparedTxnData LogicalRepCtxStruct +LogicalRepMessageData +LogicalRepMessageHandle_hook_type LogicalRepMsgType LogicalRepPartMapEntry LogicalRepPreparedTxnData -- 2.55.0
^ permalink raw reply [nested|flat] 44+ messages in thread

* Re: Add a hook for handling logical decoding messages on subscribers.
@ 2026-09-20 03:52  Bharath Rupireddy <bharath.rupireddyforpostgres@gmail.com>
  parent: Masahiko Sawada <sawada.mshk@gmail.com>
  1 sibling, 0 replies; 44+ messages in thread

From: Bharath Rupireddy @ 2026-09-20 03:52 UTC (permalink / raw)
  To: Masahiko Sawada <sawada.mshk@gmail.com>; +Cc: Chao Li <li.evan.chao@gmail.com>; Fujii Masao <masao.fujii@gmail.com>; Amit Kapila <amit.kapila16@gmail.com>; pgsql-hackers

Hi,

On Thu, Sep 17, 2026 at 5:43 PM Masahiko Sawada <sawada.mshk@gmail.com> wrote:
>
> Thanks for reviewing the patch!
>
> While I also think the patch is in good shape, I'd like to raise a
> security risk this feature might introduce, particularly around
> pg_logical_emit_message():

Nice catch. Thanks for bringing it up.

> EXECUTE on that function is granted to PUBLIC, so any role that can
> connect to the publisher database can emit a logical decoding message.
> With this patch the apply worker hands the message to a handler that
> runs with the privileges of the subscription owner, and unlike
> insert/update/delete there is no table owner to switch to. One use
> case I have in mind is DDL replication. If an extension implements it
> on top of this hook and the subscription is owned by a superuser, any
> role on the publisher can choose what the handler is given and have it
> executed with superuser privileges on the subscriber. So extensions
> should carefully consider this case. I think the same applies to other
> extensions that might use this feature. The real problem is that there
> is no reliable way for the subscriber to tell whether a message came
> from the source it expects.

Right. This lets a less privileged user on the publisher get more
privileged things done on the subscriber.

> A practical solution is to revoke EXECUTE on pg_logical_emit_message()
> from PUBLIC on the publisher and grant it to a role created for that
> purpose.

That seems like a good approach. Since the message emitted by
pg_logical_emit_message() can carry anything the user wants and goes
into the WAL stream, careful production systems may already restrict
the function this way. But we can't assume that is always the case.

> I think that covers most cases I did consider having the
> server record the emitting role in the message so that the subscriber
> could check it, but I'm not sure this feature alone justifies it.

I lean towards this idea, with one change. Record what the emitter can
do, not who it is. Role names and OIDs of ordinary roles on the
publisher mean nothing on the subscriber. What the apply side wants to
know is simple. What privileges did the emitter have, with role
memberships expanded recursively, so they can be compared against what
the role applying the message has. So record the emitter's role
attributes and the predefined roles it had the privileges of, and hand
that to the hook as-is. With this, a hook implementer can say "the
emitter was less privileged than me, so I'll skip this logical
message", or apply a stricter rule per message type. It also answers
the real problem mentioned above, since the subscriber can now tell
whether the logical message came from a source it trusts. I can also
think of another place where this helps. The logical message today is
replayed as-is during crash recovery, on read replicas, and in setups
where replay is moved to storage away from compute.

I quickly played with this idea using Claude Code to illustrate it
better. I haven't reviewed it in depth. Attached for reference. Please
have a look.

That said, I don't want the hook addition to be blocked on this. It
can be discussed separately since there can be other valid use cases.

> So
> my current thought is to document these risks and add nothing special
> for these cases.

Documenting the risks is good enough to proceed for now IMHO. But hook
implementers can easily miss it.

I'm also just thinking out loud here. How about a predefined role for
pg_logical_emit_message()? The apply side could switch to that role
before calling the hook, which gives logical messages the same kind of
switch that apply gets with the table owner, and limits what any
logical message can do to whatever that role is granted on the
subscriber. Or the apply side could call the hook only when the
subscription owner has been granted that role, so the hook stays off
unless someone chose to give the subscription owner that role.

> I've added the documentation changes and rebased the patch. Any ideas
> and feedback is very welcome.

I reviewed the v5 patch and the diff from the v4 patch and it looks good to me.

--
Bharath Rupireddy
Amazon Web Services: https://aws.amazon.com

Attachments:

nocfbot-v1-0001-WIP-Record-emitter-capabilities-in-logical-decodi.patch (application/x-patch, 43.3K) download From 3abb1cdb6c57643f0108bdc3d455094063f87b30 Mon Sep 17 00:00:00 2001 From: Bharath Rupireddy <bharath.rupireddyforpostgres@gmail.com> Date: Sat, 19 Sep 2026 23:19:38 +0000 Subject: [PATCH v1] WIP: Record emitter capabilities in logical decoding messages. pg_logical_emit_message() is executable by PUBLIC, so any role that can connect to a database can put a logical decoding message into WAL. A message has no object, no ACL and no owner, so a consumer has no way to tell whether one came from a privileged role or from any role that happened to be able to connect. Row changes do not have this problem: they are authorized by the ACL on the table they touch and are scoped by publications. Record the capabilities of the emitting role in the xl_logical_message record at emit time: its role attributes from pg_authid as flags, and the names of the predefined (pg_*) roles whose privileges it could exercise, with memberships expanded recursively through inheritable grants. Only capabilities are recorded, never identities. User-defined roles are left out because their names and OIDs mean nothing outside the emitting cluster, while the names of predefined roles are fixed and need no headers to interpret. A separate VALID flag keeps "nothing is known about the emitter" distinct from "the emitter holds no privileges". The capabilities reach the message and stream_message output plugin callbacks, so that a consumer can compare them against the privileges of the role applying the message. Core attaches no policy to them. pgoutput sends them under a new protocol version 5, which the built-in apply worker does not request, so nothing changes for existing subscribers. test_decoding grows an include-message-emitter option, off by default, used by the new tests. The role membership closure is exposed as get_role_privs_closure() in acl.c, a thin wrapper around roles_is_member_of(). Bump XLOG_PAGE_MAGIC for the WAL record format change. --- contrib/test_decoding/expected/messages.out | 66 ++++++++++ contrib/test_decoding/sql/messages.sql | 35 ++++++ contrib/test_decoding/test_decoding.c | 100 ++++++++++++--- src/backend/access/rmgrdesc/logicalmsgdesc.c | 25 +++- src/backend/replication/logical/decode.c | 12 +- src/backend/replication/logical/logical.c | 16 ++- src/backend/replication/logical/message.c | 118 ++++++++++++++++++ src/backend/replication/logical/proto.c | 25 +++- .../replication/logical/reorderbuffer.c | 50 +++++++- src/backend/replication/pgoutput/pgoutput.c | 9 +- src/backend/utils/adt/acl.c | 17 +++ src/include/access/xlog_internal.h | 2 +- src/include/replication/logicalproto.h | 10 +- src/include/replication/message.h | 45 ++++++- src/include/replication/output_plugin.h | 9 +- src/include/replication/reorderbuffer.h | 11 +- src/include/utils/acl.h | 1 + src/tools/pgindent/typedefs.list | 1 + 18 files changed, 505 insertions(+), 47 deletions(-) diff --git a/contrib/test_decoding/expected/messages.out b/contrib/test_decoding/expected/messages.out index 84baf8af3ee..5ed542a871d 100644 --- a/contrib/test_decoding/expected/messages.out +++ b/contrib/test_decoding/expected/messages.out @@ -98,6 +98,72 @@ SELECT data FROM pg_logical_slot_get_changes('regression_slot', NULL, NULL, 'for ------ (0 rows) +-- Memberships are expanded recursively, so regress_msg_reader shows +-- pg_read_all_data, which it holds only through regress_msg_parent. The +-- user-defined roles themselves are not recorded. +CREATE ROLE regress_msg_lowpriv; +CREATE ROLE regress_msg_parent IN ROLE pg_read_all_data; +CREATE ROLE regress_msg_reader IN ROLE regress_msg_parent; +SET SESSION AUTHORIZATION regress_msg_lowpriv; +SELECT 'msg8' FROM pg_logical_emit_message(true, 'test', 'from lowpriv'); + ?column? +---------- + msg8 +(1 row) + +RESET SESSION AUTHORIZATION; +SET SESSION AUTHORIZATION regress_msg_reader; +SELECT 'msg9' FROM pg_logical_emit_message(true, 'test', 'from reader'); + ?column? +---------- + msg9 +(1 row) + +RESET SESSION AUTHORIZATION; +SELECT 'msg10' FROM pg_logical_emit_message(true, 'test', 'from superuser'); + ?column? +---------- + msg10 +(1 row) + +SELECT data FROM pg_logical_slot_get_changes('regression_slot', NULL, NULL, 'force-binary', '0', 'skip-empty-xacts', '1', 'include-xids', '0', 'include-message-emitter', '1'); + data +----------------------------------------------------------------------------------------------------------------------------------------------------------------------- + BEGIN + message: transactional: 1 prefix: test, sz: 12 emitter: inherit content:from lowpriv + COMMIT + BEGIN + message: transactional: 1 prefix: test, sz: 11 emitter: inherit,pg_read_all_data content:from reader + COMMIT + BEGIN + message: transactional: 1 prefix: test, sz: 14 emitter: superuser,inherit,createrole,createdb,canlogin,replication,bypassrls,pg_database_owner content:from superuser + COMMIT +(9 rows) + +-- Same, but in a transaction large enough to spill to disk, so that the +-- capabilities go through serialize and restore. +CREATE TABLE messages_spill(data text); +BEGIN; +SET SESSION AUTHORIZATION regress_msg_reader; +SELECT 'msg11' FROM pg_logical_emit_message(true, 'test', 'spilled'); + ?column? +---------- + msg11 +(1 row) + +RESET SESSION AUTHORIZATION; +INSERT INTO messages_spill SELECT g.i FROM generate_series(1, 5000) g(i); +COMMIT; +SELECT data FROM pg_logical_slot_get_changes('regression_slot', NULL, NULL, 'force-binary', '0', 'skip-empty-xacts', '1', 'include-xids', '0', 'include-message-emitter', '1') WHERE data LIKE 'message:%'; + data +------------------------------------------------------------------------------------------------- + message: transactional: 1 prefix: test, sz: 7 emitter: inherit,pg_read_all_data content:spilled +(1 row) + +DROP TABLE messages_spill; +DROP ROLE regress_msg_reader; +DROP ROLE regress_msg_parent; +DROP ROLE regress_msg_lowpriv; SELECT 'cleanup' FROM pg_drop_replication_slot('regression_slot'); ?column? ---------- diff --git a/contrib/test_decoding/sql/messages.sql b/contrib/test_decoding/sql/messages.sql index 1f3dcb63ee7..0acf3228e07 100644 --- a/contrib/test_decoding/sql/messages.sql +++ b/contrib/test_decoding/sql/messages.sql @@ -32,4 +32,39 @@ SELECT 'otherdb2' FROM pg_logical_emit_message(true, 'test', 'otherdb2'); \c :prevdb SELECT data FROM pg_logical_slot_get_changes('regression_slot', NULL, NULL, 'force-binary', '0', 'skip-empty-xacts', '1'); +-- Memberships are expanded recursively, so regress_msg_reader shows +-- pg_read_all_data, which it holds only through regress_msg_parent. The +-- user-defined roles themselves are not recorded. +CREATE ROLE regress_msg_lowpriv; +CREATE ROLE regress_msg_parent IN ROLE pg_read_all_data; +CREATE ROLE regress_msg_reader IN ROLE regress_msg_parent; + +SET SESSION AUTHORIZATION regress_msg_lowpriv; +SELECT 'msg8' FROM pg_logical_emit_message(true, 'test', 'from lowpriv'); +RESET SESSION AUTHORIZATION; + +SET SESSION AUTHORIZATION regress_msg_reader; +SELECT 'msg9' FROM pg_logical_emit_message(true, 'test', 'from reader'); +RESET SESSION AUTHORIZATION; + +SELECT 'msg10' FROM pg_logical_emit_message(true, 'test', 'from superuser'); + +SELECT data FROM pg_logical_slot_get_changes('regression_slot', NULL, NULL, 'force-binary', '0', 'skip-empty-xacts', '1', 'include-xids', '0', 'include-message-emitter', '1'); + +-- Same, but in a transaction large enough to spill to disk, so that the +-- capabilities go through serialize and restore. +CREATE TABLE messages_spill(data text); +BEGIN; +SET SESSION AUTHORIZATION regress_msg_reader; +SELECT 'msg11' FROM pg_logical_emit_message(true, 'test', 'spilled'); +RESET SESSION AUTHORIZATION; +INSERT INTO messages_spill SELECT g.i FROM generate_series(1, 5000) g(i); +COMMIT; +SELECT data FROM pg_logical_slot_get_changes('regression_slot', NULL, NULL, 'force-binary', '0', 'skip-empty-xacts', '1', 'include-xids', '0', 'include-message-emitter', '1') WHERE data LIKE 'message:%'; +DROP TABLE messages_spill; + +DROP ROLE regress_msg_reader; +DROP ROLE regress_msg_parent; +DROP ROLE regress_msg_lowpriv; + SELECT 'cleanup' FROM pg_drop_replication_slot('regression_slot'); diff --git a/contrib/test_decoding/test_decoding.c b/contrib/test_decoding/test_decoding.c index 78185837aef..339aafd910c 100644 --- a/contrib/test_decoding/test_decoding.c +++ b/contrib/test_decoding/test_decoding.c @@ -15,6 +15,7 @@ #include "catalog/pg_type.h" #include "replication/logical.h" +#include "replication/message.h" #include "replication/origin.h" #include "utils/builtins.h" @@ -34,6 +35,7 @@ typedef struct bool include_timestamp; bool skip_empty_xacts; bool only_local; + bool include_message_emitter; } TestDecodingData; /* @@ -74,7 +76,8 @@ static bool pg_decode_filter(LogicalDecodingContext *ctx, static void pg_decode_message(LogicalDecodingContext *ctx, ReorderBufferTXN *txn, XLogRecPtr lsn, bool transactional, const char *prefix, - Size sz, const char *message); + Size sz, const char *message, + const LogicalMessageEmitter *emitter); static bool pg_decode_filter_prepare(LogicalDecodingContext *ctx, TransactionId xid, const char *gid); @@ -114,7 +117,8 @@ static void pg_decode_stream_change(LogicalDecodingContext *ctx, static void pg_decode_stream_message(LogicalDecodingContext *ctx, ReorderBufferTXN *txn, XLogRecPtr lsn, bool transactional, const char *prefix, - Size sz, const char *message); + Size sz, const char *message, + const LogicalMessageEmitter *emitter); static void pg_decode_stream_truncate(LogicalDecodingContext *ctx, ReorderBufferTXN *txn, int nrelations, Relation relations[], @@ -171,6 +175,7 @@ pg_decode_startup(LogicalDecodingContext *ctx, OutputPluginOptions *opt, data->include_timestamp = false; data->skip_empty_xacts = false; data->only_local = false; + data->include_message_emitter = false; ctx->output_plugin_private = data; @@ -252,6 +257,16 @@ pg_decode_startup(LogicalDecodingContext *ctx, OutputPluginOptions *opt, errmsg("could not parse value \"%s\" for parameter \"%s\"", strVal(elem->arg), elem->defname))); } + else if (strcmp(elem->defname, "include-message-emitter") == 0) + { + if (elem->arg == NULL) + data->include_message_emitter = true; + else if (!parse_bool(strVal(elem->arg), &data->include_message_emitter)) + ereport(ERROR, + (errcode(ERRCODE_INVALID_PARAMETER_VALUE), + errmsg("could not parse value \"%s\" for parameter \"%s\"", + strVal(elem->arg), elem->defname))); + } else if (strcmp(elem->defname, "stream-changes") == 0) { if (elem->arg == NULL) @@ -751,10 +766,62 @@ pg_decode_truncate(LogicalDecodingContext *ctx, ReorderBufferTXN *txn, OutputPluginWrite(ctx, true); } +/* + * Print the emitter's role attributes, then its predefined roles. + */ +static void +print_emitter(StringInfo s, const LogicalMessageEmitter *emitter) +{ + static const struct + { + uint32 flag; + const char *name; + } attributes[] = { + {LOGICALMSG_EMITTER_SUPERUSER, "superuser"}, + {LOGICALMSG_EMITTER_INHERIT, "inherit"}, + {LOGICALMSG_EMITTER_CREATEROLE, "createrole"}, + {LOGICALMSG_EMITTER_CREATEDB, "createdb"}, + {LOGICALMSG_EMITTER_CANLOGIN, "canlogin"}, + {LOGICALMSG_EMITTER_REPLICATION, "replication"}, + {LOGICALMSG_EMITTER_BYPASSRLS, "bypassrls"}, + }; + const char *rolename = emitter->roles; + const char *sep = ""; + + appendStringInfoString(s, " emitter: "); + + if (!(emitter->flags & LOGICALMSG_EMITTER_VALID)) + { + appendStringInfoString(s, "unknown"); + return; + } + + for (int i = 0; i < lengthof(attributes); i++) + { + if (emitter->flags & attributes[i].flag) + { + appendStringInfo(s, "%s%s", sep, attributes[i].name); + sep = ","; + } + } + + for (uint32 i = 0; i < emitter->nroles; i++) + { + appendStringInfo(s, "%s%s", sep, rolename); + rolename += strlen(rolename) + 1; + sep = ","; + } + + /* no attributes and no predefined roles */ + if (sep[0] == '\0') + appendStringInfoString(s, "unprivileged"); +} + static void pg_decode_message(LogicalDecodingContext *ctx, ReorderBufferTXN *txn, XLogRecPtr lsn, bool transactional, - const char *prefix, Size sz, const char *message) + const char *prefix, Size sz, const char *message, + const LogicalMessageEmitter *emitter) { TestDecodingData *data = ctx->output_plugin_private; TestDecodingTxnData *txndata; @@ -769,8 +836,11 @@ pg_decode_message(LogicalDecodingContext *ctx, txndata->xact_wrote_changes = true; OutputPluginPrepareWrite(ctx, true); - appendStringInfo(ctx->out, "message: transactional: %d prefix: %s, sz: %zu content:", + appendStringInfo(ctx->out, "message: transactional: %d prefix: %s, sz: %zu", transactional, prefix, sz); + if (data->include_message_emitter) + print_emitter(ctx->out, emitter); + appendStringInfoString(ctx->out, " content:"); appendBinaryStringInfo(ctx->out, message, sz); OutputPluginWrite(ctx, true); } @@ -955,12 +1025,14 @@ pg_decode_stream_change(LogicalDecodingContext *ctx, static void pg_decode_stream_message(LogicalDecodingContext *ctx, ReorderBufferTXN *txn, XLogRecPtr lsn, bool transactional, - const char *prefix, Size sz, const char *message) + const char *prefix, Size sz, const char *message, + const LogicalMessageEmitter *emitter) { + TestDecodingData *data = ctx->output_plugin_private; + /* Output stream start if we haven't yet for transactional messages. */ if (transactional) { - TestDecodingData *data = ctx->output_plugin_private; TestDecodingTxnData *txndata = txn->output_plugin_private; if (data->skip_empty_xacts && !txndata->stream_wrote_changes) @@ -972,15 +1044,15 @@ pg_decode_stream_message(LogicalDecodingContext *ctx, OutputPluginPrepareWrite(ctx, true); - if (transactional) - { - appendStringInfo(ctx->out, "streaming message: transactional: %d prefix: %s, sz: %zu", - transactional, prefix, sz); - } - else + appendStringInfo(ctx->out, "streaming message: transactional: %d prefix: %s, sz: %zu", + transactional, prefix, sz); + + if (data->include_message_emitter) + print_emitter(ctx->out, emitter); + + if (!transactional) { - appendStringInfo(ctx->out, "streaming message: transactional: %d prefix: %s, sz: %zu content:", - transactional, prefix, sz); + appendStringInfoString(ctx->out, " content:"); appendBinaryStringInfo(ctx->out, message, sz); } diff --git a/src/backend/access/rmgrdesc/logicalmsgdesc.c b/src/backend/access/rmgrdesc/logicalmsgdesc.c index 3e8a4f33ee6..b6e7e2c2bf2 100644 --- a/src/backend/access/rmgrdesc/logicalmsgdesc.c +++ b/src/backend/access/rmgrdesc/logicalmsgdesc.c @@ -24,15 +24,32 @@ logicalmsg_desc(StringInfo buf, XLogReaderState *record) if (info == XLOG_LOGICAL_MESSAGE) { xl_logical_message *xlrec = (xl_logical_message *) rec; - char *prefix = xlrec->message; - char *message = xlrec->message + xlrec->prefix_size; + char *prefix = XLLogicalMessagePrefix(xlrec); + char *message = XLLogicalMessagePayload(xlrec); char *sep = ""; Assert(prefix[xlrec->prefix_size - 1] == '\0'); - appendStringInfo(buf, "%s, prefix \"%s\"; payload (%zu bytes): ", + appendStringInfo(buf, "%s, prefix \"%s\"; emitter flags %X", xlrec->transactional ? "transactional" : "non-transactional", - prefix, xlrec->message_size); + prefix, xlrec->emitter_flags); + + if (xlrec->emitter_nroles > 0) + { + char *rolename = XLLogicalMessageRoles(xlrec); + char *rolesep = ""; + + appendStringInfoString(buf, ", emitter roles "); + for (uint32 i = 0; i < xlrec->emitter_nroles; i++) + { + appendStringInfo(buf, "%s%s", rolesep, rolename); + rolename += strlen(rolename) + 1; + rolesep = ","; + } + } + + appendStringInfo(buf, "; payload (%zu bytes): ", + xlrec->message_size); /* Write message payload as a series of hex bytes */ for (Size cnt = 0; cnt < xlrec->message_size; cnt++) { diff --git a/src/backend/replication/logical/decode.c b/src/backend/replication/logical/decode.c index 4a739230264..c22d8d5f54e 100644 --- a/src/backend/replication/logical/decode.c +++ b/src/backend/replication/logical/decode.c @@ -613,6 +613,7 @@ logicalmsg_decode(LogicalDecodingContext *ctx, XLogRecordBuffer *buf) ReplOriginId origin_id = XLogRecGetOrigin(r); Snapshot snapshot = NULL; xl_logical_message *message; + LogicalMessageEmitter emitter; if (info != XLOG_LOGICAL_MESSAGE) elog(ERROR, "unexpected RM_LOGICALMSG_ID record type: %u", info); @@ -668,12 +669,17 @@ logicalmsg_decode(LogicalDecodingContext *ctx, XLogRecordBuffer *buf) if (!message->transactional) snapshot = SnapBuildGetOrBuildSnapshot(builder); + emitter.flags = message->emitter_flags; + emitter.nroles = message->emitter_nroles; + emitter.roles_size = message->emitter_roles_size; + emitter.roles = XLLogicalMessageRoles(message); + ReorderBufferQueueMessage(ctx->reorder, xid, snapshot, buf->endptr, message->transactional, - message->message, /* first part of message is - * prefix */ + XLLogicalMessagePrefix(message), message->message_size, - message->message + message->prefix_size); + XLLogicalMessagePayload(message), + &emitter); } /* diff --git a/src/backend/replication/logical/logical.c b/src/backend/replication/logical/logical.c index 98e5f1dd8f9..034cbf3a9d8 100644 --- a/src/backend/replication/logical/logical.c +++ b/src/backend/replication/logical/logical.c @@ -81,7 +81,8 @@ static void truncate_cb_wrapper(ReorderBuffer *cache, ReorderBufferTXN *txn, int nrelations, Relation relations[], ReorderBufferChange *change); static void message_cb_wrapper(ReorderBuffer *cache, ReorderBufferTXN *txn, XLogRecPtr message_lsn, bool transactional, - const char *prefix, Size message_size, const char *message); + const char *prefix, Size message_size, const char *message, + const LogicalMessageEmitter *emitter); /* streaming callbacks */ static void stream_start_cb_wrapper(ReorderBuffer *cache, ReorderBufferTXN *txn, @@ -98,7 +99,8 @@ static void stream_change_cb_wrapper(ReorderBuffer *cache, ReorderBufferTXN *txn Relation relation, ReorderBufferChange *change); static void stream_message_cb_wrapper(ReorderBuffer *cache, ReorderBufferTXN *txn, XLogRecPtr message_lsn, bool transactional, - const char *prefix, Size message_size, const char *message); + const char *prefix, Size message_size, const char *message, + const LogicalMessageEmitter *emitter); static void stream_truncate_cb_wrapper(ReorderBuffer *cache, ReorderBufferTXN *txn, int nrelations, Relation relations[], ReorderBufferChange *change); @@ -1311,7 +1313,8 @@ filter_by_origin_cb_wrapper(LogicalDecodingContext *ctx, ReplOriginId origin_id) static void message_cb_wrapper(ReorderBuffer *cache, ReorderBufferTXN *txn, XLogRecPtr message_lsn, bool transactional, - const char *prefix, Size message_size, const char *message) + const char *prefix, Size message_size, const char *message, + const LogicalMessageEmitter *emitter) { LogicalDecodingContext *ctx = cache->private_data; LogicalErrorCallbackState state; @@ -1339,7 +1342,7 @@ message_cb_wrapper(ReorderBuffer *cache, ReorderBufferTXN *txn, /* do the actual work: call callback */ ctx->callbacks.message_cb(ctx, txn, message_lsn, transactional, prefix, - message_size, message); + message_size, message, emitter); /* Pop the error context stack */ error_context_stack = errcallback.previous; @@ -1622,7 +1625,8 @@ stream_change_cb_wrapper(ReorderBuffer *cache, ReorderBufferTXN *txn, static void stream_message_cb_wrapper(ReorderBuffer *cache, ReorderBufferTXN *txn, XLogRecPtr message_lsn, bool transactional, - const char *prefix, Size message_size, const char *message) + const char *prefix, Size message_size, const char *message, + const LogicalMessageEmitter *emitter) { LogicalDecodingContext *ctx = cache->private_data; LogicalErrorCallbackState state; @@ -1654,7 +1658,7 @@ stream_message_cb_wrapper(ReorderBuffer *cache, ReorderBufferTXN *txn, /* do the actual work: call callback */ ctx->callbacks.stream_message_cb(ctx, txn, message_lsn, transactional, prefix, - message_size, message); + message_size, message, emitter); /* Pop the error context stack */ error_context_stack = errcallback.previous; diff --git a/src/backend/replication/logical/message.c b/src/backend/replication/logical/message.c index 06825d66e7f..a1f2f3ce467 100644 --- a/src/backend/replication/logical/message.c +++ b/src/backend/replication/logical/message.c @@ -31,10 +31,117 @@ #include "postgres.h" +#include "access/htup_details.h" +#include "access/transam.h" #include "access/xact.h" #include "access/xloginsert.h" +#include "catalog/pg_authid.h" #include "miscadmin.h" #include "replication/message.h" +#include "utils/acl.h" +#include "utils/syscache.h" + +/* + * Collect the capabilities of the role emitting a message; see + * LogicalMessageEmitter. The role names are sorted, so that the same + * capabilities always produce the same bytes. + */ +static void +GetEmitterCapabilities(LogicalMessageEmitter *emitter) +{ + Oid userid = GetUserId(); + HeapTuple tuple; + List *memberships; + ListCell *lc; + char **names; + int nnames = 0; + StringInfoData buf; + + emitter->flags = 0; + emitter->nroles = 0; + emitter->roles_size = 0; + emitter->roles = NULL; + + /* + * Role lookups need catalog access, which we don't have when C code emits + * a non-transactional message outside a transaction. Record nothing + * rather than something wrong. + */ + if (!IsTransactionState()) + return; + + emitter->flags = LOGICALMSG_EMITTER_VALID; + + tuple = SearchSysCache1(AUTHOID, ObjectIdGetDatum(userid)); + if (HeapTupleIsValid(tuple)) + { + Form_pg_authid authform = (Form_pg_authid) GETSTRUCT(tuple); + + if (authform->rolsuper) + emitter->flags |= LOGICALMSG_EMITTER_SUPERUSER; + if (authform->rolinherit) + emitter->flags |= LOGICALMSG_EMITTER_INHERIT; + if (authform->rolcreaterole) + emitter->flags |= LOGICALMSG_EMITTER_CREATEROLE; + if (authform->rolcreatedb) + emitter->flags |= LOGICALMSG_EMITTER_CREATEDB; + if (authform->rolcanlogin) + emitter->flags |= LOGICALMSG_EMITTER_CANLOGIN; + if (authform->rolreplication) + emitter->flags |= LOGICALMSG_EMITTER_REPLICATION; + if (authform->rolbypassrls) + emitter->flags |= LOGICALMSG_EMITTER_BYPASSRLS; + + ReleaseSysCache(tuple); + } + + /* pick the predefined roles out of the emitter's memberships */ + memberships = get_role_privs_closure(userid); + names = palloc(list_length(memberships) * sizeof(char *)); + + foreach(lc, memberships) + { + Oid roleid = lfirst_oid(lc); + char *rolename; + + /* every predefined role has an OID assigned at bootstrap */ + if (roleid >= FirstNormalObjectId) + continue; + + rolename = GetUserNameFromId(roleid, true); + if (rolename == NULL) + continue; /* concurrently dropped */ + + /* the "pg_" prefix is reserved, so this is not a user-defined role */ + if (strncmp(rolename, "pg_", 3) == 0) + names[nnames++] = rolename; + else + pfree(rolename); + } + + list_free(memberships); + + if (nnames == 0) + { + pfree(names); + return; + } + + qsort(names, nnames, sizeof(char *), pg_qsort_strcmp); + + initStringInfo(&buf); + for (int i = 0; i < nnames; i++) + { + appendStringInfoString(&buf, names[i]); + appendStringInfoChar(&buf, '\0'); + pfree(names[i]); + } + pfree(names); + + emitter->nroles = nnames; + emitter->roles_size = buf.len; + emitter->roles = buf.data; +} /* * Write logical decoding message into XLog. @@ -45,6 +152,7 @@ LogLogicalMessage(const char *prefix, const char *message, size_t size, { xl_logical_message xlrec; XLogRecPtr lsn; + LogicalMessageEmitter emitter; /* * Force xid to be allocated if we're emitting a transactional message. @@ -55,7 +163,12 @@ LogLogicalMessage(const char *prefix, const char *message, size_t size, GetCurrentTransactionId(); } + GetEmitterCapabilities(&emitter); + xlrec.dbId = MyDatabaseId; + xlrec.emitter_flags = emitter.flags; + xlrec.emitter_nroles = emitter.nroles; + xlrec.emitter_roles_size = emitter.roles_size; xlrec.transactional = transactional; /* trailing zero is critical; see logicalmsg_desc */ xlrec.prefix_size = strlen(prefix) + 1; @@ -65,12 +178,17 @@ LogLogicalMessage(const char *prefix, const char *message, size_t size, XLogRegisterData(&xlrec, SizeOfLogicalMessage); XLogRegisterData(prefix, xlrec.prefix_size); XLogRegisterData(message, size); + if (emitter.roles_size > 0) + XLogRegisterData(emitter.roles, emitter.roles_size); /* allow origin filtering */ XLogSetRecordFlags(XLOG_INCLUDE_ORIGIN); lsn = XLogInsert(RM_LOGICALMSG_ID, XLOG_LOGICAL_MESSAGE); + if (emitter.roles != NULL) + pfree(emitter.roles); + /* * Make sure that the message hits disk before leaving if emitting a * non-transactional message when flush is requested. diff --git a/src/backend/replication/logical/proto.c b/src/backend/replication/logical/proto.c index 86ad97cd937..a88eb723ff0 100644 --- a/src/backend/replication/logical/proto.c +++ b/src/backend/replication/logical/proto.c @@ -26,6 +26,7 @@ #define LOGICALREP_IS_REPLICA_IDENTITY 1 #define MESSAGE_TRANSACTIONAL (1<<0) +#define MESSAGE_EMITTER_INFO (1<<1) #define TRUNCATE_CASCADE (1<<0) #define TRUNCATE_RESTART_SEQS (1<<1) @@ -635,11 +636,17 @@ logicalrep_read_truncate(StringInfo in, /* * Write MESSAGE to stream + * + * include_emitter says whether to send the emitter's capabilities, which only + * a subscriber on protocol version + * LOGICALREP_PROTO_MESSAGE_EMITTER_VERSION_NUM or above gets. The + * MESSAGE_EMITTER_INFO flag tells the reader whether they are there. */ void logicalrep_write_message(StringInfo out, TransactionId xid, XLogRecPtr lsn, bool transactional, const char *prefix, Size sz, - const char *message) + const char *message, bool include_emitter, + const LogicalMessageEmitter *emitter) { uint8 flags = 0; @@ -648,12 +655,28 @@ logicalrep_write_message(StringInfo out, TransactionId xid, XLogRecPtr lsn, /* encode and send message flags */ if (transactional) flags |= MESSAGE_TRANSACTIONAL; + if (include_emitter) + flags |= MESSAGE_EMITTER_INFO; /* transaction ID (if not valid, we're not streaming) */ if (TransactionIdIsValid(xid)) pq_sendint32(out, xid); pq_sendint8(out, flags); + + if (include_emitter) + { + const char *rolename = emitter->roles; + + pq_sendint32(out, emitter->flags); + pq_sendint32(out, emitter->nroles); + for (uint32 i = 0; i < emitter->nroles; i++) + { + pq_sendstring(out, rolename); + rolename += strlen(rolename) + 1; + } + } + pq_sendint64(out, lsn); pq_sendstring(out, prefix); pq_sendint32(out, sz); diff --git a/src/backend/replication/logical/reorderbuffer.c b/src/backend/replication/logical/reorderbuffer.c index ede117d339e..34e8ba30f27 100644 --- a/src/backend/replication/logical/reorderbuffer.c +++ b/src/backend/replication/logical/reorderbuffer.c @@ -554,6 +554,9 @@ ReorderBufferFreeChange(ReorderBuffer *rb, ReorderBufferChange *change, if (change->data.msg.message != NULL) pfree(change->data.msg.message); change->data.msg.message = NULL; + if (change->data.msg.emitter.roles != NULL) + pfree(change->data.msg.emitter.roles); + change->data.msg.emitter.roles = NULL; break; case REORDER_BUFFER_CHANGE_INVALIDATION: if (change->data.inval.invalidations) @@ -874,7 +877,8 @@ void ReorderBufferQueueMessage(ReorderBuffer *rb, TransactionId xid, Snapshot snap, XLogRecPtr lsn, bool transactional, const char *prefix, - Size message_size, const char *message) + Size message_size, const char *message, + const LogicalMessageEmitter *emitter) { if (transactional) { @@ -898,6 +902,15 @@ ReorderBufferQueueMessage(ReorderBuffer *rb, TransactionId xid, change->data.msg.message_size = message_size; change->data.msg.message = palloc(message_size); memcpy(change->data.msg.message, message, message_size); + change->data.msg.emitter = *emitter; + if (emitter->roles_size > 0) + { + change->data.msg.emitter.roles = palloc(emitter->roles_size); + memcpy(change->data.msg.emitter.roles, emitter->roles, + emitter->roles_size); + } + else + change->data.msg.emitter.roles = NULL; ReorderBufferQueueChange(rb, xid, lsn, change, false); @@ -918,7 +931,8 @@ ReorderBufferQueueMessage(ReorderBuffer *rb, TransactionId xid, SetupHistoricSnapshot(snapshot_now, NULL); PG_TRY(); { - rb->message(rb, txn, lsn, false, prefix, message_size, message); + rb->message(rb, txn, lsn, false, prefix, message_size, message, + emitter); TeardownHistoricSnapshot(false); } @@ -2105,12 +2119,14 @@ ReorderBufferApplyMessage(ReorderBuffer *rb, ReorderBufferTXN *txn, rb->stream_message(rb, txn, change->lsn, true, change->data.msg.prefix, change->data.msg.message_size, - change->data.msg.message); + change->data.msg.message, + &change->data.msg.emitter); else rb->message(rb, txn, change->lsn, true, change->data.msg.prefix, change->data.msg.message_size, - change->data.msg.message); + change->data.msg.message, + &change->data.msg.emitter); } /* @@ -4193,9 +4209,10 @@ ReorderBufferSerializeChange(ReorderBuffer *rb, ReorderBufferTXN *txn, { char *data; Size prefix_size = strlen(change->data.msg.prefix) + 1; + Size roles_size = change->data.msg.emitter.roles_size; sz += prefix_size + change->data.msg.message_size + - sizeof(Size) + sizeof(Size); + sizeof(Size) + sizeof(Size) + roles_size; ReorderBufferSerializeReserve(rb, sz); data = ((char *) rb->outbuf) + sizeof(ReorderBufferDiskChange); @@ -4217,6 +4234,13 @@ ReorderBufferSerializeChange(ReorderBuffer *rb, ReorderBufferTXN *txn, change->data.msg.message_size); data += change->data.msg.message_size; + /* write the role names; their size is in the change itself */ + if (roles_size > 0) + { + memcpy(data, change->data.msg.emitter.roles, roles_size); + data += roles_size; + } + break; } case REORDER_BUFFER_CHANGE_INVALIDATION: @@ -4529,7 +4553,8 @@ ReorderBufferChangeSize(ReorderBufferChange *change) Size prefix_size = strlen(change->data.msg.prefix) + 1; sz += prefix_size + change->data.msg.message_size + - sizeof(Size) + sizeof(Size); + sizeof(Size) + sizeof(Size) + + change->data.msg.emitter.roles_size; break; } @@ -4808,6 +4833,19 @@ ReorderBufferRestoreChange(ReorderBuffer *rb, ReorderBufferTXN *txn, change->data.msg.message_size); data += change->data.msg.message_size; + /* read the role names, whose size we already have */ + if (change->data.msg.emitter.roles_size > 0) + { + change->data.msg.emitter.roles = + MemoryContextAlloc(rb->context, + change->data.msg.emitter.roles_size); + memcpy(change->data.msg.emitter.roles, data, + change->data.msg.emitter.roles_size); + data += change->data.msg.emitter.roles_size; + } + else + change->data.msg.emitter.roles = NULL; + break; } case REORDER_BUFFER_CHANGE_INVALIDATION: diff --git a/src/backend/replication/pgoutput/pgoutput.c b/src/backend/replication/pgoutput/pgoutput.c index b481d5bb388..144fc18d960 100644 --- a/src/backend/replication/pgoutput/pgoutput.c +++ b/src/backend/replication/pgoutput/pgoutput.c @@ -57,7 +57,8 @@ static void pgoutput_truncate(LogicalDecodingContext *ctx, static void pgoutput_message(LogicalDecodingContext *ctx, ReorderBufferTXN *txn, XLogRecPtr message_lsn, bool transactional, const char *prefix, - Size sz, const char *message); + Size sz, const char *message, + const LogicalMessageEmitter *emitter); static bool pgoutput_origin_filter(LogicalDecodingContext *ctx, ReplOriginId origin_id); static void pgoutput_begin_prepare_txn(LogicalDecodingContext *ctx, @@ -1724,7 +1725,7 @@ pgoutput_truncate(LogicalDecodingContext *ctx, ReorderBufferTXN *txn, static void pgoutput_message(LogicalDecodingContext *ctx, ReorderBufferTXN *txn, XLogRecPtr message_lsn, bool transactional, const char *prefix, Size sz, - const char *message) + const char *message, const LogicalMessageEmitter *emitter) { PGOutputData *data = (PGOutputData *) ctx->output_plugin_private; TransactionId xid = InvalidTransactionId; @@ -1758,7 +1759,9 @@ pgoutput_message(LogicalDecodingContext *ctx, ReorderBufferTXN *txn, transactional, prefix, sz, - message); + message, + data->protocol_version >= LOGICALREP_PROTO_MESSAGE_EMITTER_VERSION_NUM, + emitter); OutputPluginWrite(ctx, true); } diff --git a/src/backend/utils/adt/acl.c b/src/backend/utils/adt/acl.c index a01d8f4bc41..1fb91ce1a72 100644 --- a/src/backend/utils/adt/acl.c +++ b/src/backend/utils/adt/acl.c @@ -5298,6 +5298,23 @@ roles_is_member_of(Oid roleid, enum RoleRecurseType type, } +/* + * get_role_privs_closure + * Return the OIDs of all roles whose privileges roleid can exercise: + * roleid itself plus its memberships expanded recursively through + * inheritable grants. + * + * The result is a copy the caller owns, not the cached list that + * roles_is_member_of() returns. Note that only actual memberships are + * returned; a superuser has every role's privileges implicitly. + */ +List * +get_role_privs_closure(Oid roleid) +{ + return list_copy(roles_is_member_of(roleid, ROLERECURSE_PRIVS, + InvalidOid, NULL)); +} + /* * Does member have the privileges of role (directly or indirectly)? * diff --git a/src/include/access/xlog_internal.h b/src/include/access/xlog_internal.h index bf609c3c703..ed8f488dda5 100644 --- a/src/include/access/xlog_internal.h +++ b/src/include/access/xlog_internal.h @@ -32,7 +32,7 @@ /* * Each page of XLOG file has a header like this: */ -#define XLOG_PAGE_MAGIC 0xD122 /* can be used as WAL version indicator */ +#define XLOG_PAGE_MAGIC 0xD123 /* can be used as WAL version indicator */ typedef struct XLogPageHeaderData { diff --git a/src/include/replication/logicalproto.h b/src/include/replication/logicalproto.h index 058a955e20c..fe9f7b0acc9 100644 --- a/src/include/replication/logicalproto.h +++ b/src/include/replication/logicalproto.h @@ -36,13 +36,17 @@ * LOGICALREP_PROTO_STREAM_PARALLEL_VERSION_NUM is the minimum protocol version * where we support applying large streaming transactions in parallel. * Introduced in PG16. + * + * LOGICALREP_PROTO_MESSAGE_EMITTER_VERSION_NUM is the minimum protocol + * version where a message carries the capabilities of its emitter. */ #define LOGICALREP_PROTO_MIN_VERSION_NUM 1 #define LOGICALREP_PROTO_VERSION_NUM 1 #define LOGICALREP_PROTO_STREAM_VERSION_NUM 2 #define LOGICALREP_PROTO_TWOPHASE_VERSION_NUM 3 #define LOGICALREP_PROTO_STREAM_PARALLEL_VERSION_NUM 4 -#define LOGICALREP_PROTO_MAX_VERSION_NUM LOGICALREP_PROTO_STREAM_PARALLEL_VERSION_NUM +#define LOGICALREP_PROTO_MESSAGE_EMITTER_VERSION_NUM 5 +#define LOGICALREP_PROTO_MAX_VERSION_NUM LOGICALREP_PROTO_MESSAGE_EMITTER_VERSION_NUM /* * Logical message types @@ -247,7 +251,9 @@ extern void logicalrep_write_truncate(StringInfo out, TransactionId xid, extern List *logicalrep_read_truncate(StringInfo in, bool *cascade, bool *restart_seqs); extern void logicalrep_write_message(StringInfo out, TransactionId xid, XLogRecPtr lsn, - bool transactional, const char *prefix, Size sz, const char *message); + bool transactional, const char *prefix, Size sz, + const char *message, bool include_emitter, + const LogicalMessageEmitter *emitter); extern void logicalrep_write_rel(StringInfo out, TransactionId xid, Relation rel, Bitmapset *columns, PublishGencolsType include_gencols_type); diff --git a/src/include/replication/message.h b/src/include/replication/message.h index d1c72755084..2243a4eb43b 100644 --- a/src/include/replication/message.h +++ b/src/include/replication/message.h @@ -14,21 +14,64 @@ #include "access/xlogdefs.h" #include "access/xlogreader.h" +/* + * Capabilities of the role that emitted a logical decoding message, as of the + * time it was emitted. + * + * flags carries the emitting role's own attributes, and roles the names of + * the predefined (pg_*) roles whose privileges it could exercise, that is, + * its memberships expanded recursively through inheritable grants. + * User-defined roles are left out because their names and OIDs mean nothing + * outside the emitting cluster. + * + * LOGICALMSG_EMITTER_VALID separates an emitter known to hold no privileges + * from one that nothing is known about. + */ +typedef struct LogicalMessageEmitter +{ + uint32 flags; /* LOGICALMSG_EMITTER_* bits */ + uint32 nroles; /* number of predefined role names */ + uint32 roles_size; /* total size of the packed names */ + char *roles; /* nroles null-terminated names, sorted */ +} LogicalMessageEmitter; + +/* the attribute bits follow pg_authid's columns, in column order */ +#define LOGICALMSG_EMITTER_VALID (1 << 0) +#define LOGICALMSG_EMITTER_SUPERUSER (1 << 1) /* rolsuper */ +#define LOGICALMSG_EMITTER_INHERIT (1 << 2) /* rolinherit */ +#define LOGICALMSG_EMITTER_CREATEROLE (1 << 3) /* rolcreaterole */ +#define LOGICALMSG_EMITTER_CREATEDB (1 << 4) /* rolcreatedb */ +#define LOGICALMSG_EMITTER_CANLOGIN (1 << 5) /* rolcanlogin */ +#define LOGICALMSG_EMITTER_REPLICATION (1 << 6) /* rolreplication */ +#define LOGICALMSG_EMITTER_BYPASSRLS (1 << 7) /* rolbypassrls */ + /* * Generic logical decoding message wal record. */ typedef struct xl_logical_message { Oid dbId; /* database Oid emitted from */ + uint32 emitter_flags; /* LOGICALMSG_EMITTER_* bits */ + uint32 emitter_nroles; /* number of predefined role names */ + uint32 emitter_roles_size; /* total size of the packed names */ bool transactional; /* is message transactional? */ Size prefix_size; /* length of prefix */ Size message_size; /* size of the message */ - /* payload, including null-terminated prefix of length prefix_size */ + + /* payload: prefix, then message, then emitter_nroles role names */ char message[FLEXIBLE_ARRAY_MEMBER]; } xl_logical_message; #define SizeOfLogicalMessage (offsetof(xl_logical_message, message)) +/* payload accessors */ +#define XLLogicalMessagePrefix(xlrec) \ + ((xlrec)->message) +#define XLLogicalMessagePayload(xlrec) \ + ((xlrec)->message + (xlrec)->prefix_size) +#define XLLogicalMessageRoles(xlrec) \ + (XLLogicalMessagePayload(xlrec) + (xlrec)->message_size) + extern XLogRecPtr LogLogicalMessage(const char *prefix, const char *message, size_t size, bool transactional, bool flush); diff --git a/src/include/replication/output_plugin.h b/src/include/replication/output_plugin.h index 842fcde67f9..6a3fb100a52 100644 --- a/src/include/replication/output_plugin.h +++ b/src/include/replication/output_plugin.h @@ -80,7 +80,8 @@ typedef void (*LogicalDecodeCommitCB) (struct LogicalDecodingContext *ctx, XLogRecPtr commit_lsn); /* - * Called for the generic logical decoding messages. + * Called for the generic logical decoding messages. emitter describes what + * the emitting role was allowed to do; see LogicalMessageEmitter. */ typedef void (*LogicalDecodeMessageCB) (struct LogicalDecodingContext *ctx, ReorderBufferTXN *txn, @@ -88,7 +89,8 @@ typedef void (*LogicalDecodeMessageCB) (struct LogicalDecodingContext *ctx, bool transactional, const char *prefix, Size message_size, - const char *message); + const char *message, + const LogicalMessageEmitter *emitter); /* * Filter changes by origin. @@ -199,7 +201,8 @@ typedef void (*LogicalDecodeStreamMessageCB) (struct LogicalDecodingContext *ctx bool transactional, const char *prefix, Size message_size, - const char *message); + const char *message, + const LogicalMessageEmitter *emitter); /* * Callback for streaming truncates from in-progress transactions. diff --git a/src/include/replication/reorderbuffer.h b/src/include/replication/reorderbuffer.h index ff825e4b7b2..0ed7f2f1cea 100644 --- a/src/include/replication/reorderbuffer.h +++ b/src/include/replication/reorderbuffer.h @@ -12,6 +12,7 @@ #include "access/htup_details.h" #include "lib/ilist.h" #include "lib/pairingheap.h" +#include "replication/message.h" #include "storage/sinval.h" #include "utils/hsearch.h" #include "utils/relcache.h" @@ -124,6 +125,7 @@ typedef struct ReorderBufferChange char *prefix; Size message_size; char *message; + LogicalMessageEmitter emitter; /* capabilities of the emitter */ } msg; /* New snapshot, set when action == *_INTERNAL_SNAPSHOT */ @@ -498,7 +500,8 @@ typedef void (*ReorderBufferMessageCB) (ReorderBuffer *rb, XLogRecPtr message_lsn, bool transactional, const char *prefix, Size sz, - const char *message); + const char *message, + const LogicalMessageEmitter *emitter); /* begin prepare callback signature */ typedef void (*ReorderBufferBeginPrepareCB) (ReorderBuffer *rb, @@ -557,7 +560,8 @@ typedef void (*ReorderBufferStreamMessageCB) (ReorderBuffer *rb, XLogRecPtr message_lsn, bool transactional, const char *prefix, Size sz, - const char *message); + const char *message, + const LogicalMessageEmitter *emitter); /* stream truncate callback signature */ typedef void (*ReorderBufferStreamTruncateCB) (ReorderBuffer *rb, @@ -721,7 +725,8 @@ extern void ReorderBufferQueueChange(ReorderBuffer *rb, TransactionId xid, extern void ReorderBufferQueueMessage(ReorderBuffer *rb, TransactionId xid, Snapshot snap, XLogRecPtr lsn, bool transactional, const char *prefix, - Size message_size, const char *message); + Size message_size, const char *message, + const LogicalMessageEmitter *emitter); extern void ReorderBufferCommit(ReorderBuffer *rb, TransactionId xid, XLogRecPtr commit_lsn, XLogRecPtr end_lsn, TimestampTz commit_time, ReplOriginId origin_id, XLogRecPtr origin_lsn); diff --git a/src/include/utils/acl.h b/src/include/utils/acl.h index 13fe6df0988..8f61151b145 100644 --- a/src/include/utils/acl.h +++ b/src/include/utils/acl.h @@ -209,6 +209,7 @@ extern AclMode aclmask(const Acl *acl, Oid roleid, Oid ownerId, AclMode mask, AclMaskHow how); extern int aclmembers(const Acl *acl, Oid **roleids); +extern List *get_role_privs_closure(Oid roleid); extern bool has_privs_of_role(Oid member, Oid role); extern bool member_can_set_role(Oid member, Oid role); extern void check_can_set_role(Oid member, Oid role); diff --git a/src/tools/pgindent/typedefs.list b/src/tools/pgindent/typedefs.list index 5d432074c2c..75848653349 100644 --- a/src/tools/pgindent/typedefs.list +++ b/src/tools/pgindent/typedefs.list @@ -1676,6 +1676,7 @@ LogicalDecodeTruncateCB LogicalDecodingContext LogicalDecodingCtlData LogicalErrorCallbackState +LogicalMessageEmitter LogicalOutputPluginInit LogicalOutputPluginWriterPrepareWrite LogicalOutputPluginWriterUpdateProgress -- 2.47.3
^ permalink raw reply [nested|flat] 44+ messages in thread

* Re: Add a hook for handling logical decoding messages on subscribers.
@ 2026-09-21 06:23  Amit Kapila <amit.kapila16@gmail.com>
  parent: Masahiko Sawada <sawada.mshk@gmail.com>
  1 sibling, 3 replies; 44+ messages in thread

From: Amit Kapila @ 2026-09-21 06:23 UTC (permalink / raw)
  To: Masahiko Sawada <sawada.mshk@gmail.com>; +Cc: Bharath Rupireddy <bharath.rupireddyforpostgres@gmail.com>; Chao Li <li.evan.chao@gmail.com>; Fujii Masao <masao.fujii@gmail.com>; pgsql-hackers

On Fri, Sep 18, 2026 at 6:13 AM Masahiko Sawada <sawada.mshk@gmail.com> wrote:
>
> Thanks for reviewing the patch!
>
> While I also think the patch is in good shape, I'd like to raise a
> security risk this feature might introduce, particularly around
> pg_logical_emit_message():
>

Review comments:
===============
1. Typo (/logcal/logical): <function>pg_logcal_emit_message()</function></link>.

2.
+ * The handler always runs with the privileges of the subscription owner.
+ * The payload is chosen by whoever called pg_logical_emit_message() on the
+ * publisher, which by default any role may do, so a handler that acts on the
+ * payload gives every role on the publisher the subscription owner's
+ * privileges. A handler that needs less than that can lower them itself
+ * with SwitchToUntrustedUser().

One thing to consider about this is advice is that unlike DML there's
no target role to derive, and SwitchToUntrustedUser() errors outright
if the subscription owner can't SET ROLE to whatever the handler
picks. So, it doesn't sound straight-forward to follow this advice.

3.
Restrict who can emit messages
+   on the publisher. Revoke the default privilege and grant it to a
role created
+   for this purpose:
+<programlisting>
+REVOKE EXECUTE ON FUNCTION pg_logical_emit_message(boolean, text,
text, boolean) FROM PUBLIC;
+REVOKE EXECUTE ON FUNCTION pg_logical_emit_message(boolean, text,
bytea, boolean) FROM PUBLIC;
+GRANT EXECUTE ON FUNCTION pg_logical_emit_message(boolean, text,
text, boolean) TO message_emitter;
+GRANT EXECUTE ON FUNCTION pg_logical_emit_message(boolean, text,
bytea, boolean) TO message_emitter;
+</programlisting>

That above security recipe sounds reasonable, but it asks the
subscriber's security to be enforced by an action in a cluster the
subscriber admin frequently does not control. The Security section
should say something on the lines of: if you don't administer both the
subscriber and publisher, don't install a message handler that does
anything you wouldn't let every publisher role do. Also, I think the
above REVOKE has to be performed in all databases separately.

4. Based on above two points, I think we can fill the hole with something like:

CREATE SUBSCRIPTION ... WITH (messages = true, message_handler_role =
'msg_handler')

and then run the hook under SwitchToUntrustedUser(that_role). That
gives the subscriber admin a local, explicit privilege mechanism that
does not depend on the publisher's ACLs, doesn't touch WAL or the
protocol, and reuses machinery already in usercontext.c. If the option
is unset, either don't call the hook or keep today's behavior with the
documented warning.

-- 
With Regards,
Amit Kapila.





^ permalink  raw  reply  [nested|flat] 44+ messages in thread

* Re: Add a hook for handling logical decoding messages on subscribers.
@ 2026-09-21 06:54  Amit Kapila <amit.kapila16@gmail.com>
  parent: Amit Kapila <amit.kapila16@gmail.com>
  2 siblings, 0 replies; 44+ messages in thread

From: Amit Kapila @ 2026-09-21 06:54 UTC (permalink / raw)
  To: Masahiko Sawada <sawada.mshk@gmail.com>; +Cc: Bharath Rupireddy <bharath.rupireddyforpostgres@gmail.com>; Chao Li <li.evan.chao@gmail.com>; Fujii Masao <masao.fujii@gmail.com>; pgsql-hackers

On Mon, Sep 21, 2026 at 11:53 AM Amit Kapila <amit.kapila16@gmail.com> wrote:
>
> On Fri, Sep 18, 2026 at 6:13 AM Masahiko Sawada <sawada.mshk@gmail.com> wrote:
> >
> > Thanks for reviewing the patch!
> >
> > While I also think the patch is in good shape, I'd like to raise a
> > security risk this feature might introduce, particularly around
> > pg_logical_emit_message():
> >
>
> Review comments:
> ===============
>

One more comment:
+static void
+apply_handle_message(StringInfo s)
+{
+ LogicalRepMessageData msg;
+
+ /* Tablesync worker should never receive MESSAGE */
+ if (am_tablesync_worker())
+ ereport(ERROR,
+ errcode(ERRCODE_PROTOCOL_VIOLATION),
+ errmsg_internal("tablesync worker received a MESSAGE message"));

I think we should have a similar ERROR for parallel apply worker. I
don't think this message can reach there but we should protect it like
above.

-- 
With Regards,
Amit Kapila.






^ permalink  raw  reply  [nested|flat] 44+ messages in thread

* Re: Add a hook for handling logical decoding messages on subscribers.
@ 2026-09-21 16:20  Bharath Rupireddy <bharath.rupireddyforpostgres@gmail.com>
  parent: Amit Kapila <amit.kapila16@gmail.com>
  2 siblings, 1 reply; 44+ messages in thread

From: Bharath Rupireddy @ 2026-09-21 16:20 UTC (permalink / raw)
  To: Amit Kapila <amit.kapila16@gmail.com>; +Cc: Masahiko Sawada <sawada.mshk@gmail.com>; Chao Li <li.evan.chao@gmail.com>; Fujii Masao <masao.fujii@gmail.com>; pgsql-hackers

Hi,

On Sun, Sep 20, 2026 at 11:24 PM Amit Kapila <amit.kapila16@gmail.com> wrote:
>
> 3.
> Restrict who can emit messages
> +   on the publisher. Revoke the default privilege and grant it to a
> role created
>
> That above security recipe sounds reasonable, but it asks the
> subscriber's security to be enforced by an action in a cluster the
> subscriber admin frequently does not control. The Security section
> should say something on the lines of: if you don't administer both the
> subscriber and publisher, don't install a message handler that does
> anything you wouldn't let every publisher role do. Also, I think the
> above REVOKE has to be performed in all databases separately.
>
> 4. Based on above two points, I think we can fill the hole with something like:
>
> CREATE SUBSCRIPTION ... WITH (messages = true, message_handler_role =
> 'msg_handler')
>
> and then run the hook under SwitchToUntrustedUser(that_role). That
> gives the subscriber admin a local, explicit privilege mechanism that
> does not depend on the publisher's ACLs, doesn't touch WAL or the
> protocol, and reuses machinery already in usercontext.c. If the option
> is unset, either don't call the hook or keep today's behavior with the
> documented warning.

I like this idea more than all the other approaches. The user
installing the hook would have to make a conscious choice, and even if
the role applying the message is more privileged than the emitter on
the publisher, that is the user's choice. Also, I don't think
defaulting the message handler role to the subscription owner (which
can lead to the same security concern) or not calling the hook
silently (which can lead to silent data loss) are good defaults. So, I
think requiring the handler role to be specified by the user is a good
choice, in my opinion.

That said, not having a default role would make the ALTER SUBSCRIPTION
code for this option a bit more complex I guess (I haven't checked it
in depth though).

On Sun, Sep 20, 2026 at 11:54 PM Amit Kapila <amit.kapila16@gmail.com> wrote:
>
> One more comment:
> +static void
> +apply_handle_message(StringInfo s)
> +{
> + LogicalRepMessageData msg;
> +
> + /* Tablesync worker should never receive MESSAGE */
> + if (am_tablesync_worker())
> + ereport(ERROR,
> + errcode(ERRCODE_PROTOCOL_VIOLATION),
> + errmsg_internal("tablesync worker received a MESSAGE message"));
>
> I think we should have a similar ERROR for parallel apply worker. I
> don't think this message can reach there but we should protect it like
> above.

Also, error out for the sequence sync worker?

--
Bharath Rupireddy
Amazon Web Services: https://aws.amazon.com






^ permalink  raw  reply  [nested|flat] 44+ messages in thread

* Re: Add a hook for handling logical decoding messages on subscribers.
@ 2026-10-16 20:49  Masahiko Sawada <sawada.mshk@gmail.com>
  parent: Amit Kapila <amit.kapila16@gmail.com>
  2 siblings, 0 replies; 44+ messages in thread

From: Masahiko Sawada @ 2026-10-16 20:49 UTC (permalink / raw)
  To: Amit Kapila <amit.kapila16@gmail.com>; +Cc: Bharath Rupireddy <bharath.rupireddyforpostgres@gmail.com>; Chao Li <li.evan.chao@gmail.com>; Fujii Masao <masao.fujii@gmail.com>; pgsql-hackers

On Sun, Sep 20, 2026 at 11:24 PM Amit Kapila <amit.kapila16@gmail.com> wrote:
>
> On Fri, Sep 18, 2026 at 6:13 AM Masahiko Sawada <sawada.mshk@gmail.com> wrote:
> >
> > Thanks for reviewing the patch!
> >
> > While I also think the patch is in good shape, I'd like to raise a
> > security risk this feature might introduce, particularly around
> > pg_logical_emit_message():
> >
>
> Review comments:
> ===============
> 1. Typo (/logcal/logical): <function>pg_logcal_emit_message()</function></link>.

Fixed.

>
> 2.
> + * The handler always runs with the privileges of the subscription owner.
> + * The payload is chosen by whoever called pg_logical_emit_message() on the
> + * publisher, which by default any role may do, so a handler that acts on the
> + * payload gives every role on the publisher the subscription owner's
> + * privileges. A handler that needs less than that can lower them itself
> + * with SwitchToUntrustedUser().
>
> One thing to consider about this is advice is that unlike DML there's
> no target role to derive, and SwitchToUntrustedUser() errors outright
> if the subscription owner can't SET ROLE to whatever the handler
> picks. So, it doesn't sound straight-forward to follow this advice.

Right, but I think the error isn't specific to the hoo. When the apply
worker applies a change, it also switches to the table owner, and
errors could happen if the subscription owner cannot SET ROLE to the
table owner. I think it would be an extension's responsibility that
the apply worker switch to a role, whereas it's a extension user's
responsibility that the subscription owner can SET ROLE to that role.

>
> 3.
> Restrict who can emit messages
> +   on the publisher. Revoke the default privilege and grant it to a
> role created
> +   for this purpose:
> +<programlisting>
> +REVOKE EXECUTE ON FUNCTION pg_logical_emit_message(boolean, text,
> text, boolean) FROM PUBLIC;
> +REVOKE EXECUTE ON FUNCTION pg_logical_emit_message(boolean, text,
> bytea, boolean) FROM PUBLIC;
> +GRANT EXECUTE ON FUNCTION pg_logical_emit_message(boolean, text,
> text, boolean) TO message_emitter;
> +GRANT EXECUTE ON FUNCTION pg_logical_emit_message(boolean, text,
> bytea, boolean) TO message_emitter;
> +</programlisting>
>
> That above security recipe sounds reasonable, but it asks the
> subscriber's security to be enforced by an action in a cluster the
> subscriber admin frequently does not control. The Security section
> should say something on the lines of: if you don't administer both the
> subscriber and publisher, don't install a message handler that does
> anything you wouldn't let every publisher role do. Also, I think the
> above REVOKE has to be performed in all databases separately.

Agreed on both, I'll add that.

>
> 4. Based on above two points, I think we can fill the hole with something like:
>
> CREATE SUBSCRIPTION ... WITH (messages = true, message_handler_role =
> 'msg_handler')
>
> and then run the hook under SwitchToUntrustedUser(that_role). That
> gives the subscriber admin a local, explicit privilege mechanism that
> does not depend on the publisher's ACLs, doesn't touch WAL or the
> protocol, and reuses machinery already in usercontext.c. If the option
> is unset, either don't call the hook or keep today's behavior with the
> documented warning.

I think it works for the cases where the handler doesn't or forget to
switch the role when applying the message. But I think the handler
would have to switch the role to the target role that is suitable for
doing its job in most cases. So I think the option gives us two SET
ROLE chains instead of one. Today the subscription owner needs to be
able to SET ROLE to each target role, which is the same requirement we
already document for table owners. With message_handler_role the owner
needs SET ROLE to the handler role, and the handler role needs SET
ROLE to every target role. That's noe more role and one more place for
the DBA to get it wrong, and either mistake shows up as an apply
error.

Regards,

-- 
Masahiko Sawada
Amazon Web Services: https://aws.amazon.com






^ permalink  raw  reply  [nested|flat] 44+ messages in thread

* Re: Add a hook for handling logical decoding messages on subscribers.
@ 2026-10-16 21:44  Masahiko Sawada <sawada.mshk@gmail.com>
  parent: Bharath Rupireddy <bharath.rupireddyforpostgres@gmail.com>
  0 siblings, 1 reply; 44+ messages in thread

From: Masahiko Sawada @ 2026-10-16 21:44 UTC (permalink / raw)
  To: Bharath Rupireddy <bharath.rupireddyforpostgres@gmail.com>; +Cc: Amit Kapila <amit.kapila16@gmail.com>; Chao Li <li.evan.chao@gmail.com>; Fujii Masao <masao.fujii@gmail.com>; pgsql-hackers

On Mon, Sep 21, 2026 at 9:20 AM Bharath Rupireddy
<bharath.rupireddyforpostgres@gmail.com> wrote:
>
> Hi,
>
> On Sun, Sep 20, 2026 at 11:24 PM Amit Kapila <amit.kapila16@gmail.com> wrote:
> >
> > 3.
> > Restrict who can emit messages
> > +   on the publisher. Revoke the default privilege and grant it to a
> > role created
> >
> > That above security recipe sounds reasonable, but it asks the
> > subscriber's security to be enforced by an action in a cluster the
> > subscriber admin frequently does not control. The Security section
> > should say something on the lines of: if you don't administer both the
> > subscriber and publisher, don't install a message handler that does
> > anything you wouldn't let every publisher role do. Also, I think the
> > above REVOKE has to be performed in all databases separately.
> >
> > 4. Based on above two points, I think we can fill the hole with something like:
> >
> > CREATE SUBSCRIPTION ... WITH (messages = true, message_handler_role =
> > 'msg_handler')
> >
> > and then run the hook under SwitchToUntrustedUser(that_role). That
> > gives the subscriber admin a local, explicit privilege mechanism that
> > does not depend on the publisher's ACLs, doesn't touch WAL or the
> > protocol, and reuses machinery already in usercontext.c. If the option
> > is unset, either don't call the hook or keep today's behavior with the
> > documented warning.
>
> I like this idea more than all the other approaches. The user
> installing the hook would have to make a conscious choice, and even if
> the role applying the message is more privileged than the emitter on
> the publisher, that is the user's choice. Also, I don't think
> defaulting the message handler role to the subscription owner (which
> can lead to the same security concern) or not calling the hook
> silently (which can lead to silent data loss) are good defaults. So, I
> think requiring the handler role to be specified by the user is a good
> choice, in my opinion.
>
> That said, not having a default role would make the ALTER SUBSCRIPTION
> code for this option a bit more complex I guess (I haven't checked it
> in depth though).

 My concern is that since the handler is a C function, this doesn't
help much as a security boundary. The handler can get back to the
subscription owner by calling SetUserIdAndSecContext() with
MySubscription->owner or even switch to the superuser. That function
does no permission check, and since the security context is its second
argument, the same call clears SECURITY_RESTRICTED_OPERATION as well.
As I mentioned in the reply to Amit, the option would help in cases
where the handler doesn't switch to the target role, or forgets to,
and keeps working as the subscription owner unexpectedly. But I'm not
sure we need this option just for that case.

>
> On Sun, Sep 20, 2026 at 11:54 PM Amit Kapila <amit.kapila16@gmail.com> wrote:
> >
> > One more comment:
> > +static void
> > +apply_handle_message(StringInfo s)
> > +{
> > + LogicalRepMessageData msg;
> > +
> > + /* Tablesync worker should never receive MESSAGE */
> > + if (am_tablesync_worker())
> > + ereport(ERROR,
> > + errcode(ERRCODE_PROTOCOL_VIOLATION),
> > + errmsg_internal("tablesync worker received a MESSAGE message"));
> >
> > I think we should have a similar ERROR for parallel apply worker. I
> > don't think this message can reach there but we should protect it like
> > above.
>
> Also, error out for the sequence sync worker?

I agree to add a check for parallel apply workers, but I don't think
we need it for the sequence sync workers as they neither enter the
apply loop nor receive logical replication changes in the first place.

Regards,

-- 
Masahiko Sawada
Amazon Web Services: https://aws.amazon.com






^ permalink  raw  reply  [nested|flat] 44+ messages in thread

* Re: Add a hook for handling logical decoding messages on subscribers.
@ 2026-10-16 22:16  Bharath Rupireddy <bharath.rupireddyforpostgres@gmail.com>
  parent: Masahiko Sawada <sawada.mshk@gmail.com>
  0 siblings, 1 reply; 44+ messages in thread

From: Bharath Rupireddy @ 2026-10-16 22:16 UTC (permalink / raw)
  To: Masahiko Sawada <sawada.mshk@gmail.com>; +Cc: Amit Kapila <amit.kapila16@gmail.com>; Chao Li <li.evan.chao@gmail.com>; Fujii Masao <masao.fujii@gmail.com>; pgsql-hackers

Hi,

On Fri, Oct 16, 2026 at 2:45 PM Masahiko Sawada <sawada.mshk@gmail.com> wrote:
>
> > > 4. Based on above two points, I think we can fill the hole with something like:
>
> > > CREATE SUBSCRIPTION ... WITH (messages = true, message_handler_role =
> > > 'msg_handler')
>
> > > and then run the hook under SwitchToUntrustedUser(that_role). That
>
> > I like this idea more than all the other approaches.
>
>  My concern is that since the handler is a C function, this doesn't
> help much as a security boundary. The handler can get back to the
> subscription owner by calling SetUserIdAndSecContext() with
> MySubscription->owner or even switch to the superuser. That function
> does no permission check, and since the security context is its second
> argument, the same call clears SECURITY_RESTRICTED_OPERATION as well.
> As I mentioned in the reply to Amit, the option would help in cases
> where the handler doesn't switch to the target role, or forgets to,
> and keeps working as the subscription owner unexpectedly. But I'm not
> sure we need this option just for that case.

Agreed. In a way, we already trust every hook as far as switching to
whatever role with SetUserIdAndSecContext() is concerned. So
documenting the risk clearly here works for me.

> > > I think we should have a similar ERROR for parallel apply worker.
>
> > Also, error out for the sequence sync worker?
>
> I agree to add a check for parallel apply workers, but I don't think
> we need it for the sequence sync workers as they neither enter the
> apply loop nor receive logical replication changes in the first place.

Right, we don't need to error out for seq sync workers. However, do we
need to error out for the table sync worker, since we don't expect a
logical message there?

+ /*
+ * Logical messages are relation-agnostic, so they don't map cleanly onto
+ * the tablesync worker of a particular relation, and there would be no
+ * well-defined ordering between a message and the initial copy. Leave
+ * them to the (parallel) apply workers.
+ */

--
Bharath Rupireddy
Amazon Web Services: https://aws.amazon.com






^ permalink  raw  reply  [nested|flat] 44+ messages in thread

* Re: Add a hook for handling logical decoding messages on subscribers.
@ 2026-10-30 22:34  Masahiko Sawada <sawada.mshk@gmail.com>
  parent: Bharath Rupireddy <bharath.rupireddyforpostgres@gmail.com>
  0 siblings, 2 replies; 44+ messages in thread

From: Masahiko Sawada @ 2026-10-30 22:34 UTC (permalink / raw)
  To: Bharath Rupireddy <bharath.rupireddyforpostgres@gmail.com>; +Cc: Amit Kapila <amit.kapila16@gmail.com>; Chao Li <li.evan.chao@gmail.com>; Fujii Masao <masao.fujii@gmail.com>; pgsql-hackers

On Fri, Oct 16, 2026 at 3:16 PM Bharath Rupireddy
<bharath.rupireddyforpostgres@gmail.com> wrote:
>
> Hi,
>
> On Fri, Oct 16, 2026 at 2:45 PM Masahiko Sawada <sawada.mshk@gmail.com> wrote:
> >
> > > > 4. Based on above two points, I think we can fill the hole with something like:
> >
> > > > CREATE SUBSCRIPTION ... WITH (messages = true, message_handler_role =
> > > > 'msg_handler')
> >
> > > > and then run the hook under SwitchToUntrustedUser(that_role). That
> >
> > > I like this idea more than all the other approaches.
> >
> >  My concern is that since the handler is a C function, this doesn't
> > help much as a security boundary. The handler can get back to the
> > subscription owner by calling SetUserIdAndSecContext() with
> > MySubscription->owner or even switch to the superuser. That function
> > does no permission check, and since the security context is its second
> > argument, the same call clears SECURITY_RESTRICTED_OPERATION as well.
> > As I mentioned in the reply to Amit, the option would help in cases
> > where the handler doesn't switch to the target role, or forgets to,
> > and keeps working as the subscription owner unexpectedly. But I'm not
> > sure we need this option just for that case.
>
> Agreed. In a way, we already trust every hook as far as switching to
> whatever role with SetUserIdAndSecContext() is concerned. So
> documenting the risk clearly here works for me.
>
> > > > I think we should have a similar ERROR for parallel apply worker.
> >
> > > Also, error out for the sequence sync worker?
> >
> > I agree to add a check for parallel apply workers, but I don't think
> > we need it for the sequence sync workers as they neither enter the
> > apply loop nor receive logical replication changes in the first place.
>
> Right, we don't need to error out for seq sync workers. However, do we
> need to error out for the table sync worker, since we don't expect a
> logical message there?

It seems to me worth adding a check for tablesync workers since both
the apply workers and the tablesync workers call
walrcv_startstreaming() and it would prevent mistakes by future
changes.

I've attached the updated patch.

Regards,


--
Masahiko Sawada
Amazon Web Services: https://aws.amazon.com

Attachments:

v6-0001-Add-a-hook-for-handling-logical-messages-on-subsc.patch (text/x-patch, 115.8K) download From 596f670389d2ccfdc32a52d742b1fb824f12c935 Mon Sep 17 00:00:00 2001 From: Masahiko Sawada <sawada.mshk@gmail.com> Date: Mon, 8 Jun 2026 11:25:47 -0700 Subject: [PATCH v6] Add a hook for handling logical messages on subscribers. Previously, pgoutput could emit logical decoding messages, those written with pg_logical_emit_message(), but the built-in subscriber never requested them from the publisher, and apply_handle_message() was only a placeholder that discarded the MESSAGE message type. Acting on such messages therefore meant writing a separate logical decoding client, and giving up the apply worker's streaming, conflict handling and progress tracking. Logical messages are a general-purpose side channel carried in band with the data, useful for replicating deparsed DDL, for carrying change-data-capture context that the row data does not have, and for cache invalidation delivered in commit order relative to the rows it refers to. This commit adds a new "messages" subscription option, which asks the publisher for the pgoutput "messages" option, and dispatches each received message to a new LogicalRepMessageHandle_hook. Behavior is unchanged when the option is off or when no handler is installed. A transactional message is applied as a step of the remote transaction it belongs to, so the handler's work commits atomically with that transaction, and transaction streaming and ALTER SUBSCRIPTION ... SKIP apply to it as they do to any other change. A non-transactional message belongs to no remote transaction and is committed on its own. A message may still reach the handler more than once, for example after an apply worker restart, so handlers must be idempotent. Note also that messages are not scoped by publication: every message emitted in the publisher database is sent, subject only to origin filtering, and pg_logical_emit_message() requires no special privileges. A handler must therefore treat the payload as untrusted input, and as an arbitrary string of bytes rather than as text in the subscriber's encoding. The publisher must be running PostgreSQL 14 or later, where pgoutput gained the "messages" option; against an older publisher the option is accepted but no message is ever sent. Bump catalog version. Reviewed-by: Bharath Rupireddy <bharath.rupireddyforpostgres@gmail.com> Reviewed-by: Fujii Masao <masao.fujii@gmail.com> Reviewed-by: Amit Kapila <amit.kapila16@gmail.com> Reviewed-by: Chao Li <li.evan.chao@gmail.com> Discussion: https://postgr.es/m/CAD21AoCTNGiddikkUcDKj5QLnsg-51bpr-o6L-GTHWZL4ZFYtQ@mail.gmail.com --- doc/src/sgml/catalogs.sgml | 11 + doc/src/sgml/func/func-admin.sgml | 7 +- doc/src/sgml/logical-replication.sgml | 35 +++ doc/src/sgml/ref/alter_subscription.sgml | 5 +- doc/src/sgml/ref/create_subscription.sgml | 43 ++++ src/backend/catalog/pg_subscription.c | 1 + src/backend/catalog/system_views.sql | 5 +- src/backend/commands/subscriptioncmds.c | 25 ++- .../libpqwalreceiver/libpqwalreceiver.c | 4 + src/backend/replication/logical/proto.c | 37 ++++ src/backend/replication/logical/worker.c | 90 +++++++- src/bin/pg_dump/pg_dump.c | 16 +- src/bin/pg_dump/pg_dump.h | 1 + src/bin/pg_dump/t/002_pg_dump.pl | 4 +- src/bin/psql/describe.c | 7 +- src/include/catalog/pg_subscription.h | 6 + src/include/replication/logicalproto.h | 13 ++ src/include/replication/logicalworker.h | 49 +++++ src/include/replication/walreceiver.h | 1 + src/test/modules/Makefile | 1 + src/test/modules/meson.build | 1 + .../modules/test_logicalmsg_hooks/.gitignore | 4 + .../modules/test_logicalmsg_hooks/Makefile | 20 ++ .../modules/test_logicalmsg_hooks/meson.build | 28 +++ .../test_logicalmsg_hooks/t/001_basic.pl | 207 ++++++++++++++++++ .../test_logicalmsg_hooks.c | 140 ++++++++++++ src/test/regress/expected/subscription.out | 202 +++++++++-------- src/test/regress/sql/subscription.sql | 17 ++ src/tools/pgindent/typedefs.list | 2 + 29 files changed, 875 insertions(+), 107 deletions(-) create mode 100644 src/test/modules/test_logicalmsg_hooks/.gitignore create mode 100644 src/test/modules/test_logicalmsg_hooks/Makefile create mode 100644 src/test/modules/test_logicalmsg_hooks/meson.build create mode 100644 src/test/modules/test_logicalmsg_hooks/t/001_basic.pl create mode 100644 src/test/modules/test_logicalmsg_hooks/test_logicalmsg_hooks.c diff --git a/doc/src/sgml/catalogs.sgml b/doc/src/sgml/catalogs.sgml index 78cce7e370d..6180b60eeb7 100644 --- a/doc/src/sgml/catalogs.sgml +++ b/doc/src/sgml/catalogs.sgml @@ -8231,6 +8231,17 @@ SCRAM-SHA-256$<replaceable>&lt;iteration count&gt;</replaceable>:<replaceable>&l </para></entry> </row> + <row> + <entry role="catalog_table_entry"><para role="column_definition"> + <structfield>submessages</structfield> <type>bool</type> + </para> + <para> + If true, the subscription requests the publisher to send logical + decoding messages, which are passed to a handler registered by an + extension on the subscriber. + </para></entry> + </row> + <row> <entry role="catalog_table_entry"><para role="column_definition"> <structfield>subconninfo</structfield> <type>text</type> diff --git a/doc/src/sgml/func/func-admin.sgml b/doc/src/sgml/func/func-admin.sgml index 64b0e7bb972..d64a8cca536 100644 --- a/doc/src/sgml/func/func-admin.sgml +++ b/doc/src/sgml/func/func-admin.sgml @@ -1479,7 +1479,12 @@ postgres=# SELECT '0/0'::pg_lsn + pd.segment_number * ps.setting::int + :offset <literal>false</literal>) controls if the message is immediately flushed to WAL or not. <parameter>flush</parameter> has no effect with <parameter>transactional</parameter>, as the message's WAL - record is flushed along with its transaction. + record is flushed along with its transaction. In addition to logical + decoding clients, messages emitted by this function are delivered to + subscribers whose subscriptions have + <link linkend="sql-createsubscription-params-with-messages"><literal>messages</literal></link> + enabled, where they are passed to a handler registered by an extension + on the subscriber. </para></entry> </row> diff --git a/doc/src/sgml/logical-replication.sgml b/doc/src/sgml/logical-replication.sgml index 4701a3d9d18..849724efc43 100644 --- a/doc/src/sgml/logical-replication.sgml +++ b/doc/src/sgml/logical-replication.sgml @@ -2661,6 +2661,41 @@ CONTEXT: processing remote data for replication origin "pg_16395" during "INSER not need privileges to <literal>SET ROLE</literal> to the sequence owner. </para> + <para> + Logical decoding messages are an exception to the user switching described + above. If the subscription has <link linkend="sql-createsubscription-params-with-messages"><literal>messages</literal></link> + enabled, the handler registered for them runs with the privileges of the + subscription owner. Messages are not restricted to the tables or schemas + of the subscribed publications either, and by default any role may call + <link linkend="pg-logical-emit-message"><function>pg_logical_emit_message</function></link>. + Any role that can connect to the publisher database can therefore choose + what the handler is given, and a handler that acts on what it receives does + so with the subscription owner's privileges. Restrict who can emit messages + on the publisher. Revoke the default privilege and grant it to a role created + for this purpose: +<programlisting> +REVOKE EXECUTE ON FUNCTION pg_logical_emit_message(boolean, text, text, boolean) FROM PUBLIC; +REVOKE EXECUTE ON FUNCTION pg_logical_emit_message(boolean, text, bytea, boolean) FROM PUBLIC; +GRANT EXECUTE ON FUNCTION pg_logical_emit_message(boolean, text, text, boolean) TO message_emitter; +GRANT EXECUTE ON FUNCTION pg_logical_emit_message(boolean, text, bytea, boolean) TO message_emitter; +</programlisting> + Function privileges are per database, so this has to be done in each database + that such a subscription replicates from, and in <literal>template1</literal> + if databases created later are to inherit it. No role that is not trusted to + drive the handler should be a member of the new role, since a role that can + <literal>SET ROLE</literal> to it can emit whatever it likes. Where a message + has to be emitted on behalf of an ordinary user, for example from an event + trigger that captures that user's DDL, use a <literal>SECURITY DEFINER</literal> + function owned by that role rather than granting the privilege more widely. + </para> + + <para> + This is a control on the publisher, and the subscriber has no way to verify + that it is in place. If you do not administer both the publisher and the + subscriber, do not install a message handler that does anything you would + not let every role on the publisher do. + </para> + <para> On the publisher, privileges are only checked once at the start of a replication connection and are not re-checked as each change record is read. diff --git a/doc/src/sgml/ref/alter_subscription.sgml b/doc/src/sgml/ref/alter_subscription.sgml index 545264e8a0a..72b1b0fcd83 100644 --- a/doc/src/sgml/ref/alter_subscription.sgml +++ b/doc/src/sgml/ref/alter_subscription.sgml @@ -302,8 +302,9 @@ ALTER SUBSCRIPTION <replaceable class="parameter">name</replaceable> RENAME TO < <link linkend="sql-createsubscription-params-with-two-phase"><literal>two_phase</literal></link>, <link linkend="sql-createsubscription-params-with-retain-dead-tuples"><literal>retain_dead_tuples</literal></link>, <link linkend="sql-createsubscription-params-with-max-retention-duration"><literal>max_retention_duration</literal></link>, - <link linkend="sql-createsubscription-params-with-wal-receiver-timeout"><literal>wal_receiver_timeout</literal></link>, and - <link linkend="sql-createsubscription-params-with-conflict-log-destination"><literal>conflict_log_destination</literal></link>. + <link linkend="sql-createsubscription-params-with-wal-receiver-timeout"><literal>wal_receiver_timeout</literal></link>, + <link linkend="sql-createsubscription-params-with-conflict-log-destination"><literal>conflict_log_destination</literal></link>, and + <link linkend="sql-createsubscription-params-with-messages"><literal>messages</literal></link>. Only a superuser can set <literal>password_required = false</literal>. </para> diff --git a/doc/src/sgml/ref/create_subscription.sgml b/doc/src/sgml/ref/create_subscription.sgml index 6248c228a64..1c38daaf434 100644 --- a/doc/src/sgml/ref/create_subscription.sgml +++ b/doc/src/sgml/ref/create_subscription.sgml @@ -478,6 +478,49 @@ CREATE SUBSCRIPTION <replaceable class="parameter">subscription_name</replaceabl </listitem> </varlistentry> + <varlistentry id="sql-createsubscription-params-with-messages"> + <term><literal>messages</literal> (<type>boolean</type>)</term> + <listitem> + <para> + Specifies whether the subscription requests the publisher to send + logical decoding messages, that is, those emitted with + <link linkend="pg-logical-emit-message"><function>pg_logical_emit_message</function></link>. + The default is <literal>false</literal>. + </para> + <para> + Received messages are only acted upon if an extension installed on + the subscriber has registered a handler for them; otherwise they are + received and discarded. A message may be passed to the handler more + than once, for example after an apply worker restart, so a handler + must be prepared to process the same message repeatedly. + </para> + <para> + A transactional message is applied as part of the remote transaction + that emitted it, and is therefore delivered in commit order with + respect to other transactions, not in the order in which it was written. + Note that + <link linkend="sql-createsubscription-params-with-streaming"><literal>streaming</literal></link> + and <link linkend="sql-altersubscription-params-skip"><literal>SKIP</literal></link> + apply to transactional messages just as they do to other changes. A + non-transactional message is delivered as soon as it is decoded, even + if the transaction that emitted it later aborted. + </para> + <para> + Messages are not restricted to the tables or schemas of the + subscribed publications: every message emitted in the publisher + database is sent, subject only to + <link linkend="sql-createsubscription-params-with-origin"><literal>origin</literal></link> + filtering. Message contents are supplied by whoever calls + <function>pg_logical_emit_message</function> on the publisher, which + by default any role may do, so a handler should treat them as + untrusted input. A message is an arbitrary string of bytes, and is + transmitted without character set conversion; it may therefore + contain embedded nulls, and need not be valid text in the encoding + of either database. + </para> + </listitem> + </varlistentry> + <varlistentry id="sql-createsubscription-params-with-origin"> <term><literal>origin</literal> (<type>string</type>)</term> <listitem> diff --git a/src/backend/catalog/pg_subscription.c b/src/backend/catalog/pg_subscription.c index 60c15b22194..138cdecd436 100644 --- a/src/backend/catalog/pg_subscription.c +++ b/src/backend/catalog/pg_subscription.c @@ -133,6 +133,7 @@ GetSubscription(Oid subid, bool missing_ok) sub->maxretention = subform->submaxretention; sub->retentionactive = subform->subretentionactive; sub->conflictlogrelid = subform->subconflictlogrelid; + sub->messages = subform->submessages; /* Get slotname */ datum = SysCacheGetAttr(SUBSCRIPTIONOID, diff --git a/src/backend/catalog/system_views.sql b/src/backend/catalog/system_views.sql index ddf801307eb..4baa16961d3 100644 --- a/src/backend/catalog/system_views.sql +++ b/src/backend/catalog/system_views.sql @@ -1543,8 +1543,9 @@ GRANT SELECT (oid, subdbid, subskiplsn, subname, subowner, subenabled, subbinary, substream, subtwophasestate, subdisableonerr, subpasswordrequired, subrunasowner, subfailover, subretaindeadtuples, submaxretention, subretentionactive, - subserver, subconflictlogrelid, subconflictlogdest, subslotname, - subsynccommit, subwalrcvtimeout, subpublications, suborigin) + subserver, subconflictlogrelid, subconflictlogdest, submessages, + subslotname, subsynccommit, subwalrcvtimeout, subpublications, + suborigin) ON pg_subscription TO public; CREATE VIEW pg_stat_subscription_stats AS diff --git a/src/backend/commands/subscriptioncmds.c b/src/backend/commands/subscriptioncmds.c index 36bb20a1f6e..2916026db98 100644 --- a/src/backend/commands/subscriptioncmds.c +++ b/src/backend/commands/subscriptioncmds.c @@ -84,6 +84,7 @@ #define SUBOPT_LSN 0x00020000 #define SUBOPT_ORIGIN 0x00040000 #define SUBOPT_CONFLICT_LOG_DEST 0x00080000 +#define SUBOPT_MESSAGES 0x00100000 /* check if the 'val' has 'bits' set */ #define IsSet(val, bits) (((val) & (bits)) == (bits)) @@ -110,6 +111,7 @@ typedef struct SubOpts bool runasowner; bool failover; bool retaindeadtuples; + bool messages; int32 maxretention; char *origin; ConflictLogDest conflictlogdest; @@ -208,6 +210,8 @@ parse_subscription_options(ParseState *pstate, List *stmt_options, opts->origin = pstrdup(LOGICALREP_ORIGIN_ANY); if (IsSet(supported_opts, SUBOPT_CONFLICT_LOG_DEST)) opts->conflictlogdest = CONFLICT_LOG_DEST_LOG; + if (IsSet(supported_opts, SUBOPT_MESSAGES)) + opts->messages = false; /* Parse options */ foreach(lc, stmt_options) @@ -460,6 +464,15 @@ parse_subscription_options(ParseState *pstate, List *stmt_options, opts->conflictlogdest = GetConflictLogDest(val); opts->specified_opts |= SUBOPT_CONFLICT_LOG_DEST; } + else if (IsSet(supported_opts, SUBOPT_MESSAGES) && + strcmp(defel->defname, "messages") == 0) + { + if (IsSet(opts->specified_opts, SUBOPT_MESSAGES)) + errorConflictingDefElem(defel, pstate); + + opts->specified_opts |= SUBOPT_MESSAGES; + opts->messages = defGetBoolean(defel); + } else ereport(ERROR, (errcode(ERRCODE_SYNTAX_ERROR), @@ -700,7 +713,7 @@ CreateSubscription(ParseState *pstate, CreateSubscriptionStmt *stmt, SUBOPT_RETAIN_DEAD_TUPLES | SUBOPT_MAX_RETENTION_DURATION | SUBOPT_WAL_RECEIVER_TIMEOUT | SUBOPT_ORIGIN | - SUBOPT_CONFLICT_LOG_DEST); + SUBOPT_CONFLICT_LOG_DEST | SUBOPT_MESSAGES); parse_subscription_options(pstate, stmt->options, supported_opts, &opts); /* @@ -872,6 +885,7 @@ CreateSubscription(ParseState *pstate, CreateSubscriptionStmt *stmt, values[Anum_pg_subscription_subretentionactive - 1] = BoolGetDatum(opts.retaindeadtuples); values[Anum_pg_subscription_subserver - 1] = ObjectIdGetDatum(serverid); + values[Anum_pg_subscription_submessages - 1] = BoolGetDatum(opts.messages); if (stmt->conninfo) { Assert(stmt->conninfo == conninfo && !OidIsValid(serverid)); @@ -1714,7 +1728,8 @@ AlterSubscription(ParseState *pstate, AlterSubscriptionStmt *stmt, SUBOPT_MAX_RETENTION_DURATION | SUBOPT_WAL_RECEIVER_TIMEOUT | SUBOPT_ORIGIN | - SUBOPT_CONFLICT_LOG_DEST); + SUBOPT_CONFLICT_LOG_DEST | + SUBOPT_MESSAGES); break; case ALTER_SUBSCRIPTION_ENABLED: @@ -2121,6 +2136,12 @@ AlterSubscription(ParseState *pstate, AlterSubscriptionStmt *stmt, } } + if (IsSet(opts.specified_opts, SUBOPT_MESSAGES)) + { + values[Anum_pg_subscription_submessages - 1] = BoolGetDatum(opts.messages); + replaces[Anum_pg_subscription_submessages - 1] = true; + } + update_tuple = true; break; } diff --git a/src/backend/replication/libpqwalreceiver/libpqwalreceiver.c b/src/backend/replication/libpqwalreceiver/libpqwalreceiver.c index df47ae535ce..0d94492d856 100644 --- a/src/backend/replication/libpqwalreceiver/libpqwalreceiver.c +++ b/src/backend/replication/libpqwalreceiver/libpqwalreceiver.c @@ -628,6 +628,10 @@ libpqrcv_startstreaming(WalReceiverConn *conn, appendQuotedLiteral(&cmd, options->proto.logical.origin); } + if (options->proto.logical.messages && + PQserverVersion(conn->streamConn) >= 140000) + appendStringInfo(&cmd, ", messages 'on'"); + pubnames = options->proto.logical.publication_names; pubnames_str = stringlist_to_identifierstr(pubnames); appendStringInfoString(&cmd, ", publication_names "); diff --git a/src/backend/replication/logical/proto.c b/src/backend/replication/logical/proto.c index 86ad97cd937..6461361d68d 100644 --- a/src/backend/replication/logical/proto.c +++ b/src/backend/replication/logical/proto.c @@ -660,6 +660,43 @@ logicalrep_write_message(StringInfo out, TransactionId xid, XLogRecPtr lsn, pq_sendbytes(out, message, sz); } +/* + * Read MESSAGE from stream. + */ +void +logicalrep_read_message(StringInfo in, LogicalRepMessageData *msg_data) +{ + Size len; + char *msg; + uint8 flags; + + /* read and decode flags */ + flags = pq_getmsgint(in, 1); + msg_data->transactional = (flags & MESSAGE_TRANSACTIONAL) != 0; + + msg_data->lsn = pq_getmsgint64(in); + msg_data->prefix = pstrdup(pq_getmsgstring(in)); + + /* read message length */ + len = pq_getmsgint(in, 4); + msg_data->message_size = len; + + /* and data */ + msg = palloc(len + 1); + pq_copymsgbytes(in, msg, len); + + /* + * The payload is an arbitrary string of bytes, so message_size, not + * strlen(), is what gives its length. NULL-terminate it anyway, as + * logicalrep_read_tuple() does for column values: that way it can be + * handed to code expecting a C string without reading past the end, which + * is convenient for logging and debugging. Such code will of course stop + * at the first embedded null, if any. + */ + msg[len] = '\0'; + msg_data->message = msg; +} + /* * Write relation description to the output stream. */ diff --git a/src/backend/replication/logical/worker.c b/src/backend/replication/logical/worker.c index 44d735cdc55..fc843ae8f8c 100644 --- a/src/backend/replication/logical/worker.c +++ b/src/backend/replication/logical/worker.c @@ -561,6 +561,9 @@ typedef struct ApplySubXactData static ApplySubXactData subxact_data = {0, 0, InvalidTransactionId, NULL}; +/* Hook for plugins to get control in handling logical decoding messages */ +LogicalRepMessageHandle_hook_type LogicalRepMessageHandle_hook = NULL; + static inline void subxact_filename(char *path, Oid subid, TransactionId xid); static inline void changes_filename(char *path, Oid subid, TransactionId xid); @@ -1709,6 +1712,74 @@ apply_handle_origin(StringInfo s) errmsg_internal("ORIGIN message sent out of order"))); } +/* + * Handle MESSAGE message. + * + * Invoke a hook function if set. + */ +static void +apply_handle_message(StringInfo s) +{ + LogicalRepMessageData msg; + + /* + * The publisher must not send a message that this worker did not ask for; + * see set_stream_options(). + */ + if (!MySubscription->messages || am_tablesync_worker()) + ereport(ERROR, + (errcode(ERRCODE_PROTOCOL_VIOLATION), + errmsg_internal("MESSAGE message received but messages are not enabled for this worker"))); + + if (!LogicalRepMessageHandle_hook) + return; + + if (is_skipping_changes() || + handle_streamed_transaction(LOGICAL_REP_MSG_MESSAGE, s)) + return; + + begin_replication_step(); + + logicalrep_read_message(s, &msg); + + (*LogicalRepMessageHandle_hook) (&msg); + + end_replication_step(); + + /* + * A transactional message is applied as a step of the remote transaction + * that emitted it, and is committed together with it when applying the + * commit message. A non-transactional message belongs to no remote + * transaction, so commit it here. + */ + if (!msg.transactional) + { + Assert(!in_remote_transaction); + Assert(!in_streamed_transaction); + + /* + * The message doesn't belong to any remote transaction, so there is + * no remote commit LSN nor timestamp to record. Clear the state left + * over by the previously applied transaction so that this commit + * doesn't inherit it. + */ + replorigin_xact_clear(false); + + CommitTransactionCommand(); + + pgstat_report_stat(false); + + /* + * Report the flush position only after the local commit is durable. + * Otherwise send_feedback() would report everything received so far + * as flushed, the publisher would advance confirmed_flush_lsn past + * this message, and a crash before the commit reaches disk would lose + * the handler's work without the message ever being sent again. + */ + store_flush_position(msg.lsn, XactLastCommitEnd); + } +} + /* * Initialize fileset (if not already done). * @@ -3893,12 +3964,7 @@ apply_dispatch(StringInfo s) break; case LOGICAL_REP_MSG_MESSAGE: - - /* - * Logical replication does not use generic logical messages yet. - * Although, it could be used by other applications that use this - * output plugin. - */ + apply_handle_message(s); break; case LOGICAL_REP_MSG_STREAM_START: @@ -5184,7 +5250,8 @@ maybe_reread_subscription(void) newsub->passwordrequired != MySubscription->passwordrequired || strcmp(newsub->origin, MySubscription->origin) != 0 || newsub->owner != MySubscription->owner || - !equal(newsub->publications, MySubscription->publications)) + !equal(newsub->publications, MySubscription->publications) || + newsub->messages != MySubscription->messages) { if (am_parallel_apply_worker()) ereport(LOG, @@ -5675,6 +5742,15 @@ set_stream_options(WalRcvStreamOptions *options, options->proto.logical.twophase = false; options->proto.logical.origin = pstrdup(MySubscription->origin); + + /* + * Logical messages are relation-agnostic, so they don't map cleanly onto + * the tablesync worker of a particular relation, and there would be no + * well-defined ordering between a message and the initial copy. Leave + * them to the (parallel) apply workers. + */ + options->proto.logical.messages = (MySubscription->messages && + !am_tablesync_worker()); } /* diff --git a/src/bin/pg_dump/pg_dump.c b/src/bin/pg_dump/pg_dump.c index 69171d6b859..52bc4dadfb8 100644 --- a/src/bin/pg_dump/pg_dump.c +++ b/src/bin/pg_dump/pg_dump.c @@ -5136,6 +5136,7 @@ getSubscriptions(Archive *fout) int i_subfailover; int i_subretaindeadtuples; int i_submaxretention; + int i_submessages; int i, ntups; @@ -5234,9 +5235,14 @@ getSubscriptions(Archive *fout) " '-1' AS subwalrcvtimeout,\n"); if (fout->remoteVersion >= 190000) - appendPQExpBufferStr(query, " fs.srvname AS subservername\n"); + appendPQExpBufferStr(query, " fs.srvname AS subservername,\n"); else - appendPQExpBufferStr(query, " NULL AS subservername\n"); + appendPQExpBufferStr(query, " NULL AS subservername,\n"); + + if (fout->remoteVersion >= 200000) + appendPQExpBufferStr(query, " s.submessages\n"); + else + appendPQExpBufferStr(query, " false AS submessages\n"); appendPQExpBufferStr(query, "FROM pg_subscription s\n"); @@ -5285,6 +5291,7 @@ getSubscriptions(Archive *fout) i_subpublications = PQfnumber(res, "subpublications"); i_suborigin = PQfnumber(res, "suborigin"); i_suboriginremotelsn = PQfnumber(res, "suboriginremotelsn"); + i_submessages = PQfnumber(res, "submessages"); subinfo = pg_malloc_array(SubscriptionInfo, ntups); @@ -5342,6 +5349,8 @@ getSubscriptions(Archive *fout) else subinfo[i].suboriginremotelsn = pg_strdup(PQgetvalue(res, i, i_suboriginremotelsn)); + subinfo[i].submessages = + (strcmp(PQgetvalue(res, i, i_submessages), "t") == 0); /* Decide whether we want to dump it */ selectDumpableObject(&(subinfo[i].dobj), fout); @@ -5594,6 +5603,9 @@ dumpSubscription(Archive *fout, const SubscriptionInfo *subinfo) if (subinfo->subretaindeadtuples) appendPQExpBufferStr(query, ", retain_dead_tuples = true"); + if (subinfo->submessages) + appendPQExpBufferStr(query, ", messages = true"); + if (subinfo->submaxretention) appendPQExpBuffer(query, ", max_retention_duration = %d", subinfo->submaxretention); diff --git a/src/bin/pg_dump/pg_dump.h b/src/bin/pg_dump/pg_dump.h index 4dcb6c854e8..49d17ea3850 100644 --- a/src/bin/pg_dump/pg_dump.h +++ b/src/bin/pg_dump/pg_dump.h @@ -719,6 +719,7 @@ typedef struct _SubscriptionInfo bool subrunasowner; bool subfailover; bool subretaindeadtuples; + bool submessages; int submaxretention; char *subservername; char *subconninfo; diff --git a/src/bin/pg_dump/t/002_pg_dump.pl b/src/bin/pg_dump/t/002_pg_dump.pl index c89d699e816..25215df2003 100644 --- a/src/bin/pg_dump/t/002_pg_dump.pl +++ b/src/bin/pg_dump/t/002_pg_dump.pl @@ -3350,9 +3350,9 @@ my %tests = ( create_order => 50, create_sql => 'CREATE SUBSCRIPTION sub3 CONNECTION \'dbname=doesnotexist\' PUBLICATION pub1 - WITH (connect = false, origin = any, streaming = on);', + WITH (connect = false, origin = any, streaming = on, messages = on);', regexp => qr/^ - \QCREATE SUBSCRIPTION sub3 CONNECTION 'dbname=doesnotexist' PUBLICATION pub1 WITH (connect = false, slot_name = 'sub3', streaming = on);\E + \QCREATE SUBSCRIPTION sub3 CONNECTION 'dbname=doesnotexist' PUBLICATION pub1 WITH (connect = false, slot_name = 'sub3', streaming = on, messages = true);\E /xm, like => { %full_runs, section_post_data => 1, }, unlike => { diff --git a/src/bin/psql/describe.c b/src/bin/psql/describe.c index e4a4ba47280..e6e48a9c66c 100644 --- a/src/bin/psql/describe.c +++ b/src/bin/psql/describe.c @@ -6774,7 +6774,7 @@ describeSubscriptions(const char *pattern, bool verbose) printQueryOpt myopt = pset.popt; static const bool translate_columns[] = {false, false, false, false, false, false, false, false, false, false, false, false, false, false, - false, false, false, false, false, false, false}; + false, false, false, false, false, false, false, false}; initPQExpBuffer(&buf); @@ -6852,6 +6852,11 @@ describeSubscriptions(const char *pattern, bool verbose) gettext_noop("Retention active")); } + if (pset.sversion >= 200000) + appendPQExpBuffer(&buf, + ", submessages AS \"%s\"\n", + gettext_noop("Messages")); + appendPQExpBuffer(&buf, ", subsynccommit AS \"%s\"\n" ", subconninfo AS \"%s\"\n", diff --git a/src/include/catalog/pg_subscription.h b/src/include/catalog/pg_subscription.h index b860f02d166..9b704310b8e 100644 --- a/src/include/catalog/pg_subscription.h +++ b/src/include/catalog/pg_subscription.h @@ -96,6 +96,9 @@ CATALOG(pg_subscription,6100,SubscriptionRelationId) BKI_SHARED_RELATION BKI_ROW * server */ Oid subconflictlogrelid; /* Relid of the conflict log table. */ + + bool submessages; /* True if the subscription wants to receive + * logical messages. */ #ifdef CATALOG_VARLEN /* variable-length fields start here */ /* @@ -173,6 +176,9 @@ typedef struct Subscription * exceeded max_retention_duration, when * defined */ Oid conflictlogrelid; /* conflict log table Oid */ + bool messages; /* True if the subscription wants to receive + * logical messages */ + char *conninfo; /* Connection string to the publisher */ char *slotname; /* Name of the replication slot */ char *synccommit; /* Synchronous commit setting for worker */ char *walrcvtimeout; /* wal_receiver_timeout setting for worker */ diff --git a/src/include/replication/logicalproto.h b/src/include/replication/logicalproto.h index 058a955e20c..83741a2080d 100644 --- a/src/include/replication/logicalproto.h +++ b/src/include/replication/logicalproto.h @@ -191,6 +191,18 @@ typedef struct LogicalRepStreamAbortData TimestampTz abort_time; } LogicalRepStreamAbortData; +/* + * Logical decoding message information + */ +typedef struct LogicalRepMessageData +{ + XLogRecPtr lsn; + bool transactional; + const char *prefix; + Size message_size; /* length of message in bytes */ + const char *message; /* payload; may contain embedded nulls */ +} LogicalRepMessageData; + extern void logicalrep_write_begin(StringInfo out, ReorderBufferTXN *txn); extern void logicalrep_read_begin(StringInfo in, LogicalRepBeginData *begin_data); @@ -248,6 +260,7 @@ extern List *logicalrep_read_truncate(StringInfo in, bool *cascade, bool *restart_seqs); extern void logicalrep_write_message(StringInfo out, TransactionId xid, XLogRecPtr lsn, bool transactional, const char *prefix, Size sz, const char *message); +extern void logicalrep_read_message(StringInfo in, LogicalRepMessageData *msg_data); extern void logicalrep_write_rel(StringInfo out, TransactionId xid, Relation rel, Bitmapset *columns, PublishGencolsType include_gencols_type); diff --git a/src/include/replication/logicalworker.h b/src/include/replication/logicalworker.h index 7d748a28da8..3f745201c1a 100644 --- a/src/include/replication/logicalworker.h +++ b/src/include/replication/logicalworker.h @@ -16,6 +16,55 @@ extern PGDLLIMPORT volatile sig_atomic_t ParallelApplyMessagePending; +/* + * Forward-declared so that this header, which is included by several files + * unrelated to logical replication, does not have to include logicalproto.h. + * A handler that dereferences the struct must include that header itself. + */ +struct LogicalRepMessageData; + +/* + * Hook for extensions to handle logical decoding messages (see + * pg_logical_emit_message()) received from the publisher. Called only when + * the subscription has the "messages" option enabled. + * + * The handler runs in the apply worker, inside a transaction, with an active + * snapshot pushed. For a transactional message the handler's work is part of + * the remote transaction and commits with it; a non-transactional message is + * committed as an independent unit. The same message may be delivered more + * than once, so the handler must be idempotent. + * + * The payload is an arbitrary string of bytes. The WAL record makes no + * distinction between the text and the bytea form of + * pg_logical_emit_message(), and the bytes are transmitted without character + * set conversion, so the payload may contain embedded nulls and bytes that + * are not valid in the subscriber's encoding. Its length is given by + * msg->message_size, not by strlen(); the buffer is null-terminated only for + * convenience, and that null is not part of the payload. + * + * The handler always runs with the privileges of the subscription owner. + * The payload is chosen by whoever called pg_logical_emit_message() on the + * publisher, which by default any role may do, so a handler that acts on the + * payload gives every role on the publisher the subscription owner's + * privileges. A handler that needs less than that can switch to a less + * privileged role with SwitchToUntrustedUser(). It must take that role + * from its own configuration and never from the payload, which is written + * by the very user the switch defends against, and it should check the + * role with member_can_set_role() first: SwitchToUntrustedUser() throws an + * error if the subscription owner cannot SET ROLE to it, and that error + * loops as described below. MySubscription, declared in + * worker_internal.h, is available where the choice differs per + * subscription. + * + * Note that the apply worker runs with an empty search_path. An ERROR thrown + * by the handler restarts the apply worker, or disables the subscription if + * disable_on_error is set. Because the message is delivered again after the + * restart, a handler cannot reject a message by throwing an error; that only + * loops. A message the handler does not want must simply be ignored. + */ +typedef void (*LogicalRepMessageHandle_hook_type) (const struct LogicalRepMessageData *msg); +extern PGDLLIMPORT LogicalRepMessageHandle_hook_type LogicalRepMessageHandle_hook; + extern void ApplyWorkerMain(Datum main_arg); extern void ParallelApplyWorkerMain(Datum main_arg); extern void TableSyncWorkerMain(Datum main_arg); diff --git a/src/include/replication/walreceiver.h b/src/include/replication/walreceiver.h index 5cfe90f9989..6831bd1f751 100644 --- a/src/include/replication/walreceiver.h +++ b/src/include/replication/walreceiver.h @@ -188,6 +188,7 @@ typedef struct * prepare time */ char *origin; /* Only publish data originating from the * specified origin */ + bool messages; /* Logical messages */ } logical; } proto; } WalRcvStreamOptions; diff --git a/src/test/modules/Makefile b/src/test/modules/Makefile index 6c1504ac932..a8a31f2b23e 100644 --- a/src/test/modules/Makefile +++ b/src/test/modules/Makefile @@ -37,6 +37,7 @@ SUBDIRS = \ test_integerset \ test_json_parser \ test_lfind \ + test_logicalmsg_hooks \ test_lwlock_tranches \ test_misc \ test_oat_hooks \ diff --git a/src/test/modules/meson.build b/src/test/modules/meson.build index ef3cdb8faf9..f1aaeae71ab 100644 --- a/src/test/modules/meson.build +++ b/src/test/modules/meson.build @@ -39,6 +39,7 @@ subdir('test_int128') subdir('test_integerset') subdir('test_json_parser') subdir('test_lfind') +subdir('test_logicalmsg_hooks') subdir('test_lwlock_tranches') subdir('test_misc') subdir('test_oat_hooks') diff --git a/src/test/modules/test_logicalmsg_hooks/.gitignore b/src/test/modules/test_logicalmsg_hooks/.gitignore new file mode 100644 index 00000000000..5dcb3ff9723 --- /dev/null +++ b/src/test/modules/test_logicalmsg_hooks/.gitignore @@ -0,0 +1,4 @@ +# Generated subdirectories +/log/ +/results/ +/tmp_check/ diff --git a/src/test/modules/test_logicalmsg_hooks/Makefile b/src/test/modules/test_logicalmsg_hooks/Makefile new file mode 100644 index 00000000000..af129009b8a --- /dev/null +++ b/src/test/modules/test_logicalmsg_hooks/Makefile @@ -0,0 +1,20 @@ +# src/test/modules/test_logicalmsg_hooks/Makefile + +MODULE_big = test_logicalmsg_hooks +OBJS = \ + $(WIN32RES) \ + test_logicalmsg_hooks.o +PGFILEDESC = "test_logicalmsg_hooks - test logical replication message hooks" + +TAP_TESTS = 1 + +ifdef USE_PGXS +PG_CONFIG = pg_config +PGXS := $(shell $(PG_CONFIG) --pgxs) +include $(PGXS) +else +subdir = src/test/modules/test_logicalmsg_hooks +top_builddir = ../../../.. +include $(top_builddir)/src/Makefile.global +include $(top_srcdir)/contrib/contrib-global.mk +endif diff --git a/src/test/modules/test_logicalmsg_hooks/meson.build b/src/test/modules/test_logicalmsg_hooks/meson.build new file mode 100644 index 00000000000..8084d82e47b --- /dev/null +++ b/src/test/modules/test_logicalmsg_hooks/meson.build @@ -0,0 +1,28 @@ +# Copyright (c) 2022-2026, PostgreSQL Global Development Group + +test_logicalmsg_hooks_sources = files( + 'test_logicalmsg_hooks.c', +) + +if host_system == 'windows' + test_logicalmsg_hooks_sources += rc_lib_gen.process(win32ver_rc, extra_args: [ + '--NAME', 'test_logicalmsg_hooks', + '--FILEDESC', 'test_logicalmsg_hooks - test logical message hooks',]) +endif + +test_logicalmsg_hooks = shared_module('test_logicalmsg_hooks', + test_logicalmsg_hooks_sources, + kwargs: pg_test_mod_args, +) +test_install_libs += test_logicalmsg_hooks + +tests += { + 'name': 'test_logicalmsg_hooks', + 'sd': meson.current_source_dir(), + 'bd': meson.current_build_dir(), + 'tap': { + 'tests': [ + 't/001_basic.pl', + ], + }, +} diff --git a/src/test/modules/test_logicalmsg_hooks/t/001_basic.pl b/src/test/modules/test_logicalmsg_hooks/t/001_basic.pl new file mode 100644 index 00000000000..a9ff6171ad6 --- /dev/null +++ b/src/test/modules/test_logicalmsg_hooks/t/001_basic.pl @@ -0,0 +1,207 @@ +# Copyright (c) 2026, PostgreSQL Global Development Group + +# Test LogicalRepMessageHandle_hook. The test module records every message it +# receives into public.test_logicalmsg_log on the subscriber, so most checks +# here are plain queries against that table. + +use strict; +use warnings FATAL => 'all'; + +use PostgreSQL::Test::Cluster; +use PostgreSQL::Test::Utils; +use Test::More; + +my $pub = PostgreSQL::Test::Cluster->new('publisher'); +$pub->init(allows_streaming => 'logical'); + +# Keep the memory limit low so that the transactions below are streamed. +$pub->append_conf('postgresql.conf', "logical_decoding_work_mem = 64kB\n"); +$pub->start; + +my $sub = PostgreSQL::Test::Cluster->new('subscriber'); +$sub->init(allows_streaming => 'logical'); +$sub->append_conf( + 'postgresql.conf', q{ +shared_preload_libraries = 'test_logicalmsg_hooks' +max_parallel_apply_workers_per_subscription = 2 +}); +$sub->start; + +$pub->safe_psql( + 'postgres', qq{ +CREATE TABLE test (a int); +INSERT INTO test VALUES (1); +CREATE PUBLICATION pub FOR ALL TABLES; +}); + +my $pub_connstr = $pub->connstr . ' dbname=postgres'; + +$sub->safe_psql( + 'postgres', qq{ +CREATE TABLE test (a int primary key); +CREATE SUBSCRIPTION sub CONNECTION '$pub_connstr' PUBLICATION pub WITH (messages = true, disable_on_error = true) +}); + +$sub->wait_for_subscription_sync($pub, 'sub'); + +# Number of messages recorded under the given prefix. +sub logged_count +{ + my ($prefix) = @_; + + return $sub->safe_psql('postgres', + "SELECT count(*) FROM test_logicalmsg_log WHERE prefix = '$prefix'"); +} + +# Verify that a transactional message is passed to the hook, and that the +# message data given to the hook is correct. +$pub->safe_psql('postgres', + q{SELECT pg_logical_emit_message(true, 'basic', 'transactional message')} +); +$pub->wait_for_catchup('sub'); + +is( $sub->safe_psql( + 'postgres', + q{SELECT transactional, message_size, message FROM test_logicalmsg_log + WHERE prefix = 'basic'}), + 't|21|transactional message', + "transactional message is passed to the hook"); + +# Verify that a non-transactional message is applied even if the transaction +# that emitted it rolls back. +$pub->safe_psql( + 'postgres', q{ +BEGIN; +SELECT pg_logical_emit_message(false, 'nontransactional', 'emitted then rolled back'); +ROLLBACK; +}); + +# Commit another message to flush WAL, as ROLLBACK doesn't flush. +$pub->safe_psql('postgres', + q{SELECT pg_logical_emit_message(true, 'flush', 'flush')}); +$pub->wait_for_catchup('sub'); + +is( $sub->safe_psql( + 'postgres', + q{SELECT transactional, message FROM test_logicalmsg_log + WHERE prefix = 'nontransactional'}), + 'f|emitted then rolled back', + "non-transactional message of an aborted transaction is applied"); + +# Verify that a transactional message emitted in a subtransaction that is +# rolled back is not passed to the hook. +$pub->safe_psql( + 'postgres', q{ +BEGIN; +SAVEPOINT sp; +SELECT pg_logical_emit_message(true, 'subxact_aborted', 'not delivered'); +ROLLBACK TO SAVEPOINT sp; +SELECT pg_logical_emit_message(true, 'subxact_committed', 'delivered'); +COMMIT; +}); +$pub->wait_for_catchup('sub'); + +is(logged_count('subxact_aborted'), + 0, "message of an aborted subtransaction is not applied"); +is(logged_count('subxact_committed'), + 1, "message of a committed subtransaction is applied"); + +# Verify that transactional messages are applied in streaming transactions, +# both when the leader apply worker serializes the transaction to a file and +# when it hands the transaction to a parallel apply worker. +foreach my $mode ('on', 'parallel') +{ + $sub->safe_psql('postgres', + "ALTER SUBSCRIPTION sub SET (streaming = $mode)"); + + # Reconnect so that the new streaming mode takes effect immediately. + $sub->safe_psql('postgres', 'ALTER SUBSCRIPTION sub DISABLE'); + $sub->safe_psql('postgres', 'ALTER SUBSCRIPTION sub ENABLE'); + + $pub->safe_psql('postgres', + q{SELECT pg_stat_reset_replication_slot('sub')}); + + $pub->safe_psql( + 'postgres', qq{ +BEGIN; +INSERT INTO test SELECT generate_series(1000, 6000); +SELECT pg_logical_emit_message(true, 'streaming_$mode', 'streamed message'); +COMMIT; +}); + $pub->wait_for_catchup('sub'); + + # Make sure the transaction really was streamed, so that this test cannot + # silently degrade into a non-streaming one. + $pub->poll_query_until('postgres', + q{SELECT stream_txns > 0 FROM pg_stat_replication_slots WHERE slot_name = 'sub'} + ) or die "transaction was not streamed"; + + is(logged_count("streaming_$mode"), + 1, + "message in a streamed transaction is applied (streaming = $mode)"); + + $pub->safe_psql('postgres', 'TRUNCATE test'); + $pub->wait_for_catchup('sub'); +} + +$sub->safe_psql('postgres', 'ALTER SUBSCRIPTION sub SET (streaming = off)'); + +# Verify that ALTER SUBSCRIPTION ... SKIP skips a transactional message along +# with the rest of its transaction. The message is emitted before the +# conflicting change, so the hook runs and records the message on the first +# attempt; that work must be rolled back with the failed transaction, and must +# not be redone once the transaction is skipped. +$pub->safe_psql('postgres', 'INSERT INTO test VALUES (1)'); +$pub->wait_for_catchup('sub'); + +my $log_location = -s $sub->logfile; + +$pub->safe_psql( + 'postgres', q{ +BEGIN; +SELECT pg_logical_emit_message(true, 'skipped', 'rolled back, then skipped'); +INSERT INTO test VALUES (1); +COMMIT; +}); + +# Wait until the conflict disables the subscription. +$sub->poll_query_until('postgres', + q{SELECT subenabled = FALSE FROM pg_subscription WHERE subname = 'sub'}); + +# The hook did run before the transaction failed. +ok( $sub->log_contains( + qr/LOG[^\n]+received message: [^\n]+prefix: skipped/, + $log_location), + "hook is called for a message in a transaction that later fails"); + +# ... but its work was rolled back with the transaction. +is(logged_count('skipped'), 0, + "work done by the hook is rolled back with the remote transaction"); + +# Get the finish LSN of the failed transaction. +my $contents = slurp_file($sub->logfile, $log_location); +$contents =~ + qr/conflict detected on relation "public.test".*\n.*DETAIL:.*Could not apply remote change.*\n.*Key already exists in unique index "test_pkey", modified in transaction \d+: key .*, local row .*\n.*CONTEXT:.* for replication target relation "public.test" in transaction \d+, finished at ([[:xdigit:]]+\/[[:xdigit:]]+)/m + or die "could not get error-LSN"; +my $lsn = $1; + +$log_location = -s $sub->logfile; + +# Set skip LSN and re-enable the subscription. +$sub->safe_psql('postgres', qq{ALTER SUBSCRIPTION sub SKIP (lsn = '$lsn')}); +$sub->safe_psql('postgres', 'ALTER SUBSCRIPTION sub ENABLE'); + +# Wait for the failed transaction to be skipped. +$sub->poll_query_until('postgres', + q{SELECT subskiplsn = '0/0' FROM pg_subscription WHERE subname = 'sub'}); + +ok( !$sub->log_contains( + qr/LOG[^\n]+received message: [^\n]+prefix: skipped/, + $log_location), + "hook is not called for a message in a skipped transaction"); +is(logged_count('skipped'), 0, + "message in a skipped transaction is not applied"); + +$sub->stop; +$pub->stop; +done_testing(); diff --git a/src/test/modules/test_logicalmsg_hooks/test_logicalmsg_hooks.c b/src/test/modules/test_logicalmsg_hooks/test_logicalmsg_hooks.c new file mode 100644 index 00000000000..d8d6fc7c619 --- /dev/null +++ b/src/test/modules/test_logicalmsg_hooks/test_logicalmsg_hooks.c @@ -0,0 +1,140 @@ +/*-------------------------------------------------------------------------- + * + * test_logicalmsg_hooks.c + * Code for testing LogicalRepMessageHandle_hook + * + * The handler records every logical decoding message it receives into the + * table public.test_logicalmsg_log, so that tests can assert on the contents + * of a table rather than on the server log. Recording the messages this way + * also exercises the transactional behavior of the hook: a transactional + * message is recorded as part of the remote transaction that emitted it, and + * therefore disappears together with it if that transaction is rolled back or + * skipped, while a non-transactional message is recorded independently. + * + * The handler creates that table itself on the first message, so a test does + * not have to set it up. For a transactional message the creation is part of + * the remote transaction too, and is rolled back with it, so the table does + * not exist until some message has been applied successfully; a test must not + * query it before then. + * + * The messages are also logged, which lets a test tell "the handler ran but + * its work was rolled back" apart from "the handler was never called". + * + * Portions Copyright (c) 1996-2026, PostgreSQL Global Development Group + * Portions Copyright (c) 1994, Regents of the University of California + * + * IDENTIFICATION + * src/test/modules/test_logicalmsg_hooks/test_logicalmsg_hooks.c + * + * ------------------------------------------------------------------------- + */ + +#include "postgres.h" + +#include "catalog/namespace.h" +#include "catalog/pg_type.h" +#include "executor/spi.h" +#include "replication/logicalproto.h" +#include "replication/logicalworker.h" +#include "utils/builtins.h" +#include "utils/lsyscache.h" +#include "utils/pg_lsn.h" + +PG_MODULE_MAGIC; + +#define LOG_SCHEMA "public" +#define LOG_TABLE "test_logicalmsg_log" + +static LogicalRepMessageHandle_hook_type prev_logical_message_handler = NULL; +static void test_logical_message_handler(const LogicalRepMessageData *msg); + +/* + * Module load callback + */ +void +_PG_init(void) +{ + prev_logical_message_handler = LogicalRepMessageHandle_hook; + LogicalRepMessageHandle_hook = &test_logical_message_handler; +} + +/* + * Create the log table if not exists. + */ +static void +ensure_log_table_exists(void) +{ + Oid relid; + int ret; + + relid = get_relname_relid(LOG_TABLE, get_namespace_oid(LOG_SCHEMA, true)); + if (OidIsValid(relid)) + return; + + SPI_connect(); + + ret = SPI_execute("CREATE TABLE " LOG_SCHEMA "." LOG_TABLE + "(lsn pg_lsn, transactional boolean, prefix text, message_size int, message text)", + false, 0); + if (ret != SPI_OK_UTILITY) + elog(ERROR, "could not create " LOG_SCHEMA "." LOG_TABLE); + + SPI_finish(); +} + +static void +test_logical_message_handler(const LogicalRepMessageData *msg) +{ + Oid argtypes[5] = {LSNOID, BOOLOID, TEXTOID, INT4OID, TEXTOID}; + Datum values[5]; + int ret; + + /* + * Give a handler installed before ours its turn, so that several + * extensions can act on the same message. + */ + if (prev_logical_message_handler) + (*prev_logical_message_handler) (msg); + + ereport(LOG, + (errmsg("received message: LSN %X/%08X, prefix: %s, size: %zu, transactional: %d", + LSN_FORMAT_ARGS(msg->lsn), msg->prefix, + msg->message_size, msg->transactional))); + + ensure_log_table_exists(); + + /* + * Pass the message as query parameters rather than interpolating it into + * the query text. Message contents come from the publisher and are not to + * be trusted, and the payload may contain characters that would need + * quoting. Note also that message_size, not strlen(), is authoritative + * for the payload length. + * + * The payload is recorded as text because the accompanying test only ever + * emits text messages. A handler must not assume that in general: a + * payload is an arbitrary string of bytes, which may contain embedded + * nulls and bytes that are not valid in the subscriber's encoding. See + * the comments on LogicalRepMessageHandle_hook_type. + */ + values[0] = LSNGetDatum(msg->lsn); + values[1] = BoolGetDatum(msg->transactional); + values[2] = CStringGetTextDatum(msg->prefix); + values[3] = Int32GetDatum((int32) msg->message_size); + values[4] = PointerGetDatum(cstring_to_text_with_len(msg->message, + msg->message_size)); + + SPI_connect(); + + /* + * The apply worker runs with an empty search_path, so the table name must + * be schema-qualified. + */ + ret = SPI_execute_with_args("INSERT INTO " LOG_SCHEMA "." LOG_TABLE + " (lsn, transactional, prefix, message_size, message)" + " VALUES ($1, $2, $3, $4, $5)", + 5, argtypes, values, NULL, false, 0); + if (ret != SPI_OK_INSERT) + elog(ERROR, "could not record logical message: SPI returned %d", ret); + + SPI_finish(); +} diff --git a/src/test/regress/expected/subscription.out b/src/test/regress/expected/subscription.out index 681dc66dca6..52487149a81 100644 --- a/src/test/regress/expected/subscription.out +++ b/src/test/regress/expected/subscription.out @@ -139,18 +139,18 @@ CREATE SUBSCRIPTION regress_testsub4 CONNECTION 'dbname=regress_doesnotexist' PU WARNING: subscription was created, but is not connected HINT: To initiate replication, you must manually create the replication slot, enable the subscription, and alter the subscription to refresh publications. \dRs+ regress_testsub4 - List of subscriptions - Name | Owner | Enabled | Publication | Binary | Streaming | Two-phase commit | Disable on error | Origin | Password required | Run as owner? | Failover | Server | Retain dead tuples | Max retention duration | Retention active | Synchronous commit | Conninfo | Receiver timeout | Skip LSN | Description -------------------+---------------------------+---------+-------------+--------+-----------+------------------+------------------+--------+-------------------+---------------+----------+--------+--------------------+------------------------+------------------+--------------------+-----------------------------+------------------+------------+------------- - regress_testsub4 | regress_subscription_user | f | {testpub} | f | parallel | d | f | none | t | f | f | | f | 0 | f | off | dbname=regress_doesnotexist | -1 | 0/00000000 | + List of subscriptions + Name | Owner | Enabled | Publication | Binary | Streaming | Two-phase commit | Disable on error | Origin | Password required | Run as owner? | Failover | Server | Retain dead tuples | Max retention duration | Retention active | Messages | Synchronous commit | Conninfo | Receiver timeout | Skip LSN | Description +------------------+---------------------------+---------+-------------+--------+-----------+------------------+------------------+--------+-------------------+---------------+----------+--------+--------------------+------------------------+------------------+----------+--------------------+-----------------------------+------------------+------------+------------- + regress_testsub4 | regress_subscription_user | f | {testpub} | f | parallel | d | f | none | t | f | f | | f | 0 | f | f | off | dbname=regress_doesnotexist | -1 | 0/00000000 | (1 row) ALTER SUBSCRIPTION regress_testsub4 SET (origin = any); \dRs+ regress_testsub4 - List of subscriptions - Name | Owner | Enabled | Publication | Binary | Streaming | Two-phase commit | Disable on error | Origin | Password required | Run as owner? | Failover | Server | Retain dead tuples | Max retention duration | Retention active | Synchronous commit | Conninfo | Receiver timeout | Skip LSN | Description -------------------+---------------------------+---------+-------------+--------+-----------+------------------+------------------+--------+-------------------+---------------+----------+--------+--------------------+------------------------+------------------+--------------------+-----------------------------+------------------+------------+------------- - regress_testsub4 | regress_subscription_user | f | {testpub} | f | parallel | d | f | any | t | f | f | | f | 0 | f | off | dbname=regress_doesnotexist | -1 | 0/00000000 | + List of subscriptions + Name | Owner | Enabled | Publication | Binary | Streaming | Two-phase commit | Disable on error | Origin | Password required | Run as owner? | Failover | Server | Retain dead tuples | Max retention duration | Retention active | Messages | Synchronous commit | Conninfo | Receiver timeout | Skip LSN | Description +------------------+---------------------------+---------+-------------+--------+-----------+------------------+------------------+--------+-------------------+---------------+----------+--------+--------------------+------------------------+------------------+----------+--------------------+-----------------------------+------------------+------------+------------- + regress_testsub4 | regress_subscription_user | f | {testpub} | f | parallel | d | f | any | t | f | f | | f | 0 | f | f | off | dbname=regress_doesnotexist | -1 | 0/00000000 | (1 row) DROP SUBSCRIPTION regress_testsub3; @@ -255,10 +255,10 @@ ALTER SUBSCRIPTION regress_testsub CONNECTION 'foobar'; ERROR: invalid connection string syntax: missing "=" after "foobar" in connection info string \dRs+ - List of subscriptions - Name | Owner | Enabled | Publication | Binary | Streaming | Two-phase commit | Disable on error | Origin | Password required | Run as owner? | Failover | Server | Retain dead tuples | Max retention duration | Retention active | Synchronous commit | Conninfo | Receiver timeout | Skip LSN | Description ------------------+---------------------------+---------+-------------+--------+-----------+------------------+------------------+--------+-------------------+---------------+----------+--------+--------------------+------------------------+------------------+--------------------+-----------------------------+------------------+------------+------------------- - regress_testsub | regress_subscription_user | f | {testpub} | f | parallel | d | f | any | t | f | f | | f | 0 | f | off | dbname=regress_doesnotexist | -1 | 0/00000000 | test subscription + List of subscriptions + Name | Owner | Enabled | Publication | Binary | Streaming | Two-phase commit | Disable on error | Origin | Password required | Run as owner? | Failover | Server | Retain dead tuples | Max retention duration | Retention active | Messages | Synchronous commit | Conninfo | Receiver timeout | Skip LSN | Description +-----------------+---------------------------+---------+-------------+--------+-----------+------------------+------------------+--------+-------------------+---------------+----------+--------+--------------------+------------------------+------------------+----------+--------------------+-----------------------------+------------------+------------+------------------- + regress_testsub | regress_subscription_user | f | {testpub} | f | parallel | d | f | any | t | f | f | | f | 0 | f | f | off | dbname=regress_doesnotexist | -1 | 0/00000000 | test subscription (1 row) ALTER SUBSCRIPTION regress_testsub SET PUBLICATION testpub2, testpub3 WITH (refresh = false); @@ -267,10 +267,10 @@ ALTER SUBSCRIPTION regress_testsub SET (slot_name = 'newname'); ALTER SUBSCRIPTION regress_testsub SET (password_required = false); ALTER SUBSCRIPTION regress_testsub SET (run_as_owner = true); \dRs+ - List of subscriptions - Name | Owner | Enabled | Publication | Binary | Streaming | Two-phase commit | Disable on error | Origin | Password required | Run as owner? | Failover | Server | Retain dead tuples | Max retention duration | Retention active | Synchronous commit | Conninfo | Receiver timeout | Skip LSN | Description ------------------+---------------------------+---------+---------------------+--------+-----------+------------------+------------------+--------+-------------------+---------------+----------+--------+--------------------+------------------------+------------------+--------------------+------------------------------+------------------+------------+------------------- - regress_testsub | regress_subscription_user | f | {testpub2,testpub3} | f | parallel | d | f | any | f | t | f | | f | 0 | f | off | dbname=regress_doesnotexist2 | -1 | 0/00000000 | test subscription + List of subscriptions + Name | Owner | Enabled | Publication | Binary | Streaming | Two-phase commit | Disable on error | Origin | Password required | Run as owner? | Failover | Server | Retain dead tuples | Max retention duration | Retention active | Messages | Synchronous commit | Conninfo | Receiver timeout | Skip LSN | Description +-----------------+---------------------------+---------+---------------------+--------+-----------+------------------+------------------+--------+-------------------+---------------+----------+--------+--------------------+------------------------+------------------+----------+--------------------+------------------------------+------------------+------------+------------------- + regress_testsub | regress_subscription_user | f | {testpub2,testpub3} | f | parallel | d | f | any | f | t | f | | f | 0 | f | f | off | dbname=regress_doesnotexist2 | -1 | 0/00000000 | test subscription (1 row) ALTER SUBSCRIPTION regress_testsub SET (password_required = true); @@ -286,10 +286,10 @@ ERROR: unrecognized subscription parameter: "create_slot" -- ok ALTER SUBSCRIPTION regress_testsub SKIP (lsn = '0/12345'); \dRs+ - List of subscriptions - Name | Owner | Enabled | Publication | Binary | Streaming | Two-phase commit | Disable on error | Origin | Password required | Run as owner? | Failover | Server | Retain dead tuples | Max retention duration | Retention active | Synchronous commit | Conninfo | Receiver timeout | Skip LSN | Description ------------------+---------------------------+---------+---------------------+--------+-----------+------------------+------------------+--------+-------------------+---------------+----------+--------+--------------------+------------------------+------------------+--------------------+------------------------------+------------------+------------+------------------- - regress_testsub | regress_subscription_user | f | {testpub2,testpub3} | f | parallel | d | f | any | t | f | f | | f | 0 | f | off | dbname=regress_doesnotexist2 | -1 | 0/00012345 | test subscription + List of subscriptions + Name | Owner | Enabled | Publication | Binary | Streaming | Two-phase commit | Disable on error | Origin | Password required | Run as owner? | Failover | Server | Retain dead tuples | Max retention duration | Retention active | Messages | Synchronous commit | Conninfo | Receiver timeout | Skip LSN | Description +-----------------+---------------------------+---------+---------------------+--------+-----------+------------------+------------------+--------+-------------------+---------------+----------+--------+--------------------+------------------------+------------------+----------+--------------------+------------------------------+------------------+------------+------------------- + regress_testsub | regress_subscription_user | f | {testpub2,testpub3} | f | parallel | d | f | any | t | f | f | | f | 0 | f | f | off | dbname=regress_doesnotexist2 | -1 | 0/00012345 | test subscription (1 row) -- ok - with lsn = NONE @@ -298,10 +298,10 @@ ALTER SUBSCRIPTION regress_testsub SKIP (lsn = NONE); ALTER SUBSCRIPTION regress_testsub SKIP (lsn = '0/0'); ERROR: invalid WAL location (LSN): 0/0 \dRs+ - List of subscriptions - Name | Owner | Enabled | Publication | Binary | Streaming | Two-phase commit | Disable on error | Origin | Password required | Run as owner? | Failover | Server | Retain dead tuples | Max retention duration | Retention active | Synchronous commit | Conninfo | Receiver timeout | Skip LSN | Description ------------------+---------------------------+---------+---------------------+--------+-----------+------------------+------------------+--------+-------------------+---------------+----------+--------+--------------------+------------------------+------------------+--------------------+------------------------------+------------------+------------+------------------- - regress_testsub | regress_subscription_user | f | {testpub2,testpub3} | f | parallel | d | f | any | t | f | f | | f | 0 | f | off | dbname=regress_doesnotexist2 | -1 | 0/00000000 | test subscription + List of subscriptions + Name | Owner | Enabled | Publication | Binary | Streaming | Two-phase commit | Disable on error | Origin | Password required | Run as owner? | Failover | Server | Retain dead tuples | Max retention duration | Retention active | Messages | Synchronous commit | Conninfo | Receiver timeout | Skip LSN | Description +-----------------+---------------------------+---------+---------------------+--------+-----------+------------------+------------------+--------+-------------------+---------------+----------+--------+--------------------+------------------------+------------------+----------+--------------------+------------------------------+------------------+------------+------------------- + regress_testsub | regress_subscription_user | f | {testpub2,testpub3} | f | parallel | d | f | any | t | f | f | | f | 0 | f | f | off | dbname=regress_doesnotexist2 | -1 | 0/00000000 | test subscription (1 row) BEGIN; @@ -337,10 +337,10 @@ ALTER SUBSCRIPTION regress_testsub_foo SET (wal_receiver_timeout = '80s'); ALTER SUBSCRIPTION regress_testsub_foo SET (wal_receiver_timeout = 'foobar'); ERROR: invalid value for parameter "wal_receiver_timeout": "foobar" \dRs+ - List of subscriptions - Name | Owner | Enabled | Publication | Binary | Streaming | Two-phase commit | Disable on error | Origin | Password required | Run as owner? | Failover | Server | Retain dead tuples | Max retention duration | Retention active | Synchronous commit | Conninfo | Receiver timeout | Skip LSN | Description ----------------------+---------------------------+---------+---------------------+--------+-----------+------------------+------------------+--------+-------------------+---------------+----------+--------+--------------------+------------------------+------------------+--------------------+------------------------------+------------------+------------+------------------- - regress_testsub_foo | regress_subscription_user | f | {testpub2,testpub3} | f | parallel | d | f | any | t | f | f | | f | 0 | f | local | dbname=regress_doesnotexist2 | 80s | 0/00000000 | test subscription + List of subscriptions + Name | Owner | Enabled | Publication | Binary | Streaming | Two-phase commit | Disable on error | Origin | Password required | Run as owner? | Failover | Server | Retain dead tuples | Max retention duration | Retention active | Messages | Synchronous commit | Conninfo | Receiver timeout | Skip LSN | Description +---------------------+---------------------------+---------+---------------------+--------+-----------+------------------+------------------+--------+-------------------+---------------+----------+--------+--------------------+------------------------+------------------+----------+--------------------+------------------------------+------------------+------------+------------------- + regress_testsub_foo | regress_subscription_user | f | {testpub2,testpub3} | f | parallel | d | f | any | t | f | f | | f | 0 | f | f | local | dbname=regress_doesnotexist2 | 80s | 0/00000000 | test subscription (1 row) -- rename back to keep the rest simple @@ -369,19 +369,19 @@ CREATE SUBSCRIPTION regress_testsub CONNECTION 'dbname=regress_doesnotexist' PUB WARNING: subscription was created, but is not connected HINT: To initiate replication, you must manually create the replication slot, enable the subscription, and alter the subscription to refresh publications. \dRs+ - List of subscriptions - Name | Owner | Enabled | Publication | Binary | Streaming | Two-phase commit | Disable on error | Origin | Password required | Run as owner? | Failover | Server | Retain dead tuples | Max retention duration | Retention active | Synchronous commit | Conninfo | Receiver timeout | Skip LSN | Description ------------------+---------------------------+---------+-------------+--------+-----------+------------------+------------------+--------+-------------------+---------------+----------+--------+--------------------+------------------------+------------------+--------------------+-----------------------------+------------------+------------+------------- - regress_testsub | regress_subscription_user | f | {testpub} | t | parallel | d | f | any | t | f | f | | f | 0 | f | off | dbname=regress_doesnotexist | -1 | 0/00000000 | + List of subscriptions + Name | Owner | Enabled | Publication | Binary | Streaming | Two-phase commit | Disable on error | Origin | Password required | Run as owner? | Failover | Server | Retain dead tuples | Max retention duration | Retention active | Messages | Synchronous commit | Conninfo | Receiver timeout | Skip LSN | Description +-----------------+---------------------------+---------+-------------+--------+-----------+------------------+------------------+--------+-------------------+---------------+----------+--------+--------------------+------------------------+------------------+----------+--------------------+-----------------------------+------------------+------------+------------- + regress_testsub | regress_subscription_user | f | {testpub} | t | parallel | d | f | any | t | f | f | | f | 0 | f | f | off | dbname=regress_doesnotexist | -1 | 0/00000000 | (1 row) ALTER SUBSCRIPTION regress_testsub SET (binary = false); ALTER SUBSCRIPTION regress_testsub SET (slot_name = NONE); \dRs+ - List of subscriptions - Name | Owner | Enabled | Publication | Binary | Streaming | Two-phase commit | Disable on error | Origin | Password required | Run as owner? | Failover | Server | Retain dead tuples | Max retention duration | Retention active | Synchronous commit | Conninfo | Receiver timeout | Skip LSN | Description ------------------+---------------------------+---------+-------------+--------+-----------+------------------+------------------+--------+-------------------+---------------+----------+--------+--------------------+------------------------+------------------+--------------------+-----------------------------+------------------+------------+------------- - regress_testsub | regress_subscription_user | f | {testpub} | f | parallel | d | f | any | t | f | f | | f | 0 | f | off | dbname=regress_doesnotexist | -1 | 0/00000000 | + List of subscriptions + Name | Owner | Enabled | Publication | Binary | Streaming | Two-phase commit | Disable on error | Origin | Password required | Run as owner? | Failover | Server | Retain dead tuples | Max retention duration | Retention active | Messages | Synchronous commit | Conninfo | Receiver timeout | Skip LSN | Description +-----------------+---------------------------+---------+-------------+--------+-----------+------------------+------------------+--------+-------------------+---------------+----------+--------+--------------------+------------------------+------------------+----------+--------------------+-----------------------------+------------------+------------+------------- + regress_testsub | regress_subscription_user | f | {testpub} | f | parallel | d | f | any | t | f | f | | f | 0 | f | f | off | dbname=regress_doesnotexist | -1 | 0/00000000 | (1 row) DROP SUBSCRIPTION regress_testsub; @@ -393,27 +393,27 @@ CREATE SUBSCRIPTION regress_testsub CONNECTION 'dbname=regress_doesnotexist' PUB WARNING: subscription was created, but is not connected HINT: To initiate replication, you must manually create the replication slot, enable the subscription, and alter the subscription to refresh publications. \dRs+ - List of subscriptions - Name | Owner | Enabled | Publication | Binary | Streaming | Two-phase commit | Disable on error | Origin | Password required | Run as owner? | Failover | Server | Retain dead tuples | Max retention duration | Retention active | Synchronous commit | Conninfo | Receiver timeout | Skip LSN | Description ------------------+---------------------------+---------+-------------+--------+-----------+------------------+------------------+--------+-------------------+---------------+----------+--------+--------------------+------------------------+------------------+--------------------+-----------------------------+------------------+------------+------------- - regress_testsub | regress_subscription_user | f | {testpub} | f | on | d | f | any | t | f | f | | f | 0 | f | off | dbname=regress_doesnotexist | -1 | 0/00000000 | + List of subscriptions + Name | Owner | Enabled | Publication | Binary | Streaming | Two-phase commit | Disable on error | Origin | Password required | Run as owner? | Failover | Server | Retain dead tuples | Max retention duration | Retention active | Messages | Synchronous commit | Conninfo | Receiver timeout | Skip LSN | Description +-----------------+---------------------------+---------+-------------+--------+-----------+------------------+------------------+--------+-------------------+---------------+----------+--------+--------------------+------------------------+------------------+----------+--------------------+-----------------------------+------------------+------------+------------- + regress_testsub | regress_subscription_user | f | {testpub} | f | on | d | f | any | t | f | f | | f | 0 | f | f | off | dbname=regress_doesnotexist | -1 | 0/00000000 | (1 row) ALTER SUBSCRIPTION regress_testsub SET (streaming = parallel); \dRs+ - List of subscriptions - Name | Owner | Enabled | Publication | Binary | Streaming | Two-phase commit | Disable on error | Origin | Password required | Run as owner? | Failover | Server | Retain dead tuples | Max retention duration | Retention active | Synchronous commit | Conninfo | Receiver timeout | Skip LSN | Description ------------------+---------------------------+---------+-------------+--------+-----------+------------------+------------------+--------+-------------------+---------------+----------+--------+--------------------+------------------------+------------------+--------------------+-----------------------------+------------------+------------+------------- - regress_testsub | regress_subscription_user | f | {testpub} | f | parallel | d | f | any | t | f | f | | f | 0 | f | off | dbname=regress_doesnotexist | -1 | 0/00000000 | + List of subscriptions + Name | Owner | Enabled | Publication | Binary | Streaming | Two-phase commit | Disable on error | Origin | Password required | Run as owner? | Failover | Server | Retain dead tuples | Max retention duration | Retention active | Messages | Synchronous commit | Conninfo | Receiver timeout | Skip LSN | Description +-----------------+---------------------------+---------+-------------+--------+-----------+------------------+------------------+--------+-------------------+---------------+----------+--------+--------------------+------------------------+------------------+----------+--------------------+-----------------------------+------------------+------------+------------- + regress_testsub | regress_subscription_user | f | {testpub} | f | parallel | d | f | any | t | f | f | | f | 0 | f | f | off | dbname=regress_doesnotexist | -1 | 0/00000000 | (1 row) ALTER SUBSCRIPTION regress_testsub SET (streaming = false); ALTER SUBSCRIPTION regress_testsub SET (slot_name = NONE); \dRs+ - List of subscriptions - Name | Owner | Enabled | Publication | Binary | Streaming | Two-phase commit | Disable on error | Origin | Password required | Run as owner? | Failover | Server | Retain dead tuples | Max retention duration | Retention active | Synchronous commit | Conninfo | Receiver timeout | Skip LSN | Description ------------------+---------------------------+---------+-------------+--------+-----------+------------------+------------------+--------+-------------------+---------------+----------+--------+--------------------+------------------------+------------------+--------------------+-----------------------------+------------------+------------+------------- - regress_testsub | regress_subscription_user | f | {testpub} | f | off | d | f | any | t | f | f | | f | 0 | f | off | dbname=regress_doesnotexist | -1 | 0/00000000 | + List of subscriptions + Name | Owner | Enabled | Publication | Binary | Streaming | Two-phase commit | Disable on error | Origin | Password required | Run as owner? | Failover | Server | Retain dead tuples | Max retention duration | Retention active | Messages | Synchronous commit | Conninfo | Receiver timeout | Skip LSN | Description +-----------------+---------------------------+---------+-------------+--------+-----------+------------------+------------------+--------+-------------------+---------------+----------+--------+--------------------+------------------------+------------------+----------+--------------------+-----------------------------+------------------+------------+------------- + regress_testsub | regress_subscription_user | f | {testpub} | f | off | d | f | any | t | f | f | | f | 0 | f | f | off | dbname=regress_doesnotexist | -1 | 0/00000000 | (1 row) -- fail - publication already exists @@ -428,10 +428,10 @@ ALTER SUBSCRIPTION regress_testsub ADD PUBLICATION testpub1, testpub2 WITH (refr ALTER SUBSCRIPTION regress_testsub ADD PUBLICATION testpub1, testpub2 WITH (refresh = false); ERROR: publication "testpub1" is already in subscription "regress_testsub" \dRs+ - List of subscriptions - Name | Owner | Enabled | Publication | Binary | Streaming | Two-phase commit | Disable on error | Origin | Password required | Run as owner? | Failover | Server | Retain dead tuples | Max retention duration | Retention active | Synchronous commit | Conninfo | Receiver timeout | Skip LSN | Description ------------------+---------------------------+---------+-----------------------------+--------+-----------+------------------+------------------+--------+-------------------+---------------+----------+--------+--------------------+------------------------+------------------+--------------------+-----------------------------+------------------+------------+------------- - regress_testsub | regress_subscription_user | f | {testpub,testpub1,testpub2} | f | off | d | f | any | t | f | f | | f | 0 | f | off | dbname=regress_doesnotexist | -1 | 0/00000000 | + List of subscriptions + Name | Owner | Enabled | Publication | Binary | Streaming | Two-phase commit | Disable on error | Origin | Password required | Run as owner? | Failover | Server | Retain dead tuples | Max retention duration | Retention active | Messages | Synchronous commit | Conninfo | Receiver timeout | Skip LSN | Description +-----------------+---------------------------+---------+-----------------------------+--------+-----------+------------------+------------------+--------+-------------------+---------------+----------+--------+--------------------+------------------------+------------------+----------+--------------------+-----------------------------+------------------+------------+------------- + regress_testsub | regress_subscription_user | f | {testpub,testpub1,testpub2} | f | off | d | f | any | t | f | f | | f | 0 | f | f | off | dbname=regress_doesnotexist | -1 | 0/00000000 | (1 row) -- fail - publication used more than once @@ -446,10 +446,10 @@ ERROR: publication "testpub3" is not in subscription "regress_testsub" -- ok - delete publications ALTER SUBSCRIPTION regress_testsub DROP PUBLICATION testpub1, testpub2 WITH (refresh = false); \dRs+ - List of subscriptions - Name | Owner | Enabled | Publication | Binary | Streaming | Two-phase commit | Disable on error | Origin | Password required | Run as owner? | Failover | Server | Retain dead tuples | Max retention duration | Retention active | Synchronous commit | Conninfo | Receiver timeout | Skip LSN | Description ------------------+---------------------------+---------+-------------+--------+-----------+------------------+------------------+--------+-------------------+---------------+----------+--------+--------------------+------------------------+------------------+--------------------+-----------------------------+------------------+------------+------------- - regress_testsub | regress_subscription_user | f | {testpub} | f | off | d | f | any | t | f | f | | f | 0 | f | off | dbname=regress_doesnotexist | -1 | 0/00000000 | + List of subscriptions + Name | Owner | Enabled | Publication | Binary | Streaming | Two-phase commit | Disable on error | Origin | Password required | Run as owner? | Failover | Server | Retain dead tuples | Max retention duration | Retention active | Messages | Synchronous commit | Conninfo | Receiver timeout | Skip LSN | Description +-----------------+---------------------------+---------+-------------+--------+-----------+------------------+------------------+--------+-------------------+---------------+----------+--------+--------------------+------------------------+------------------+----------+--------------------+-----------------------------+------------------+------------+------------- + regress_testsub | regress_subscription_user | f | {testpub} | f | off | d | f | any | t | f | f | | f | 0 | f | f | off | dbname=regress_doesnotexist | -1 | 0/00000000 | (1 row) DROP SUBSCRIPTION regress_testsub; @@ -485,19 +485,19 @@ CREATE SUBSCRIPTION regress_testsub CONNECTION 'dbname=regress_doesnotexist' PUB WARNING: subscription was created, but is not connected HINT: To initiate replication, you must manually create the replication slot, enable the subscription, and alter the subscription to refresh publications. \dRs+ - List of subscriptions - Name | Owner | Enabled | Publication | Binary | Streaming | Two-phase commit | Disable on error | Origin | Password required | Run as owner? | Failover | Server | Retain dead tuples | Max retention duration | Retention active | Synchronous commit | Conninfo | Receiver timeout | Skip LSN | Description ------------------+---------------------------+---------+-------------+--------+-----------+------------------+------------------+--------+-------------------+---------------+----------+--------+--------------------+------------------------+------------------+--------------------+-----------------------------+------------------+------------+------------- - regress_testsub | regress_subscription_user | f | {testpub} | f | parallel | p | f | any | t | f | f | | f | 0 | f | off | dbname=regress_doesnotexist | -1 | 0/00000000 | + List of subscriptions + Name | Owner | Enabled | Publication | Binary | Streaming | Two-phase commit | Disable on error | Origin | Password required | Run as owner? | Failover | Server | Retain dead tuples | Max retention duration | Retention active | Messages | Synchronous commit | Conninfo | Receiver timeout | Skip LSN | Description +-----------------+---------------------------+---------+-------------+--------+-----------+------------------+------------------+--------+-------------------+---------------+----------+--------+--------------------+------------------------+------------------+----------+--------------------+-----------------------------+------------------+------------+------------- + regress_testsub | regress_subscription_user | f | {testpub} | f | parallel | p | f | any | t | f | f | | f | 0 | f | f | off | dbname=regress_doesnotexist | -1 | 0/00000000 | (1 row) -- we can alter streaming when two_phase enabled ALTER SUBSCRIPTION regress_testsub SET (streaming = true); \dRs+ - List of subscriptions - Name | Owner | Enabled | Publication | Binary | Streaming | Two-phase commit | Disable on error | Origin | Password required | Run as owner? | Failover | Server | Retain dead tuples | Max retention duration | Retention active | Synchronous commit | Conninfo | Receiver timeout | Skip LSN | Description ------------------+---------------------------+---------+-------------+--------+-----------+------------------+------------------+--------+-------------------+---------------+----------+--------+--------------------+------------------------+------------------+--------------------+-----------------------------+------------------+------------+------------- - regress_testsub | regress_subscription_user | f | {testpub} | f | on | p | f | any | t | f | f | | f | 0 | f | off | dbname=regress_doesnotexist | -1 | 0/00000000 | + List of subscriptions + Name | Owner | Enabled | Publication | Binary | Streaming | Two-phase commit | Disable on error | Origin | Password required | Run as owner? | Failover | Server | Retain dead tuples | Max retention duration | Retention active | Messages | Synchronous commit | Conninfo | Receiver timeout | Skip LSN | Description +-----------------+---------------------------+---------+-------------+--------+-----------+------------------+------------------+--------+-------------------+---------------+----------+--------+--------------------+------------------------+------------------+----------+--------------------+-----------------------------+------------------+------------+------------- + regress_testsub | regress_subscription_user | f | {testpub} | f | on | p | f | any | t | f | f | | f | 0 | f | f | off | dbname=regress_doesnotexist | -1 | 0/00000000 | (1 row) ALTER SUBSCRIPTION regress_testsub SET (slot_name = NONE); @@ -507,10 +507,10 @@ CREATE SUBSCRIPTION regress_testsub CONNECTION 'dbname=regress_doesnotexist' PUB WARNING: subscription was created, but is not connected HINT: To initiate replication, you must manually create the replication slot, enable the subscription, and alter the subscription to refresh publications. \dRs+ - List of subscriptions - Name | Owner | Enabled | Publication | Binary | Streaming | Two-phase commit | Disable on error | Origin | Password required | Run as owner? | Failover | Server | Retain dead tuples | Max retention duration | Retention active | Synchronous commit | Conninfo | Receiver timeout | Skip LSN | Description ------------------+---------------------------+---------+-------------+--------+-----------+------------------+------------------+--------+-------------------+---------------+----------+--------+--------------------+------------------------+------------------+--------------------+-----------------------------+------------------+------------+------------- - regress_testsub | regress_subscription_user | f | {testpub} | f | on | p | f | any | t | f | f | | f | 0 | f | off | dbname=regress_doesnotexist | -1 | 0/00000000 | + List of subscriptions + Name | Owner | Enabled | Publication | Binary | Streaming | Two-phase commit | Disable on error | Origin | Password required | Run as owner? | Failover | Server | Retain dead tuples | Max retention duration | Retention active | Messages | Synchronous commit | Conninfo | Receiver timeout | Skip LSN | Description +-----------------+---------------------------+---------+-------------+--------+-----------+------------------+------------------+--------+-------------------+---------------+----------+--------+--------------------+------------------------+------------------+----------+--------------------+-----------------------------+------------------+------------+------------- + regress_testsub | regress_subscription_user | f | {testpub} | f | on | p | f | any | t | f | f | | f | 0 | f | f | off | dbname=regress_doesnotexist | -1 | 0/00000000 | (1 row) ALTER SUBSCRIPTION regress_testsub SET (slot_name = NONE); @@ -523,18 +523,18 @@ CREATE SUBSCRIPTION regress_testsub CONNECTION 'dbname=regress_doesnotexist' PUB WARNING: subscription was created, but is not connected HINT: To initiate replication, you must manually create the replication slot, enable the subscription, and alter the subscription to refresh publications. \dRs+ - List of subscriptions - Name | Owner | Enabled | Publication | Binary | Streaming | Two-phase commit | Disable on error | Origin | Password required | Run as owner? | Failover | Server | Retain dead tuples | Max retention duration | Retention active | Synchronous commit | Conninfo | Receiver timeout | Skip LSN | Description ------------------+---------------------------+---------+-------------+--------+-----------+------------------+------------------+--------+-------------------+---------------+----------+--------+--------------------+------------------------+------------------+--------------------+-----------------------------+------------------+------------+------------- - regress_testsub | regress_subscription_user | f | {testpub} | f | parallel | d | f | any | t | f | f | | f | 0 | f | off | dbname=regress_doesnotexist | -1 | 0/00000000 | + List of subscriptions + Name | Owner | Enabled | Publication | Binary | Streaming | Two-phase commit | Disable on error | Origin | Password required | Run as owner? | Failover | Server | Retain dead tuples | Max retention duration | Retention active | Messages | Synchronous commit | Conninfo | Receiver timeout | Skip LSN | Description +-----------------+---------------------------+---------+-------------+--------+-----------+------------------+------------------+--------+-------------------+---------------+----------+--------+--------------------+------------------------+------------------+----------+--------------------+-----------------------------+------------------+------------+------------- + regress_testsub | regress_subscription_user | f | {testpub} | f | parallel | d | f | any | t | f | f | | f | 0 | f | f | off | dbname=regress_doesnotexist | -1 | 0/00000000 | (1 row) ALTER SUBSCRIPTION regress_testsub SET (disable_on_error = true); \dRs+ - List of subscriptions - Name | Owner | Enabled | Publication | Binary | Streaming | Two-phase commit | Disable on error | Origin | Password required | Run as owner? | Failover | Server | Retain dead tuples | Max retention duration | Retention active | Synchronous commit | Conninfo | Receiver timeout | Skip LSN | Description ------------------+---------------------------+---------+-------------+--------+-----------+------------------+------------------+--------+-------------------+---------------+----------+--------+--------------------+------------------------+------------------+--------------------+-----------------------------+------------------+------------+------------- - regress_testsub | regress_subscription_user | f | {testpub} | f | parallel | d | t | any | t | f | f | | f | 0 | f | off | dbname=regress_doesnotexist | -1 | 0/00000000 | + List of subscriptions + Name | Owner | Enabled | Publication | Binary | Streaming | Two-phase commit | Disable on error | Origin | Password required | Run as owner? | Failover | Server | Retain dead tuples | Max retention duration | Retention active | Messages | Synchronous commit | Conninfo | Receiver timeout | Skip LSN | Description +-----------------+---------------------------+---------+-------------+--------+-----------+------------------+------------------+--------+-------------------+---------------+----------+--------+--------------------+------------------------+------------------+----------+--------------------+-----------------------------+------------------+------------+------------- + regress_testsub | regress_subscription_user | f | {testpub} | f | parallel | d | t | any | t | f | f | | f | 0 | f | f | off | dbname=regress_doesnotexist | -1 | 0/00000000 | (1 row) ALTER SUBSCRIPTION regress_testsub SET (slot_name = NONE); @@ -547,10 +547,10 @@ CREATE SUBSCRIPTION regress_testsub CONNECTION 'dbname=regress_doesnotexist' PUB WARNING: subscription was created, but is not connected HINT: To initiate replication, you must manually create the replication slot, enable the subscription, and alter the subscription to refresh publications. \dRs+ - List of subscriptions - Name | Owner | Enabled | Publication | Binary | Streaming | Two-phase commit | Disable on error | Origin | Password required | Run as owner? | Failover | Server | Retain dead tuples | Max retention duration | Retention active | Synchronous commit | Conninfo | Receiver timeout | Skip LSN | Description ------------------+---------------------------+---------+-------------+--------+-----------+------------------+------------------+--------+-------------------+---------------+----------+--------+--------------------+------------------------+------------------+--------------------+-----------------------------+------------------+------------+------------- - regress_testsub | regress_subscription_user | f | {testpub} | f | parallel | d | f | any | t | f | f | | f | 0 | f | off | dbname=regress_doesnotexist | -1 | 0/00000000 | + List of subscriptions + Name | Owner | Enabled | Publication | Binary | Streaming | Two-phase commit | Disable on error | Origin | Password required | Run as owner? | Failover | Server | Retain dead tuples | Max retention duration | Retention active | Messages | Synchronous commit | Conninfo | Receiver timeout | Skip LSN | Description +-----------------+---------------------------+---------+-------------+--------+-----------+------------------+------------------+--------+-------------------+---------------+----------+--------+--------------------+------------------------+------------------+----------+--------------------+-----------------------------+------------------+------------+------------- + regress_testsub | regress_subscription_user | f | {testpub} | f | parallel | d | f | any | t | f | f | | f | 0 | f | f | off | dbname=regress_doesnotexist | -1 | 0/00000000 | (1 row) ALTER SUBSCRIPTION regress_testsub SET (slot_name = NONE); @@ -567,10 +567,10 @@ NOTICE: max_retention_duration is ineffective when retain_dead_tuples is disabl WARNING: subscription was created, but is not connected HINT: To initiate replication, you must manually create the replication slot, enable the subscription, and alter the subscription to refresh publications. \dRs+ - List of subscriptions - Name | Owner | Enabled | Publication | Binary | Streaming | Two-phase commit | Disable on error | Origin | Password required | Run as owner? | Failover | Server | Retain dead tuples | Max retention duration | Retention active | Synchronous commit | Conninfo | Receiver timeout | Skip LSN | Description ------------------+---------------------------+---------+-------------+--------+-----------+------------------+------------------+--------+-------------------+---------------+----------+--------+--------------------+------------------------+------------------+--------------------+-----------------------------+------------------+------------+------------- - regress_testsub | regress_subscription_user | f | {testpub} | f | parallel | d | f | any | t | f | f | | f | 1000 | f | off | dbname=regress_doesnotexist | -1 | 0/00000000 | + List of subscriptions + Name | Owner | Enabled | Publication | Binary | Streaming | Two-phase commit | Disable on error | Origin | Password required | Run as owner? | Failover | Server | Retain dead tuples | Max retention duration | Retention active | Messages | Synchronous commit | Conninfo | Receiver timeout | Skip LSN | Description +-----------------+---------------------------+---------+-------------+--------+-----------+------------------+------------------+--------+-------------------+---------------+----------+--------+--------------------+------------------------+------------------+----------+--------------------+-----------------------------+------------------+------------+------------- + regress_testsub | regress_subscription_user | f | {testpub} | f | parallel | d | f | any | t | f | f | | f | 1000 | f | f | off | dbname=regress_doesnotexist | -1 | 0/00000000 | (1 row) -- fail - max_retention_duration must be non-negative @@ -579,12 +579,38 @@ ERROR: option "max_retention_duration" cannot be negative -- ok ALTER SUBSCRIPTION regress_testsub SET (max_retention_duration = 0); \dRs+ - List of subscriptions - Name | Owner | Enabled | Publication | Binary | Streaming | Two-phase commit | Disable on error | Origin | Password required | Run as owner? | Failover | Server | Retain dead tuples | Max retention duration | Retention active | Synchronous commit | Conninfo | Receiver timeout | Skip LSN | Description ------------------+---------------------------+---------+-------------+--------+-----------+------------------+------------------+--------+-------------------+---------------+----------+--------+--------------------+------------------------+------------------+--------------------+-----------------------------+------------------+------------+------------- - regress_testsub | regress_subscription_user | f | {testpub} | f | parallel | d | f | any | t | f | f | | f | 0 | f | off | dbname=regress_doesnotexist | -1 | 0/00000000 | + List of subscriptions + Name | Owner | Enabled | Publication | Binary | Streaming | Two-phase commit | Disable on error | Origin | Password required | Run as owner? | Failover | Server | Retain dead tuples | Max retention duration | Retention active | Messages | Synchronous commit | Conninfo | Receiver timeout | Skip LSN | Description +-----------------+---------------------------+---------+-------------+--------+-----------+------------------+------------------+--------+-------------------+---------------+----------+--------+--------------------+------------------------+------------------+----------+--------------------+-----------------------------+------------------+------------+------------- + regress_testsub | regress_subscription_user | f | {testpub} | f | parallel | d | f | any | t | f | f | | f | 0 | f | f | off | dbname=regress_doesnotexist | -1 | 0/00000000 | +(1 row) + +ALTER SUBSCRIPTION regress_testsub SET (slot_name = NONE); +DROP SUBSCRIPTION regress_testsub; +-- fail - messages must be boolean +CREATE SUBSCRIPTION regress_testsub CONNECTION 'dbname=regress_doesnotexist' PUBLICATION testpub WITH (connect = false, messages = foo); +ERROR: messages requires a Boolean value +-- ok +CREATE SUBSCRIPTION regress_testsub CONNECTION 'dbname=regress_doesnotexist' PUBLICATION testpub WITH (connect = false, messages = true); +WARNING: subscription was created, but is not connected +HINT: To initiate replication, you must manually create the replication slot, enable the subscription, and alter the subscription to refresh publications. +\dRs+ + List of subscriptions + Name | Owner | Enabled | Publication | Binary | Streaming | Two-phase commit | Disable on error | Origin | Password required | Run as owner? | Failover | Server | Retain dead tuples | Max retention duration | Retention active | Messages | Synchronous commit | Conninfo | Receiver timeout | Skip LSN | Description +-----------------+---------------------------+---------+-------------+--------+-----------+------------------+------------------+--------+-------------------+---------------+----------+--------+--------------------+------------------------+------------------+----------+--------------------+-----------------------------+------------------+------------+------------- + regress_testsub | regress_subscription_user | f | {testpub} | f | parallel | d | f | any | t | f | f | | f | 0 | f | t | off | dbname=regress_doesnotexist | -1 | 0/00000000 | +(1 row) + +-- ok +ALTER SUBSCRIPTION regress_testsub SET (messages = false); +\dRs+ + List of subscriptions + Name | Owner | Enabled | Publication | Binary | Streaming | Two-phase commit | Disable on error | Origin | Password required | Run as owner? | Failover | Server | Retain dead tuples | Max retention duration | Retention active | Messages | Synchronous commit | Conninfo | Receiver timeout | Skip LSN | Description +-----------------+---------------------------+---------+-------------+--------+-----------+------------------+------------------+--------+-------------------+---------------+----------+--------+--------------------+------------------------+------------------+----------+--------------------+-----------------------------+------------------+------------+------------- + regress_testsub | regress_subscription_user | f | {testpub} | f | parallel | d | f | any | t | f | f | | f | 0 | f | f | off | dbname=regress_doesnotexist | -1 | 0/00000000 | (1 row) +-- cleanup ALTER SUBSCRIPTION regress_testsub SET (slot_name = NONE); DROP SUBSCRIPTION regress_testsub; -- let's do some tests with pg_create_subscription rather than superuser diff --git a/src/test/regress/sql/subscription.sql b/src/test/regress/sql/subscription.sql index cfeebaf9302..c7d5476fe7c 100644 --- a/src/test/regress/sql/subscription.sql +++ b/src/test/regress/sql/subscription.sql @@ -439,6 +439,23 @@ ALTER SUBSCRIPTION regress_testsub SET (max_retention_duration = 0); ALTER SUBSCRIPTION regress_testsub SET (slot_name = NONE); DROP SUBSCRIPTION regress_testsub; +-- fail - messages must be boolean +CREATE SUBSCRIPTION regress_testsub CONNECTION 'dbname=regress_doesnotexist' PUBLICATION testpub WITH (connect = false, messages = foo); + +-- ok +CREATE SUBSCRIPTION regress_testsub CONNECTION 'dbname=regress_doesnotexist' PUBLICATION testpub WITH (connect = false, messages = true); + +\dRs+ + +-- ok +ALTER SUBSCRIPTION regress_testsub SET (messages = false); + +\dRs+ + +-- cleanup +ALTER SUBSCRIPTION regress_testsub SET (slot_name = NONE); +DROP SUBSCRIPTION regress_testsub; + -- let's do some tests with pg_create_subscription rather than superuser SET SESSION AUTHORIZATION regress_subscription_user3; diff --git a/src/tools/pgindent/typedefs.list b/src/tools/pgindent/typedefs.list index b964e5ee907..02f711920e9 100644 --- a/src/tools/pgindent/typedefs.list +++ b/src/tools/pgindent/typedefs.list @@ -1682,6 +1682,8 @@ LogicalRepBeginData LogicalRepCommitData LogicalRepCommitPreparedTxnData LogicalRepCtxStruct +LogicalRepMessageData +LogicalRepMessageHandle_hook_type LogicalRepMsgType LogicalRepPartMapEntry LogicalRepPreparedTxnData -- 2.56.0
^ permalink raw reply [nested|flat] 44+ messages in thread

* Re: Add a hook for handling logical decoding messages on subscribers.
@ 2026-10-31 12:35  Zhijie Hou <houzhijie22@gmail.com>
  parent: Masahiko Sawada <sawada.mshk@gmail.com>
  1 sibling, 1 reply; 44+ messages in thread

From: Zhijie Hou @ 2026-10-31 12:35 UTC (permalink / raw)
  To: Masahiko Sawada <sawada.mshk@gmail.com>; Bharath Rupireddy <bharath.rupireddyforpostgres@gmail.com>; +Cc: Amit Kapila <amit.kapila16@gmail.com>; Chao Li <li.evan.chao@gmail.com>; Fujii Masao <masao.fujii@gmail.com>; pgsql-hackers

Hi,

On Sat, Oct 31, 2026 at 06:34 AM Masahiko Sawada wrote:
>
> It seems to me worth adding a check for tablesync workers since both
> the apply workers and the tablesync workers call
> walrcv_startstreaming() and it would prevent mistakes by future
> changes.
> 
> I've attached the updated patch.

While reviewing how transactional logical messages interact with 
subtransaction aborts, I noticed an issue with streamed transactions. In 
non-streaming mode, it works as expected (i.e., the message is not 
applied if the subtransaction is aborted). However, in streaming mode, 
if a transactional message is emitted in a subtransaction that is later 
aborted, the STREAM ABORT message does not discard or roll back that 
logical message, so it still gets applied in both streaming=on and 
streaming=parallel modes.

I think the reason is that ReorderBufferApplyMessage() passes the 
top-level txn rather than change->txn to rb->stream_message(), so 
logical decoding always sends the top-level XID instead of the 
subtransaction XID when outputting a streamed message. Thus the apply 
worker cannot associate the message with the subtransaction and cannot 
roll it back properly.

Strictly speaking, this is an existing issue, but it becomes more 
visible once subscribers can handle logical messages via this hook. So 
it may be worth considering whether to fix it along with this patch or 
separately.

Attached 0002 adds a test diff in 001_basic.pl to show the issue when 
streaming a message in an aborted subtransaction. 0001 is a draft patch 
showing that passing the subtransaction of the message when streaming 
allows the message to be rolled back properly (we may not necessarily 
fix it in this exact style, just sharing it to show the behavior change).


-- 
Best Regards,
Zhijie Hou

Attachments: download all (2 files, .tar.gz)

nocfbot-0001-Fix-subtransaction-XID-in-streamed-logical-messag.patch (text/plain, 5.8K) download From 01d4606950996fd8d5d7e2af2b5a19a752b561c2 Mon Sep 17 00:00:00 2001 From: Zhijie Hou <houzhijie22@gmail.com> Date: Sat, 31 Oct 2026 19:55:25 +0800 Subject: [PATCH] Fix subtransaction XID in streamed logical messages. When streaming a transactional logical decoding message emitted inside a subtransaction, ReorderBufferApplyMessage() passed the top-level ReorderBufferTXN to stream_message_cb instead of the subtransaction's ReorderBufferTXN (change->txn). As a result, stream_message_cb_wrapper() and pgoutput_message() recorded the top-level XID instead of the subtransaction XID for the streamed message, unlike pgoutput_change() and pgoutput_truncate(). Any downstream consumer of streamed logical messages therefore could not associate the message with its subtransaction or roll it back upon receiving STREAM ABORT for a rolled-back savepoint. Fix this by passing change->txn to rb->stream_message() in ReorderBufferApplyMessage(). Also update pg_decode_stream_message() in test_decoding to look up output_plugin_private on the top-level transaction via rbtxn_get_toptxn(txn), matching pg_decode_stream_abort(). --- contrib/test_decoding/expected/stream.out | 28 +++++++++++++++++++ contrib/test_decoding/sql/stream.sql | 20 +++++++++++++ contrib/test_decoding/test_decoding.c | 11 ++++++-- .../replication/logical/reorderbuffer.c | 10 ++++++- 4 files changed, 66 insertions(+), 3 deletions(-) diff --git a/contrib/test_decoding/expected/stream.out b/contrib/test_decoding/expected/stream.out index 0ec5c933610..f34a07ba761 100644 --- a/contrib/test_decoding/expected/stream.out +++ b/contrib/test_decoding/expected/stream.out @@ -157,6 +157,34 @@ SELECT data FROM pg_logical_slot_get_changes('regression_slot', NULL, NULL, 'inc COMMIT (3 rows) +-- Verify that a transactional message emitted in a streamed subtransaction +-- is associated with the subtransaction's XID. +SET debug_logical_replication_streaming = immediate; +BEGIN; +SAVEPOINT sp; +SELECT 'msg_subxact' FROM pg_logical_emit_message(true, 'test', 'msg'); + ?column? +------------- + msg_subxact +(1 row) + +ROLLBACK TO SAVEPOINT sp; +COMMIT; +WITH changes AS ( + SELECT xid, data + FROM pg_logical_slot_get_changes('regression_slot', NULL, NULL, + 'include-xids', '0', 'skip-empty-xacts', '1', 'stream-changes', '1') +) +SELECT m.xid = a.xid AS subxid_matches +FROM changes m, changes a +WHERE m.data LIKE 'streaming message:%' + AND a.data = 'aborting streamed (sub)transaction'; + subxid_matches +---------------- + t +(1 row) + +RESET debug_logical_replication_streaming; DROP TABLE stream_test; SELECT pg_drop_replication_slot('regression_slot'); pg_drop_replication_slot diff --git a/contrib/test_decoding/sql/stream.sql b/contrib/test_decoding/sql/stream.sql index 5e45a8e9b64..ec68efed421 100644 --- a/contrib/test_decoding/sql/stream.sql +++ b/contrib/test_decoding/sql/stream.sql @@ -82,5 +82,25 @@ ROLLBACK; INSERT INTO stream_test VALUES ('after-abort'); SELECT data FROM pg_logical_slot_get_changes('regression_slot', NULL, NULL, 'include-xids', '0', 'skip-empty-xacts', '1'); +-- Verify that a transactional message emitted in a streamed subtransaction +-- is associated with the subtransaction's XID. +SET debug_logical_replication_streaming = immediate; +BEGIN; +SAVEPOINT sp; +SELECT 'msg_subxact' FROM pg_logical_emit_message(true, 'test', 'msg'); +ROLLBACK TO SAVEPOINT sp; +COMMIT; + +WITH changes AS ( + SELECT xid, data + FROM pg_logical_slot_get_changes('regression_slot', NULL, NULL, + 'include-xids', '0', 'skip-empty-xacts', '1', 'stream-changes', '1') +) +SELECT m.xid = a.xid AS subxid_matches +FROM changes m, changes a +WHERE m.data LIKE 'streaming message:%' + AND a.data = 'aborting streamed (sub)transaction'; +RESET debug_logical_replication_streaming; + DROP TABLE stream_test; SELECT pg_drop_replication_slot('regression_slot'); diff --git a/contrib/test_decoding/test_decoding.c b/contrib/test_decoding/test_decoding.c index 78185837aef..961dc475a12 100644 --- a/contrib/test_decoding/test_decoding.c +++ b/contrib/test_decoding/test_decoding.c @@ -961,11 +961,18 @@ pg_decode_stream_message(LogicalDecodingContext *ctx, if (transactional) { TestDecodingData *data = ctx->output_plugin_private; - TestDecodingTxnData *txndata = txn->output_plugin_private; + + /* + * stream_message_cb can be called for an individual subtransaction + * but we maintain the output_plugin_private only under the toptxn so + * if this is not the toptxn then fetch the toptxn. + */ + ReorderBufferTXN *toptxn = rbtxn_get_toptxn(txn); + TestDecodingTxnData *txndata = toptxn->output_plugin_private; if (data->skip_empty_xacts && !txndata->stream_wrote_changes) { - pg_output_stream_start(ctx, data, txn, false); + pg_output_stream_start(ctx, data, toptxn, false); } txndata->xact_wrote_changes = txndata->stream_wrote_changes = true; } diff --git a/src/backend/replication/logical/reorderbuffer.c b/src/backend/replication/logical/reorderbuffer.c index 46843985317..7d53879d18e 100644 --- a/src/backend/replication/logical/reorderbuffer.c +++ b/src/backend/replication/logical/reorderbuffer.c @@ -2104,10 +2104,18 @@ ReorderBufferApplyMessage(ReorderBuffer *rb, ReorderBufferTXN *txn, ReorderBufferChange *change, bool streaming) { if (streaming) - rb->stream_message(rb, txn, change->lsn, true, + { + /* + * Pass the change's transaction entry (which could be a + * subtransaction) so that the output plugin can associate the message + * with it and discard the message if the subtransaction is aborted + * later. + */ + rb->stream_message(rb, change->txn, change->lsn, true, change->data.msg.prefix, change->data.msg.message_size, change->data.msg.message); + } else rb->message(rb, txn, change->lsn, true, change->data.msg.prefix, -- 2.54.0 (Apple Git-157)
nocfbot-0002-Top-up-for-v6-0001-Test-streamed-subtransaction-.patch (text/plain, 1.6K) download From 4b377025db31174f8f2a43123f0b2c4205b240ed Mon Sep 17 00:00:00 2001 From: Zhijie Hou <houzhijie22@gmail.com> Date: Sat, 31 Oct 2026 16:28:45 +0800 Subject: [PATCH] Top-up for v6-0001: Test streamed subtransaction rollback with logical messages. Verify in 001_basic.pl that a transactional logical message emitted in a streamed subtransaction that later rolls back to a savepoint is not applied under both streaming = on and streaming = parallel. --- src/test/modules/test_logicalmsg_hooks/t/001_basic.pl | 7 +++++++ 1 file changed, 7 insertions(+) diff --git a/src/test/modules/test_logicalmsg_hooks/t/001_basic.pl b/src/test/modules/test_logicalmsg_hooks/t/001_basic.pl index a9ff6171ad6..59d38769c03 100644 --- a/src/test/modules/test_logicalmsg_hooks/t/001_basic.pl +++ b/src/test/modules/test_logicalmsg_hooks/t/001_basic.pl @@ -124,7 +124,10 @@ foreach my $mode ('on', 'parallel') $pub->safe_psql( 'postgres', qq{ BEGIN; +SAVEPOINT sp; +SELECT pg_logical_emit_message(true, 'streaming_subxact_aborted_$mode', 'not delivered'); INSERT INTO test SELECT generate_series(1000, 6000); +ROLLBACK TO SAVEPOINT sp; SELECT pg_logical_emit_message(true, 'streaming_$mode', 'streamed message'); COMMIT; }); @@ -136,6 +139,10 @@ COMMIT; q{SELECT stream_txns > 0 FROM pg_stat_replication_slots WHERE slot_name = 'sub'} ) or die "transaction was not streamed"; + is(logged_count("streaming_subxact_aborted_$mode"), + 0, + "message of an aborted streamed subtransaction is not applied (streaming = $mode)" + ); is(logged_count("streaming_$mode"), 1, "message in a streamed transaction is applied (streaming = $mode)"); -- 2.54.0 (Apple Git-157)
^ permalink raw reply [nested|flat] 44+ messages in thread

* Re: Add a hook for handling logical decoding messages on subscribers.
@ 2026-11-02 12:47  Amit Kapila <amit.kapila16@gmail.com>
  parent: Zhijie Hou <houzhijie22@gmail.com>
  0 siblings, 1 reply; 44+ messages in thread

From: Amit Kapila @ 2026-11-02 12:47 UTC (permalink / raw)
  To: Zhijie Hou <houzhijie22@gmail.com>; +Cc: Masahiko Sawada <sawada.mshk@gmail.com>; Bharath Rupireddy <bharath.rupireddyforpostgres@gmail.com>; Chao Li <li.evan.chao@gmail.com>; Fujii Masao <masao.fujii@gmail.com>; pgsql-hackers

On Sat, Oct 31, 2026 at 6:05 PM Zhijie Hou <houzhijie22@gmail.com> wrote:
>
> On Sat, Oct 31, 2026 at 06:34 AM Masahiko Sawada wrote:
> >
> > It seems to me worth adding a check for tablesync workers since both
> > the apply workers and the tablesync workers call
> > walrcv_startstreaming() and it would prevent mistakes by future
> > changes.
> >
> > I've attached the updated patch.
>
> While reviewing how transactional logical messages interact with
> subtransaction aborts, I noticed an issue with streamed transactions. In
> non-streaming mode, it works as expected (i.e., the message is not
> applied if the subtransaction is aborted). However, in streaming mode,
> if a transactional message is emitted in a subtransaction that is later
> aborted, the STREAM ABORT message does not discard or roll back that
> logical message, so it still gets applied in both streaming=on and
> streaming=parallel modes.
>
> I think the reason is that ReorderBufferApplyMessage() passes the
> top-level txn rather than change->txn to rb->stream_message(), so
> logical decoding always sends the top-level XID instead of the
> subtransaction XID when outputting a streamed message. Thus the apply
> worker cannot associate the message with the subtransaction and cannot
> roll it back properly.
>
> Strictly speaking, this is an existing issue, but it becomes more
> visible once subscribers can handle logical messages via this hook. So
> it may be worth considering whether to fix it along with this patch or
> separately.
>

Right, it didn't have any impact till now for native replication as we
were just ignoring these messages on subscribers. I think this could
have impacted output-plugins if they used this functionality. I think
we can fix this separately but it is better to fix it before this
patch as this patch can expose a hidden issue.

-- 
With Regards,
Amit Kapila.






^ permalink  raw  reply  [nested|flat] 44+ messages in thread

* Re: Add a hook for handling logical decoding messages on subscribers.
@ 2026-11-02 15:06  torikoshia <torikoshia@oss.nttdata.com>
  parent: Masahiko Sawada <sawada.mshk@gmail.com>
  1 sibling, 1 reply; 44+ messages in thread

From: torikoshia @ 2026-11-02 15:06 UTC (permalink / raw)
  To: Masahiko Sawada <sawada.mshk@gmail.com>; +Cc: Bharath Rupireddy <bharath.rupireddyforpostgres@gmail.com>; Amit Kapila <amit.kapila16@gmail.com>; Chao Li <li.evan.chao@gmail.com>; Fujii Masao <masao.fujii@gmail.com>; pgsql-hackers

On 2026-10-31 07:34, Masahiko Sawada wrote:
> On Fri, Oct 16, 2026 at 3:16 PM Bharath Rupireddy
> <bharath.rupireddyforpostgres@gmail.com> wrote:
>> 
>> Hi,
>> 
>> On Fri, Oct 16, 2026 at 2:45 PM Masahiko Sawada 
>> <sawada.mshk@gmail.com> wrote:
>> >
>> > > > 4. Based on above two points, I think we can fill the hole with something like:
>> >
>> > > > CREATE SUBSCRIPTION ... WITH (messages = true, message_handler_role =
>> > > > 'msg_handler')
>> >
>> > > > and then run the hook under SwitchToUntrustedUser(that_role). That
>> >
>> > > I like this idea more than all the other approaches.
>> >
>> >  My concern is that since the handler is a C function, this doesn't
>> > help much as a security boundary. The handler can get back to the
>> > subscription owner by calling SetUserIdAndSecContext() with
>> > MySubscription->owner or even switch to the superuser. That function
>> > does no permission check, and since the security context is its second
>> > argument, the same call clears SECURITY_RESTRICTED_OPERATION as well.
>> > As I mentioned in the reply to Amit, the option would help in cases
>> > where the handler doesn't switch to the target role, or forgets to,
>> > and keeps working as the subscription owner unexpectedly. But I'm not
>> > sure we need this option just for that case.
>> 
>> Agreed. In a way, we already trust every hook as far as switching to
>> whatever role with SetUserIdAndSecContext() is concerned. So
>> documenting the risk clearly here works for me.
>> 
>> > > > I think we should have a similar ERROR for parallel apply worker.
>> >
>> > > Also, error out for the sequence sync worker?
>> >
>> > I agree to add a check for parallel apply workers, but I don't think
>> > we need it for the sequence sync workers as they neither enter the
>> > apply loop nor receive logical replication changes in the first place.
>> 
>> Right, we don't need to error out for seq sync workers. However, do we
>> need to error out for the table sync worker, since we don't expect a
>> logical message there?
> 
> It seems to me worth adding a check for tablesync workers since both
> the apply workers and the tablesync workers call
> walrcv_startstreaming() and it would prevent mistakes by future
> changes.
> 
> I've attached the updated patch.

Thanks for the proposal and patch!

As a precaution, I checked whether this patch could cause data loss
similar to the issue I previously reported with Debezium [1], where
logical messages report their end LSN while ordinary changes report
their start LSN.

Based on my code review and some tests, I haven't found a similar
issue with this patch.

The issue occurred with transactional messages.
Debezium tracks processing progress at the event level, including
individual DML operations and logical messages. In PostgreSQL's
built-in logical replication, however, the handler's changes are
committed as part of the replicated transaction, and replication
progress is tracked at the transaction boundary. Therefore, the
difference between start and end LSNs does not appear to cause
the same problem.

I also wondered whether there could be a timing window in which
consecutive non-transactional logical messages might lead to data
loss, so I checked this case as well. It seems that the following
condition in logicalmsg_decode() handles this case correctly,
so I don't see a problem here either.

   else if (!message->transactional &&
          (SnapBuildCurrentState(builder) != SNAPBUILD_CONSISTENT ||
           SnapBuildXactNeedsSkip(builder, buf->origptr)))
     return;


[1] 
https://www.postgresql.org/message-id/d99c688994ab3a998afe26e61fe4f69f%40oss.nttdata.com


-- 
Thanks,

--
Atsushi Torikoshi
Seconded from NTT DATA CORPORATION to SRA OSS K.K.






^ permalink  raw  reply  [nested|flat] 44+ messages in thread

* Re: Add a hook for handling logical decoding messages on subscribers.
@ 2026-11-09 19:02  Masahiko Sawada <sawada.mshk@gmail.com>
  parent: Amit Kapila <amit.kapila16@gmail.com>
  0 siblings, 1 reply; 44+ messages in thread

From: Masahiko Sawada @ 2026-11-09 19:02 UTC (permalink / raw)
  To: Amit Kapila <amit.kapila16@gmail.com>; +Cc: Zhijie Hou <houzhijie22@gmail.com>; Bharath Rupireddy <bharath.rupireddyforpostgres@gmail.com>; Chao Li <li.evan.chao@gmail.com>; Fujii Masao <masao.fujii@gmail.com>; pgsql-hackers

On Mon, Nov 2, 2026 at 4:47 AM Amit Kapila <amit.kapila16@gmail.com> wrote:
>
> On Sat, Oct 31, 2026 at 6:05 PM Zhijie Hou <houzhijie22@gmail.com> wrote:
> >
> > On Sat, Oct 31, 2026 at 06:34 AM Masahiko Sawada wrote:
> > >
> > > It seems to me worth adding a check for tablesync workers since both
> > > the apply workers and the tablesync workers call
> > > walrcv_startstreaming() and it would prevent mistakes by future
> > > changes.
> > >
> > > I've attached the updated patch.
> >
> > While reviewing how transactional logical messages interact with
> > subtransaction aborts, I noticed an issue with streamed transactions. In
> > non-streaming mode, it works as expected (i.e., the message is not
> > applied if the subtransaction is aborted). However, in streaming mode,
> > if a transactional message is emitted in a subtransaction that is later
> > aborted, the STREAM ABORT message does not discard or roll back that
> > logical message, so it still gets applied in both streaming=on and
> > streaming=parallel modes.
> >
> > I think the reason is that ReorderBufferApplyMessage() passes the
> > top-level txn rather than change->txn to rb->stream_message(), so
> > logical decoding always sends the top-level XID instead of the
> > subtransaction XID when outputting a streamed message. Thus the apply
> > worker cannot associate the message with the subtransaction and cannot
> > roll it back properly.
> >
> > Strictly speaking, this is an existing issue, but it becomes more
> > visible once subscribers can handle logical messages via this hook. So
> > it may be worth considering whether to fix it along with this patch or
> > separately.
> >
>
> Right, it didn't have any impact till now for native replication as we
> were just ignoring these messages on subscribers. I think this could
> have impacted output-plugins if they used this functionality. I think
> we can fix this separately but it is better to fix it before this
> patch as this patch can expose a hidden issue.

Agreed that we should fix it before this patch.

The Hou's patch looks good. I prefer to apply it to only master as
external output plugin might require changes similar to what the patch
does for test_decoding.

And thank you for the 0002 patch. I'll include it in the next version patch.

Regards,

-- 
Masahiko Sawada
Amazon Web Services: https://aws.amazon.com






^ permalink  raw  reply  [nested|flat] 44+ messages in thread

* Re: Add a hook for handling logical decoding messages on subscribers.
@ 2026-11-09 23:59  Masahiko Sawada <sawada.mshk@gmail.com>
  parent: torikoshia <torikoshia@oss.nttdata.com>
  0 siblings, 0 replies; 44+ messages in thread

From: Masahiko Sawada @ 2026-11-09 23:59 UTC (permalink / raw)
  To: torikoshia <torikoshia@oss.nttdata.com>; +Cc: Bharath Rupireddy <bharath.rupireddyforpostgres@gmail.com>; Amit Kapila <amit.kapila16@gmail.com>; Chao Li <li.evan.chao@gmail.com>; Fujii Masao <masao.fujii@gmail.com>; pgsql-hackers

On Mon, Nov 2, 2026 at 7:06 AM torikoshia <torikoshia@oss.nttdata.com> wrote:
>
> On 2026-10-31 07:34, Masahiko Sawada wrote:
> > On Fri, Oct 16, 2026 at 3:16 PM Bharath Rupireddy
> > <bharath.rupireddyforpostgres@gmail.com> wrote:
> >>
> >> Hi,
> >>
> >> On Fri, Oct 16, 2026 at 2:45 PM Masahiko Sawada
> >> <sawada.mshk@gmail.com> wrote:
> >> >
> >> > > > 4. Based on above two points, I think we can fill the hole with something like:
> >> >
> >> > > > CREATE SUBSCRIPTION ... WITH (messages = true, message_handler_role =
> >> > > > 'msg_handler')
> >> >
> >> > > > and then run the hook under SwitchToUntrustedUser(that_role). That
> >> >
> >> > > I like this idea more than all the other approaches.
> >> >
> >> >  My concern is that since the handler is a C function, this doesn't
> >> > help much as a security boundary. The handler can get back to the
> >> > subscription owner by calling SetUserIdAndSecContext() with
> >> > MySubscription->owner or even switch to the superuser. That function
> >> > does no permission check, and since the security context is its second
> >> > argument, the same call clears SECURITY_RESTRICTED_OPERATION as well.
> >> > As I mentioned in the reply to Amit, the option would help in cases
> >> > where the handler doesn't switch to the target role, or forgets to,
> >> > and keeps working as the subscription owner unexpectedly. But I'm not
> >> > sure we need this option just for that case.
> >>
> >> Agreed. In a way, we already trust every hook as far as switching to
> >> whatever role with SetUserIdAndSecContext() is concerned. So
> >> documenting the risk clearly here works for me.
> >>
> >> > > > I think we should have a similar ERROR for parallel apply worker.
> >> >
> >> > > Also, error out for the sequence sync worker?
> >> >
> >> > I agree to add a check for parallel apply workers, but I don't think
> >> > we need it for the sequence sync workers as they neither enter the
> >> > apply loop nor receive logical replication changes in the first place.
> >>
> >> Right, we don't need to error out for seq sync workers. However, do we
> >> need to error out for the table sync worker, since we don't expect a
> >> logical message there?
> >
> > It seems to me worth adding a check for tablesync workers since both
> > the apply workers and the tablesync workers call
> > walrcv_startstreaming() and it would prevent mistakes by future
> > changes.
> >
> > I've attached the updated patch.
>
> Thanks for the proposal and patch!
>
> As a precaution, I checked whether this patch could cause data loss
> similar to the issue I previously reported with Debezium [1], where
> logical messages report their end LSN while ordinary changes report
> their start LSN.
>
> Based on my code review and some tests, I haven't found a similar
> issue with this patch.
>
> The issue occurred with transactional messages.
> Debezium tracks processing progress at the event level, including
> individual DML operations and logical messages. In PostgreSQL's
> built-in logical replication, however, the handler's changes are
> committed as part of the replicated transaction, and replication
> progress is tracked at the transaction boundary. Therefore, the
> difference between start and end LSNs does not appear to cause
> the same problem.
>
> I also wondered whether there could be a timing window in which
> consecutive non-transactional logical messages might lead to data
> loss, so I checked this case as well. It seems that the following
> condition in logicalmsg_decode() handles this case correctly,
> so I don't see a problem here either.
>
>    else if (!message->transactional &&
>           (SnapBuildCurrentState(builder) != SNAPBUILD_CONSISTENT ||
>            SnapBuildXactNeedsSkip(builder, buf->origptr)))
>      return;

Thank you for reviewing and testing the patch!

For a non-transactional message, the apply worker commits the
handler's work as its own transaction and uses the message LSN as the
flush position. So this patch does rely on the message reporting its
end LSN in that case. If it reported the start LSN instead, the
publisher's confirmed_flush would point at the message record itself,
and the message would be decoded and sent again after a restart even
though the subscriber already committed it and sent an ack.

So I think using origptr for transactional messages and endptr for
non-transactional ones, as we discussed in that thread, is the right
thing for this patch too. I'll reply there.

Regards,

-- 
Masahiko Sawada
Amazon Web Services: https://aws.amazon.com






^ permalink  raw  reply  [nested|flat] 44+ messages in thread

* Re: Add a hook for handling logical decoding messages on subscribers.
@ 2026-11-10 01:21  Masahiko Sawada <sawada.mshk@gmail.com>
  parent: Masahiko Sawada <sawada.mshk@gmail.com>
  0 siblings, 1 reply; 44+ messages in thread

From: Masahiko Sawada @ 2026-11-10 01:21 UTC (permalink / raw)
  To: Amit Kapila <amit.kapila16@gmail.com>; +Cc: Zhijie Hou <houzhijie22@gmail.com>; Bharath Rupireddy <bharath.rupireddyforpostgres@gmail.com>; Chao Li <li.evan.chao@gmail.com>; Fujii Masao <masao.fujii@gmail.com>; pgsql-hackers

On Mon, Nov 9, 2026 at 11:02 AM Masahiko Sawada <sawada.mshk@gmail.com> wrote:
>
> On Mon, Nov 2, 2026 at 4:47 AM Amit Kapila <amit.kapila16@gmail.com> wrote:
> >
> > On Sat, Oct 31, 2026 at 6:05 PM Zhijie Hou <houzhijie22@gmail.com> wrote:
> > >
> > > On Sat, Oct 31, 2026 at 06:34 AM Masahiko Sawada wrote:
> > > >
> > > > It seems to me worth adding a check for tablesync workers since both
> > > > the apply workers and the tablesync workers call
> > > > walrcv_startstreaming() and it would prevent mistakes by future
> > > > changes.
> > > >
> > > > I've attached the updated patch.
> > >
> > > While reviewing how transactional logical messages interact with
> > > subtransaction aborts, I noticed an issue with streamed transactions. In
> > > non-streaming mode, it works as expected (i.e., the message is not
> > > applied if the subtransaction is aborted). However, in streaming mode,
> > > if a transactional message is emitted in a subtransaction that is later
> > > aborted, the STREAM ABORT message does not discard or roll back that
> > > logical message, so it still gets applied in both streaming=on and
> > > streaming=parallel modes.
> > >
> > > I think the reason is that ReorderBufferApplyMessage() passes the
> > > top-level txn rather than change->txn to rb->stream_message(), so
> > > logical decoding always sends the top-level XID instead of the
> > > subtransaction XID when outputting a streamed message. Thus the apply
> > > worker cannot associate the message with the subtransaction and cannot
> > > roll it back properly.
> > >
> > > Strictly speaking, this is an existing issue, but it becomes more
> > > visible once subscribers can handle logical messages via this hook. So
> > > it may be worth considering whether to fix it along with this patch or
> > > separately.
> > >
> >
> > Right, it didn't have any impact till now for native replication as we
> > were just ignoring these messages on subscribers. I think this could
> > have impacted output-plugins if they used this functionality. I think
> > we can fix this separately but it is better to fix it before this
> > patch as this patch can expose a hidden issue.
>
> Agreed that we should fix it before this patch.
>
> The Hou's patch looks good. I prefer to apply it to only master as
> external output plugin might require changes similar to what the patch
> does for test_decoding.
>
> And thank you for the 0002 patch. I'll include it in the next version patch.

I've attached the new patches. I just updated the commit message for
the 0001 patch, and added some fixes and regression tests for the 0002
patch. Please review them.

Regards,

-- 
Masahiko Sawada
Amazon Web Services: https://aws.amazon.com

Attachments: download all (2 files, .tar.gz)

v7-0001-Fix-subtransaction-XID-in-streamed-logical-messag.patch (text/x-patch, 5.9K) download From 27712ed56a6ddcd6e05c36c201191061de690a6a Mon Sep 17 00:00:00 2001 From: Zhijie Hou <houzhijie22@gmail.com> Date: Sat, 31 Oct 2026 19:55:25 +0800 Subject: [PATCH v7 1/2] Fix subtransaction XID in streamed logical messages. Previously, when a transaction was streamed, a transactional logical decoding message emitted in a subtransaction was passed to the stream_message callback with the top-level transaction, whereas row changes and truncates in the same subtransaction are passed with the subtransaction. pgoutput therefore sent the message with the top-level XID, so a consumer could not tell which subtransaction it belonged to. When that subtransaction was rolled back and STREAM ABORT arrived for it, the message was not discarded with it. Fix this by passing the subtransaction, as for the other change types. Although this is an oversight in commit 7259736a6e5, we don't backpatch the fix. An output plugin written after test_decoding's example would crash the same way on a subtransaction, and no in-core consumer acts on streamed messages in the back branches, so we don't think the fix is worth that risk there. Author: Zhijie Hou <houzhijie22@gmail.com> Reviewed-by: Masahiko Sawada <sawada.mshk@gmail.com> Discussion: https://postgr.es/m/b9b5bd27-6416-48b4-8f73-c5a60e2c97e2@gmail.com --- contrib/test_decoding/expected/stream.out | 28 +++++++++++++++++++ contrib/test_decoding/sql/stream.sql | 20 +++++++++++++ contrib/test_decoding/test_decoding.c | 11 ++++++-- .../replication/logical/reorderbuffer.c | 10 ++++++- 4 files changed, 66 insertions(+), 3 deletions(-) diff --git a/contrib/test_decoding/expected/stream.out b/contrib/test_decoding/expected/stream.out index 0ec5c933610..f34a07ba761 100644 --- a/contrib/test_decoding/expected/stream.out +++ b/contrib/test_decoding/expected/stream.out @@ -157,6 +157,34 @@ SELECT data FROM pg_logical_slot_get_changes('regression_slot', NULL, NULL, 'inc COMMIT (3 rows) +-- Verify that a transactional message emitted in a streamed subtransaction +-- is associated with the subtransaction's XID. +SET debug_logical_replication_streaming = immediate; +BEGIN; +SAVEPOINT sp; +SELECT 'msg_subxact' FROM pg_logical_emit_message(true, 'test', 'msg'); + ?column? +------------- + msg_subxact +(1 row) + +ROLLBACK TO SAVEPOINT sp; +COMMIT; +WITH changes AS ( + SELECT xid, data + FROM pg_logical_slot_get_changes('regression_slot', NULL, NULL, + 'include-xids', '0', 'skip-empty-xacts', '1', 'stream-changes', '1') +) +SELECT m.xid = a.xid AS subxid_matches +FROM changes m, changes a +WHERE m.data LIKE 'streaming message:%' + AND a.data = 'aborting streamed (sub)transaction'; + subxid_matches +---------------- + t +(1 row) + +RESET debug_logical_replication_streaming; DROP TABLE stream_test; SELECT pg_drop_replication_slot('regression_slot'); pg_drop_replication_slot diff --git a/contrib/test_decoding/sql/stream.sql b/contrib/test_decoding/sql/stream.sql index 5e45a8e9b64..ec68efed421 100644 --- a/contrib/test_decoding/sql/stream.sql +++ b/contrib/test_decoding/sql/stream.sql @@ -82,5 +82,25 @@ ROLLBACK; INSERT INTO stream_test VALUES ('after-abort'); SELECT data FROM pg_logical_slot_get_changes('regression_slot', NULL, NULL, 'include-xids', '0', 'skip-empty-xacts', '1'); +-- Verify that a transactional message emitted in a streamed subtransaction +-- is associated with the subtransaction's XID. +SET debug_logical_replication_streaming = immediate; +BEGIN; +SAVEPOINT sp; +SELECT 'msg_subxact' FROM pg_logical_emit_message(true, 'test', 'msg'); +ROLLBACK TO SAVEPOINT sp; +COMMIT; + +WITH changes AS ( + SELECT xid, data + FROM pg_logical_slot_get_changes('regression_slot', NULL, NULL, + 'include-xids', '0', 'skip-empty-xacts', '1', 'stream-changes', '1') +) +SELECT m.xid = a.xid AS subxid_matches +FROM changes m, changes a +WHERE m.data LIKE 'streaming message:%' + AND a.data = 'aborting streamed (sub)transaction'; +RESET debug_logical_replication_streaming; + DROP TABLE stream_test; SELECT pg_drop_replication_slot('regression_slot'); diff --git a/contrib/test_decoding/test_decoding.c b/contrib/test_decoding/test_decoding.c index 78185837aef..961dc475a12 100644 --- a/contrib/test_decoding/test_decoding.c +++ b/contrib/test_decoding/test_decoding.c @@ -961,11 +961,18 @@ pg_decode_stream_message(LogicalDecodingContext *ctx, if (transactional) { TestDecodingData *data = ctx->output_plugin_private; - TestDecodingTxnData *txndata = txn->output_plugin_private; + + /* + * stream_message_cb can be called for an individual subtransaction + * but we maintain the output_plugin_private only under the toptxn so + * if this is not the toptxn then fetch the toptxn. + */ + ReorderBufferTXN *toptxn = rbtxn_get_toptxn(txn); + TestDecodingTxnData *txndata = toptxn->output_plugin_private; if (data->skip_empty_xacts && !txndata->stream_wrote_changes) { - pg_output_stream_start(ctx, data, txn, false); + pg_output_stream_start(ctx, data, toptxn, false); } txndata->xact_wrote_changes = txndata->stream_wrote_changes = true; } diff --git a/src/backend/replication/logical/reorderbuffer.c b/src/backend/replication/logical/reorderbuffer.c index 46843985317..7d53879d18e 100644 --- a/src/backend/replication/logical/reorderbuffer.c +++ b/src/backend/replication/logical/reorderbuffer.c @@ -2104,10 +2104,18 @@ ReorderBufferApplyMessage(ReorderBuffer *rb, ReorderBufferTXN *txn, ReorderBufferChange *change, bool streaming) { if (streaming) - rb->stream_message(rb, txn, change->lsn, true, + { + /* + * Pass the change's transaction entry (which could be a + * subtransaction) so that the output plugin can associate the message + * with it and discard the message if the subtransaction is aborted + * later. + */ + rb->stream_message(rb, change->txn, change->lsn, true, change->data.msg.prefix, change->data.msg.message_size, change->data.msg.message); + } else rb->message(rb, txn, change->lsn, true, change->data.msg.prefix, -- 2.56.0
v7-0002-Add-a-hook-for-handling-logical-messages-on-subsc.patch (text/x-patch, 121.3K) download From e44669b1ccf83800c9ad3ea40a9a8538581fb151 Mon Sep 17 00:00:00 2001 From: Masahiko Sawada <sawada.mshk@gmail.com> Date: Mon, 8 Jun 2026 11:25:47 -0700 Subject: [PATCH v7 2/2] Add a hook for handling logical messages on subscribers. Previously, pgoutput could emit logical decoding messages, those written with pg_logical_emit_message(), but the built-in subscriber never requested them from the publisher, and apply_handle_message() was only a placeholder that discarded the MESSAGE message type. Acting on such messages therefore meant writing a separate logical decoding client, and giving up the apply worker's streaming, conflict handling and progress tracking. Logical messages are a general-purpose side channel carried in band with the data, useful for replicating deparsed DDL, for carrying change-data-capture context that the row data does not have, and for cache invalidation delivered in commit order relative to the rows it refers to. This commit adds a new "messages" subscription option, which asks the publisher for the pgoutput "messages" option, and dispatches each received message to a new LogicalRepMessageHandle_hook. Behavior is unchanged when the option is off or when no handler is installed. A transactional message is applied as a step of the remote transaction it belongs to, so the handler's work commits atomically with that transaction, and transaction streaming and ALTER SUBSCRIPTION ... SKIP apply to it as they do to any other change. A non-transactional message belongs to no remote transaction and is committed on its own. A message may still reach the handler more than once, for example after an apply worker restart, so handlers must be idempotent. Note also that messages are not scoped by publication: every message emitted in the publisher database is sent, subject only to origin filtering, and pg_logical_emit_message() requires no special privileges. A handler must therefore treat the payload as untrusted input, and as an arbitrary string of bytes rather than as text in the subscriber's encoding. The publisher must be running PostgreSQL 14 or later, where pgoutput gained the "messages" option; against an older publisher the option is accepted but no message is ever sent. Bump catalog version. Reviewed-by: Bharath Rupireddy <bharath.rupireddyforpostgres@gmail.com> Reviewed-by: Fujii Masao <masao.fujii@gmail.com> Reviewed-by: Amit Kapila <amit.kapila16@gmail.com> Reviewed-by: Chao Li <li.evan.chao@gmail.com> Discussion: https://postgr.es/m/CAD21AoCTNGiddikkUcDKj5QLnsg-51bpr-o6L-GTHWZL4ZFYtQ@mail.gmail.com --- doc/src/sgml/catalogs.sgml | 11 + doc/src/sgml/func/func-admin.sgml | 7 +- doc/src/sgml/logical-replication.sgml | 35 ++ doc/src/sgml/ref/alter_subscription.sgml | 5 +- doc/src/sgml/ref/create_subscription.sgml | 43 +++ src/backend/catalog/pg_subscription.c | 1 + src/backend/catalog/system_views.sql | 5 +- src/backend/commands/subscriptioncmds.c | 27 +- .../libpqwalreceiver/libpqwalreceiver.c | 4 + src/backend/replication/logical/proto.c | 37 ++ src/backend/replication/logical/worker.c | 100 +++++- src/bin/pg_dump/pg_dump.c | 16 +- src/bin/pg_dump/pg_dump.h | 1 + src/bin/pg_dump/t/002_pg_dump.pl | 4 +- src/bin/psql/describe.c | 7 +- src/bin/psql/tab-complete.in.c | 6 +- src/include/catalog/pg_subscription.h | 5 + src/include/replication/logicalproto.h | 13 + src/include/replication/logicalworker.h | 49 +++ src/include/replication/walreceiver.h | 1 + src/test/modules/Makefile | 1 + src/test/modules/meson.build | 1 + .../modules/test_logicalmsg_hooks/.gitignore | 4 + .../modules/test_logicalmsg_hooks/Makefile | 20 ++ .../modules/test_logicalmsg_hooks/meson.build | 28 ++ .../test_logicalmsg_hooks/t/001_basic.pl | 318 ++++++++++++++++++ .../test_logicalmsg_hooks.c | 140 ++++++++ src/test/regress/expected/subscription.out | 202 ++++++----- src/test/regress/sql/subscription.sql | 17 + src/tools/pgindent/typedefs.list | 2 + 30 files changed, 1000 insertions(+), 110 deletions(-) create mode 100644 src/test/modules/test_logicalmsg_hooks/.gitignore create mode 100644 src/test/modules/test_logicalmsg_hooks/Makefile create mode 100644 src/test/modules/test_logicalmsg_hooks/meson.build create mode 100644 src/test/modules/test_logicalmsg_hooks/t/001_basic.pl create mode 100644 src/test/modules/test_logicalmsg_hooks/test_logicalmsg_hooks.c diff --git a/doc/src/sgml/catalogs.sgml b/doc/src/sgml/catalogs.sgml index 07ba05dccd9..5e7f3032baa 100644 --- a/doc/src/sgml/catalogs.sgml +++ b/doc/src/sgml/catalogs.sgml @@ -8250,6 +8250,17 @@ SCRAM-SHA-256$<replaceable>&lt;iteration count&gt;</replaceable>:<replaceable>&l </para></entry> </row> + <row> + <entry role="catalog_table_entry"><para role="column_definition"> + <structfield>submessages</structfield> <type>bool</type> + </para> + <para> + If true, the subscription requests the publisher to send logical + decoding messages, which are passed to a handler registered by an + extension on the subscriber. + </para></entry> + </row> + <row> <entry role="catalog_table_entry"><para role="column_definition"> <structfield>subconninfo</structfield> <type>text</type> diff --git a/doc/src/sgml/func/func-admin.sgml b/doc/src/sgml/func/func-admin.sgml index ed872264f02..e63d72f20ff 100644 --- a/doc/src/sgml/func/func-admin.sgml +++ b/doc/src/sgml/func/func-admin.sgml @@ -1479,7 +1479,12 @@ postgres=# SELECT '0/0'::pg_lsn + pd.segment_number * ps.setting::int + :offset <literal>false</literal>) controls if the message is immediately flushed to WAL or not. <parameter>flush</parameter> has no effect with <parameter>transactional</parameter>, as the message's WAL - record is flushed along with its transaction. + record is flushed along with its transaction. In addition to logical + decoding clients, messages emitted by this function are delivered to + subscribers whose subscriptions have + <link linkend="sql-createsubscription-params-with-messages"><literal>messages</literal></link> + enabled, where they are passed to a handler registered by an extension + on the subscriber. </para></entry> </row> diff --git a/doc/src/sgml/logical-replication.sgml b/doc/src/sgml/logical-replication.sgml index 4701a3d9d18..849724efc43 100644 --- a/doc/src/sgml/logical-replication.sgml +++ b/doc/src/sgml/logical-replication.sgml @@ -2661,6 +2661,41 @@ CONTEXT: processing remote data for replication origin "pg_16395" during "INSER not need privileges to <literal>SET ROLE</literal> to the sequence owner. </para> + <para> + Logical decoding messages are an exception to the user switching described + above. If the subscription has <link linkend="sql-createsubscription-params-with-messages"><literal>messages</literal></link> + enabled, the handler registered for them runs with the privileges of the + subscription owner. Messages are not restricted to the tables or schemas + of the subscribed publications either, and by default any role may call + <link linkend="pg-logical-emit-message"><function>pg_logical_emit_message</function></link>. + Any role that can connect to the publisher database can therefore choose + what the handler is given, and a handler that acts on what it receives does + so with the subscription owner's privileges. Restrict who can emit messages + on the publisher. Revoke the default privilege and grant it to a role created + for this purpose: +<programlisting> +REVOKE EXECUTE ON FUNCTION pg_logical_emit_message(boolean, text, text, boolean) FROM PUBLIC; +REVOKE EXECUTE ON FUNCTION pg_logical_emit_message(boolean, text, bytea, boolean) FROM PUBLIC; +GRANT EXECUTE ON FUNCTION pg_logical_emit_message(boolean, text, text, boolean) TO message_emitter; +GRANT EXECUTE ON FUNCTION pg_logical_emit_message(boolean, text, bytea, boolean) TO message_emitter; +</programlisting> + Function privileges are per database, so this has to be done in each database + that such a subscription replicates from, and in <literal>template1</literal> + if databases created later are to inherit it. No role that is not trusted to + drive the handler should be a member of the new role, since a role that can + <literal>SET ROLE</literal> to it can emit whatever it likes. Where a message + has to be emitted on behalf of an ordinary user, for example from an event + trigger that captures that user's DDL, use a <literal>SECURITY DEFINER</literal> + function owned by that role rather than granting the privilege more widely. + </para> + + <para> + This is a control on the publisher, and the subscriber has no way to verify + that it is in place. If you do not administer both the publisher and the + subscriber, do not install a message handler that does anything you would + not let every role on the publisher do. + </para> + <para> On the publisher, privileges are only checked once at the start of a replication connection and are not re-checked as each change record is read. diff --git a/doc/src/sgml/ref/alter_subscription.sgml b/doc/src/sgml/ref/alter_subscription.sgml index 545264e8a0a..72b1b0fcd83 100644 --- a/doc/src/sgml/ref/alter_subscription.sgml +++ b/doc/src/sgml/ref/alter_subscription.sgml @@ -302,8 +302,9 @@ ALTER SUBSCRIPTION <replaceable class="parameter">name</replaceable> RENAME TO < <link linkend="sql-createsubscription-params-with-two-phase"><literal>two_phase</literal></link>, <link linkend="sql-createsubscription-params-with-retain-dead-tuples"><literal>retain_dead_tuples</literal></link>, <link linkend="sql-createsubscription-params-with-max-retention-duration"><literal>max_retention_duration</literal></link>, - <link linkend="sql-createsubscription-params-with-wal-receiver-timeout"><literal>wal_receiver_timeout</literal></link>, and - <link linkend="sql-createsubscription-params-with-conflict-log-destination"><literal>conflict_log_destination</literal></link>. + <link linkend="sql-createsubscription-params-with-wal-receiver-timeout"><literal>wal_receiver_timeout</literal></link>, + <link linkend="sql-createsubscription-params-with-conflict-log-destination"><literal>conflict_log_destination</literal></link>, and + <link linkend="sql-createsubscription-params-with-messages"><literal>messages</literal></link>. Only a superuser can set <literal>password_required = false</literal>. </para> diff --git a/doc/src/sgml/ref/create_subscription.sgml b/doc/src/sgml/ref/create_subscription.sgml index 6248c228a64..1c38daaf434 100644 --- a/doc/src/sgml/ref/create_subscription.sgml +++ b/doc/src/sgml/ref/create_subscription.sgml @@ -478,6 +478,49 @@ CREATE SUBSCRIPTION <replaceable class="parameter">subscription_name</replaceabl </listitem> </varlistentry> + <varlistentry id="sql-createsubscription-params-with-messages"> + <term><literal>messages</literal> (<type>boolean</type>)</term> + <listitem> + <para> + Specifies whether the subscription requests the publisher to send + logical decoding messages, that is, those emitted with + <link linkend="pg-logical-emit-message"><function>pg_logical_emit_message</function></link>. + The default is <literal>false</literal>. + </para> + <para> + Received messages are only acted upon if an extension installed on + the subscriber has registered a handler for them; otherwise they are + received and discarded. A message may be passed to the handler more + than once, for example after an apply worker restart, so a handler + must be prepared to process the same message repeatedly. + </para> + <para> + A transactional message is applied as part of the remote transaction + that emitted it, and is therefore delivered in commit order with + respect to other transactions, not in the order in which it was written. + Note that + <link linkend="sql-createsubscription-params-with-streaming"><literal>streaming</literal></link> + and <link linkend="sql-altersubscription-params-skip"><literal>SKIP</literal></link> + apply to transactional messages just as they do to other changes. A + non-transactional message is delivered as soon as it is decoded, even + if the transaction that emitted it later aborted. + </para> + <para> + Messages are not restricted to the tables or schemas of the + subscribed publications: every message emitted in the publisher + database is sent, subject only to + <link linkend="sql-createsubscription-params-with-origin"><literal>origin</literal></link> + filtering. Message contents are supplied by whoever calls + <function>pg_logical_emit_message</function> on the publisher, which + by default any role may do, so a handler should treat them as + untrusted input. A message is an arbitrary string of bytes, and is + transmitted without character set conversion; it may therefore + contain embedded nulls, and need not be valid text in the encoding + of either database. + </para> + </listitem> + </varlistentry> + <varlistentry id="sql-createsubscription-params-with-origin"> <term><literal>origin</literal> (<type>string</type>)</term> <listitem> diff --git a/src/backend/catalog/pg_subscription.c b/src/backend/catalog/pg_subscription.c index 60c15b22194..138cdecd436 100644 --- a/src/backend/catalog/pg_subscription.c +++ b/src/backend/catalog/pg_subscription.c @@ -133,6 +133,7 @@ GetSubscription(Oid subid, bool missing_ok) sub->maxretention = subform->submaxretention; sub->retentionactive = subform->subretentionactive; sub->conflictlogrelid = subform->subconflictlogrelid; + sub->messages = subform->submessages; /* Get slotname */ datum = SysCacheGetAttr(SUBSCRIPTIONOID, diff --git a/src/backend/catalog/system_views.sql b/src/backend/catalog/system_views.sql index 6dce9192ba0..19b96bba1fd 100644 --- a/src/backend/catalog/system_views.sql +++ b/src/backend/catalog/system_views.sql @@ -1545,8 +1545,9 @@ GRANT SELECT (oid, subdbid, subskiplsn, subname, subowner, subenabled, subbinary, substream, subtwophasestate, subdisableonerr, subpasswordrequired, subrunasowner, subfailover, subretaindeadtuples, submaxretention, subretentionactive, - subserver, subconflictlogrelid, subconflictlogdest, subslotname, - subsynccommit, subwalrcvtimeout, subpublications, suborigin) + subserver, subconflictlogrelid, subconflictlogdest, submessages, + subslotname, subsynccommit, subwalrcvtimeout, subpublications, + suborigin) ON pg_subscription TO public; CREATE VIEW pg_stat_subscription_stats AS diff --git a/src/backend/commands/subscriptioncmds.c b/src/backend/commands/subscriptioncmds.c index ce45f7e16d4..9dc8d608b8d 100644 --- a/src/backend/commands/subscriptioncmds.c +++ b/src/backend/commands/subscriptioncmds.c @@ -84,6 +84,7 @@ #define SUBOPT_LSN 0x00020000 #define SUBOPT_ORIGIN 0x00040000 #define SUBOPT_CONFLICT_LOG_DEST 0x00080000 +#define SUBOPT_MESSAGES 0x00100000 /* check if the 'val' has 'bits' set */ #define IsSet(val, bits) (((val) & (bits)) == (bits)) @@ -110,6 +111,7 @@ typedef struct SubOpts bool runasowner; bool failover; bool retaindeadtuples; + bool messages; int32 maxretention; char *origin; ConflictLogDest conflictlogdest; @@ -208,6 +210,8 @@ parse_subscription_options(ParseState *pstate, List *stmt_options, opts->origin = pstrdup(LOGICALREP_ORIGIN_ANY); if (IsSet(supported_opts, SUBOPT_CONFLICT_LOG_DEST)) opts->conflictlogdest = CONFLICT_LOG_DEST_LOG; + if (IsSet(supported_opts, SUBOPT_MESSAGES)) + opts->messages = false; /* Parse options */ foreach(lc, stmt_options) @@ -460,6 +464,15 @@ parse_subscription_options(ParseState *pstate, List *stmt_options, opts->conflictlogdest = GetConflictLogDest(val); opts->specified_opts |= SUBOPT_CONFLICT_LOG_DEST; } + else if (IsSet(supported_opts, SUBOPT_MESSAGES) && + strcmp(defel->defname, "messages") == 0) + { + if (IsSet(opts->specified_opts, SUBOPT_MESSAGES)) + errorConflictingDefElem(defel, pstate); + + opts->specified_opts |= SUBOPT_MESSAGES; + opts->messages = defGetBoolean(defel); + } else ereport(ERROR, (errcode(ERRCODE_SYNTAX_ERROR), @@ -700,7 +713,7 @@ CreateSubscription(ParseState *pstate, CreateSubscriptionStmt *stmt, SUBOPT_RETAIN_DEAD_TUPLES | SUBOPT_MAX_RETENTION_DURATION | SUBOPT_WAL_RECEIVER_TIMEOUT | SUBOPT_ORIGIN | - SUBOPT_CONFLICT_LOG_DEST); + SUBOPT_CONFLICT_LOG_DEST | SUBOPT_MESSAGES); parse_subscription_options(pstate, stmt->options, supported_opts, &opts); /* @@ -872,6 +885,8 @@ CreateSubscription(ParseState *pstate, CreateSubscriptionStmt *stmt, values[Anum_pg_subscription_subretentionactive - 1] = BoolGetDatum(opts.retaindeadtuples); values[Anum_pg_subscription_subserver - 1] = ObjectIdGetDatum(serverid); + values[Anum_pg_subscription_submessages - 1] = + BoolGetDatum(opts.messages); if (stmt->conninfo) { Assert(stmt->conninfo == conninfo && !OidIsValid(serverid)); @@ -1714,7 +1729,8 @@ AlterSubscription(ParseState *pstate, AlterSubscriptionStmt *stmt, SUBOPT_MAX_RETENTION_DURATION | SUBOPT_WAL_RECEIVER_TIMEOUT | SUBOPT_ORIGIN | - SUBOPT_CONFLICT_LOG_DEST); + SUBOPT_CONFLICT_LOG_DEST | + SUBOPT_MESSAGES); break; case ALTER_SUBSCRIPTION_ENABLED: @@ -2128,6 +2144,13 @@ AlterSubscription(ParseState *pstate, AlterSubscriptionStmt *stmt, } } + if (IsSet(opts.specified_opts, SUBOPT_MESSAGES)) + { + values[Anum_pg_subscription_submessages - 1] = + BoolGetDatum(opts.messages); + replaces[Anum_pg_subscription_submessages - 1] = true; + } + update_tuple = true; break; } diff --git a/src/backend/replication/libpqwalreceiver/libpqwalreceiver.c b/src/backend/replication/libpqwalreceiver/libpqwalreceiver.c index dcc738f4e38..ffd03909ab4 100644 --- a/src/backend/replication/libpqwalreceiver/libpqwalreceiver.c +++ b/src/backend/replication/libpqwalreceiver/libpqwalreceiver.c @@ -664,6 +664,10 @@ libpqrcv_startstreaming(WalReceiverConn *conn, appendQuotedLiteral(&cmd, options->proto.logical.origin); } + if (options->proto.logical.messages && + PQserverVersion(conn->streamConn) >= 140000) + appendStringInfo(&cmd, ", messages 'on'"); + pubnames = options->proto.logical.publication_names; pubnames_str = stringlist_to_identifierstr(pubnames); appendStringInfoString(&cmd, ", publication_names "); diff --git a/src/backend/replication/logical/proto.c b/src/backend/replication/logical/proto.c index 86ad97cd937..6461361d68d 100644 --- a/src/backend/replication/logical/proto.c +++ b/src/backend/replication/logical/proto.c @@ -660,6 +660,43 @@ logicalrep_write_message(StringInfo out, TransactionId xid, XLogRecPtr lsn, pq_sendbytes(out, message, sz); } +/* + * Read MESSAGE from stream. + */ +void +logicalrep_read_message(StringInfo in, LogicalRepMessageData *msg_data) +{ + Size len; + char *msg; + uint8 flags; + + /* read and decode flags */ + flags = pq_getmsgint(in, 1); + msg_data->transactional = (flags & MESSAGE_TRANSACTIONAL) != 0; + + msg_data->lsn = pq_getmsgint64(in); + msg_data->prefix = pstrdup(pq_getmsgstring(in)); + + /* read message length */ + len = pq_getmsgint(in, 4); + msg_data->message_size = len; + + /* and data */ + msg = palloc(len + 1); + pq_copymsgbytes(in, msg, len); + + /* + * The payload is an arbitrary string of bytes, so message_size, not + * strlen(), is what gives its length. NULL-terminate it anyway, as + * logicalrep_read_tuple() does for column values: that way it can be + * handed to code expecting a C string without reading past the end, which + * is convenient for logging and debugging. Such code will of course stop + * at the first embedded null, if any. + */ + msg[len] = '\0'; + msg_data->message = msg; +} + /* * Write relation description to the output stream. */ diff --git a/src/backend/replication/logical/worker.c b/src/backend/replication/logical/worker.c index 95044620e68..166fdc5b2fb 100644 --- a/src/backend/replication/logical/worker.c +++ b/src/backend/replication/logical/worker.c @@ -561,6 +561,9 @@ typedef struct ApplySubXactData static ApplySubXactData subxact_data = {0, 0, InvalidTransactionId, NULL}; +/* Hook for plugins to get control in handling logical decoding messages */ +LogicalRepMessageHandle_hook_type LogicalRepMessageHandle_hook = NULL; + static inline void subxact_filename(char *path, Oid subid, TransactionId xid); static inline void changes_filename(char *path, Oid subid, TransactionId xid); @@ -1709,6 +1712,84 @@ apply_handle_origin(StringInfo s) errmsg_internal("ORIGIN message sent out of order"))); } +/* + * Handle MESSAGE message. + * + * Invoke a hook function if set. + */ +static void +apply_handle_message(StringInfo s) +{ + LogicalRepMessageData msg; + + /* + * The publisher must not send a message that this worker did not ask for; + * see set_stream_options(). + */ + if (!MySubscription->messages || am_tablesync_worker()) + ereport(ERROR, + (errcode(ERRCODE_PROTOCOL_VIOLATION), + errmsg_internal("MESSAGE message received but messages are not enabled for this worker"))); + + if (!LogicalRepMessageHandle_hook) + return; + + if (is_skipping_changes() || + handle_streamed_transaction(LOGICAL_REP_MSG_MESSAGE, s)) + return; + + begin_replication_step(); + + logicalrep_read_message(s, &msg); + + /* + * The publisher sends a non-transactional message on its own, never + * inside a transaction or a streamed block; see + * ReorderBufferQueueMessage(). Reject one that arrives inside a + * transaction before calling the handler, as the commit below would + * otherwise split the remote transaction in two. + */ + if (!msg.transactional && + (in_remote_transaction || in_streamed_transaction)) + ereport(ERROR, + (errcode(ERRCODE_PROTOCOL_VIOLATION), + errmsg_internal("non-transactional MESSAGE message received inside a transaction"))); + + (*LogicalRepMessageHandle_hook) (&msg); + + end_replication_step(); + + /* + * A transactional message is applied as a step of the remote transaction + * that emitted it, and is committed together with it when applying the + * commit message. A non-transactional message belongs to no remote + * transaction, so commit it here. + */ + if (!msg.transactional) + { + /* + * The message doesn't belong to any remote transaction, so there is + * no remote commit LSN nor timestamp to record. Clear the state left + * over by the previously applied transaction so that this commit + * doesn't inherit it. + */ + replorigin_xact_clear(false); + + CommitTransactionCommand(); + + pgstat_report_stat(false); + + /* + * Report the flush position only after the local commit is durable. + * Otherwise send_feedback() would report everything received so far + * as flushed, the publisher would advance confirmed_flush_lsn past + * this message, and a crash before the commit reaches disk would lose + * the handler's work without the message ever being sent again. + */ + store_flush_position(msg.lsn, XactLastCommitEnd); + } +} + /* * Initialize fileset (if not already done). * @@ -3894,12 +3975,7 @@ apply_dispatch(StringInfo s) break; case LOGICAL_REP_MSG_MESSAGE: - - /* - * Logical replication does not use generic logical messages yet. - * Although, it could be used by other applications that use this - * output plugin. - */ + apply_handle_message(s); break; case LOGICAL_REP_MSG_STREAM_START: @@ -5185,7 +5261,8 @@ maybe_reread_subscription(void) newsub->passwordrequired != MySubscription->passwordrequired || strcmp(newsub->origin, MySubscription->origin) != 0 || newsub->owner != MySubscription->owner || - !equal(newsub->publications, MySubscription->publications)) + !equal(newsub->publications, MySubscription->publications) || + newsub->messages != MySubscription->messages) { if (am_parallel_apply_worker()) ereport(LOG, @@ -5676,6 +5753,15 @@ set_stream_options(WalRcvStreamOptions *options, options->proto.logical.twophase = false; options->proto.logical.origin = pstrdup(MySubscription->origin); + + /* + * Logical messages are relation-agnostic, so they don't map cleanly onto + * the tablesync worker of a particular relation, and there would be no + * well-defined ordering between a message and the initial copy. Leave + * them to the (parallel) apply workers. + */ + options->proto.logical.messages = (MySubscription->messages && + !am_tablesync_worker()); } /* diff --git a/src/bin/pg_dump/pg_dump.c b/src/bin/pg_dump/pg_dump.c index 4fc74e6e26e..efc60fdefb0 100644 --- a/src/bin/pg_dump/pg_dump.c +++ b/src/bin/pg_dump/pg_dump.c @@ -5142,6 +5142,7 @@ getSubscriptions(Archive *fout) int i_subfailover; int i_subretaindeadtuples; int i_submaxretention; + int i_submessages; int i, ntups; @@ -5240,9 +5241,14 @@ getSubscriptions(Archive *fout) " '-1' AS subwalrcvtimeout,\n"); if (fout->remoteVersion >= 190000) - appendPQExpBufferStr(query, " fs.srvname AS subservername\n"); + appendPQExpBufferStr(query, " fs.srvname AS subservername,\n"); else - appendPQExpBufferStr(query, " NULL AS subservername\n"); + appendPQExpBufferStr(query, " NULL AS subservername,\n"); + + if (fout->remoteVersion >= 200000) + appendPQExpBufferStr(query, " s.submessages\n"); + else + appendPQExpBufferStr(query, " false AS submessages\n"); appendPQExpBufferStr(query, "FROM pg_subscription s\n"); @@ -5291,6 +5297,7 @@ getSubscriptions(Archive *fout) i_subpublications = PQfnumber(res, "subpublications"); i_suborigin = PQfnumber(res, "suborigin"); i_suboriginremotelsn = PQfnumber(res, "suboriginremotelsn"); + i_submessages = PQfnumber(res, "submessages"); subinfo = pg_malloc_array(SubscriptionInfo, ntups); @@ -5348,6 +5355,8 @@ getSubscriptions(Archive *fout) else subinfo[i].suboriginremotelsn = pg_strdup(PQgetvalue(res, i, i_suboriginremotelsn)); + subinfo[i].submessages = + (strcmp(PQgetvalue(res, i, i_submessages), "t") == 0); /* Decide whether we want to dump it */ selectDumpableObject(&(subinfo[i].dobj), fout); @@ -5600,6 +5609,9 @@ dumpSubscription(Archive *fout, const SubscriptionInfo *subinfo) if (subinfo->subretaindeadtuples) appendPQExpBufferStr(query, ", retain_dead_tuples = true"); + if (subinfo->submessages) + appendPQExpBufferStr(query, ", messages = true"); + if (subinfo->submaxretention) appendPQExpBuffer(query, ", max_retention_duration = %d", subinfo->submaxretention); diff --git a/src/bin/pg_dump/pg_dump.h b/src/bin/pg_dump/pg_dump.h index 41d932f003b..1df8a5252a4 100644 --- a/src/bin/pg_dump/pg_dump.h +++ b/src/bin/pg_dump/pg_dump.h @@ -721,6 +721,7 @@ typedef struct _SubscriptionInfo bool subrunasowner; bool subfailover; bool subretaindeadtuples; + bool submessages; int submaxretention; char *subservername; char *subconninfo; diff --git a/src/bin/pg_dump/t/002_pg_dump.pl b/src/bin/pg_dump/t/002_pg_dump.pl index ec73c4d5be9..a32710d085a 100644 --- a/src/bin/pg_dump/t/002_pg_dump.pl +++ b/src/bin/pg_dump/t/002_pg_dump.pl @@ -3424,9 +3424,9 @@ my %tests = ( create_order => 50, create_sql => 'CREATE SUBSCRIPTION sub3 CONNECTION \'dbname=doesnotexist\' PUBLICATION pub1 - WITH (connect = false, origin = any, streaming = on);', + WITH (connect = false, origin = any, streaming = on, messages = on);', regexp => qr/^ - \QCREATE SUBSCRIPTION sub3 CONNECTION 'dbname=doesnotexist' PUBLICATION pub1 WITH (connect = false, slot_name = 'sub3', streaming = on);\E + \QCREATE SUBSCRIPTION sub3 CONNECTION 'dbname=doesnotexist' PUBLICATION pub1 WITH (connect = false, slot_name = 'sub3', streaming = on, messages = true);\E /xm, like => { %full_runs, section_post_data => 1, }, unlike => { diff --git a/src/bin/psql/describe.c b/src/bin/psql/describe.c index e4a4ba47280..e6e48a9c66c 100644 --- a/src/bin/psql/describe.c +++ b/src/bin/psql/describe.c @@ -6774,7 +6774,7 @@ describeSubscriptions(const char *pattern, bool verbose) printQueryOpt myopt = pset.popt; static const bool translate_columns[] = {false, false, false, false, false, false, false, false, false, false, false, false, false, false, - false, false, false, false, false, false, false}; + false, false, false, false, false, false, false, false}; initPQExpBuffer(&buf); @@ -6852,6 +6852,11 @@ describeSubscriptions(const char *pattern, bool verbose) gettext_noop("Retention active")); } + if (pset.sversion >= 200000) + appendPQExpBuffer(&buf, + ", submessages AS \"%s\"\n", + gettext_noop("Messages")); + appendPQExpBuffer(&buf, ", subsynccommit AS \"%s\"\n" ", subconninfo AS \"%s\"\n", diff --git a/src/bin/psql/tab-complete.in.c b/src/bin/psql/tab-complete.in.c index 96de78cd087..2346c8b969f 100644 --- a/src/bin/psql/tab-complete.in.c +++ b/src/bin/psql/tab-complete.in.c @@ -2360,8 +2360,8 @@ match_previous_words(int pattern_id, /* ALTER SUBSCRIPTION <name> SET ( */ else if (Matches("ALTER", "SUBSCRIPTION", MatchAny, MatchAnyN, "SET", "(")) COMPLETE_WITH("binary", "conflict_log_destination", "disable_on_error", - "failover", "max_retention_duration", "origin", - "password_required", "retain_dead_tuples", + "failover", "max_retention_duration", "messages", + "origin", "password_required", "retain_dead_tuples", "run_as_owner", "slot_name", "streaming", "synchronous_commit", "two_phase", "wal_receiver_timeout"); @@ -3897,7 +3897,7 @@ match_previous_words(int pattern_id, else if (Matches("CREATE", "SUBSCRIPTION", MatchAnyN, "WITH", "(")) COMPLETE_WITH("binary", "conflict_log_destination", "connect", "copy_data", "create_slot", "disable_on_error", "enabled", "failover", - "max_retention_duration", "origin", + "max_retention_duration", "messages", "origin", "password_required", "retain_dead_tuples", "run_as_owner", "slot_name", "streaming", "synchronous_commit", "two_phase", diff --git a/src/include/catalog/pg_subscription.h b/src/include/catalog/pg_subscription.h index b860f02d166..07b28f86e0f 100644 --- a/src/include/catalog/pg_subscription.h +++ b/src/include/catalog/pg_subscription.h @@ -96,6 +96,9 @@ CATALOG(pg_subscription,6100,SubscriptionRelationId) BKI_SHARED_RELATION BKI_ROW * server */ Oid subconflictlogrelid; /* Relid of the conflict log table. */ + + bool submessages; /* True if the subscription wants to receive + * logical messages. */ #ifdef CATALOG_VARLEN /* variable-length fields start here */ /* @@ -173,6 +176,8 @@ typedef struct Subscription * exceeded max_retention_duration, when * defined */ Oid conflictlogrelid; /* conflict log table Oid */ + bool messages; /* True if the subscription wants to receive + * logical messages */ char *slotname; /* Name of the replication slot */ char *synccommit; /* Synchronous commit setting for worker */ char *walrcvtimeout; /* wal_receiver_timeout setting for worker */ diff --git a/src/include/replication/logicalproto.h b/src/include/replication/logicalproto.h index 058a955e20c..83741a2080d 100644 --- a/src/include/replication/logicalproto.h +++ b/src/include/replication/logicalproto.h @@ -191,6 +191,18 @@ typedef struct LogicalRepStreamAbortData TimestampTz abort_time; } LogicalRepStreamAbortData; +/* + * Logical decoding message information + */ +typedef struct LogicalRepMessageData +{ + XLogRecPtr lsn; + bool transactional; + const char *prefix; + Size message_size; /* length of message in bytes */ + const char *message; /* payload; may contain embedded nulls */ +} LogicalRepMessageData; + extern void logicalrep_write_begin(StringInfo out, ReorderBufferTXN *txn); extern void logicalrep_read_begin(StringInfo in, LogicalRepBeginData *begin_data); @@ -248,6 +260,7 @@ extern List *logicalrep_read_truncate(StringInfo in, bool *cascade, bool *restart_seqs); extern void logicalrep_write_message(StringInfo out, TransactionId xid, XLogRecPtr lsn, bool transactional, const char *prefix, Size sz, const char *message); +extern void logicalrep_read_message(StringInfo in, LogicalRepMessageData *msg_data); extern void logicalrep_write_rel(StringInfo out, TransactionId xid, Relation rel, Bitmapset *columns, PublishGencolsType include_gencols_type); diff --git a/src/include/replication/logicalworker.h b/src/include/replication/logicalworker.h index 7d748a28da8..3f745201c1a 100644 --- a/src/include/replication/logicalworker.h +++ b/src/include/replication/logicalworker.h @@ -16,6 +16,55 @@ extern PGDLLIMPORT volatile sig_atomic_t ParallelApplyMessagePending; +/* + * Forward-declared so that this header, which is included by several files + * unrelated to logical replication, does not have to include logicalproto.h. + * A handler that dereferences the struct must include that header itself. + */ +struct LogicalRepMessageData; + +/* + * Hook for extensions to handle logical decoding messages (see + * pg_logical_emit_message()) received from the publisher. Called only when + * the subscription has the "messages" option enabled. + * + * The handler runs in the apply worker, inside a transaction, with an active + * snapshot pushed. For a transactional message the handler's work is part of + * the remote transaction and commits with it; a non-transactional message is + * committed as an independent unit. The same message may be delivered more + * than once, so the handler must be idempotent. + * + * The payload is an arbitrary string of bytes. The WAL record makes no + * distinction between the text and the bytea form of + * pg_logical_emit_message(), and the bytes are transmitted without character + * set conversion, so the payload may contain embedded nulls and bytes that + * are not valid in the subscriber's encoding. Its length is given by + * msg->message_size, not by strlen(); the buffer is null-terminated only for + * convenience, and that null is not part of the payload. + * + * The handler always runs with the privileges of the subscription owner. + * The payload is chosen by whoever called pg_logical_emit_message() on the + * publisher, which by default any role may do, so a handler that acts on the + * payload gives every role on the publisher the subscription owner's + * privileges. A handler that needs less than that can switch to a less + * privileged role with SwitchToUntrustedUser(). It must take that role + * from its own configuration and never from the payload, which is written + * by the very user the switch defends against, and it should check the + * role with member_can_set_role() first: SwitchToUntrustedUser() throws an + * error if the subscription owner cannot SET ROLE to it, and that error + * loops as described below. MySubscription, declared in + * worker_internal.h, is available where the choice differs per + * subscription. + * + * Note that the apply worker runs with an empty search_path. An ERROR thrown + * by the handler restarts the apply worker, or disables the subscription if + * disable_on_error is set. Because the message is delivered again after the + * restart, a handler cannot reject a message by throwing an error; that only + * loops. A message the handler does not want must simply be ignored. + */ +typedef void (*LogicalRepMessageHandle_hook_type) (const struct LogicalRepMessageData *msg); +extern PGDLLIMPORT LogicalRepMessageHandle_hook_type LogicalRepMessageHandle_hook; + extern void ApplyWorkerMain(Datum main_arg); extern void ParallelApplyWorkerMain(Datum main_arg); extern void TableSyncWorkerMain(Datum main_arg); diff --git a/src/include/replication/walreceiver.h b/src/include/replication/walreceiver.h index 5cfe90f9989..42a1dab24b2 100644 --- a/src/include/replication/walreceiver.h +++ b/src/include/replication/walreceiver.h @@ -188,6 +188,7 @@ typedef struct * prepare time */ char *origin; /* Only publish data originating from the * specified origin */ + bool messages; /* Ask publisher to send logical messages */ } logical; } proto; } WalRcvStreamOptions; diff --git a/src/test/modules/Makefile b/src/test/modules/Makefile index 6c1504ac932..a8a31f2b23e 100644 --- a/src/test/modules/Makefile +++ b/src/test/modules/Makefile @@ -37,6 +37,7 @@ SUBDIRS = \ test_integerset \ test_json_parser \ test_lfind \ + test_logicalmsg_hooks \ test_lwlock_tranches \ test_misc \ test_oat_hooks \ diff --git a/src/test/modules/meson.build b/src/test/modules/meson.build index ef3cdb8faf9..f1aaeae71ab 100644 --- a/src/test/modules/meson.build +++ b/src/test/modules/meson.build @@ -39,6 +39,7 @@ subdir('test_int128') subdir('test_integerset') subdir('test_json_parser') subdir('test_lfind') +subdir('test_logicalmsg_hooks') subdir('test_lwlock_tranches') subdir('test_misc') subdir('test_oat_hooks') diff --git a/src/test/modules/test_logicalmsg_hooks/.gitignore b/src/test/modules/test_logicalmsg_hooks/.gitignore new file mode 100644 index 00000000000..5dcb3ff9723 --- /dev/null +++ b/src/test/modules/test_logicalmsg_hooks/.gitignore @@ -0,0 +1,4 @@ +# Generated subdirectories +/log/ +/results/ +/tmp_check/ diff --git a/src/test/modules/test_logicalmsg_hooks/Makefile b/src/test/modules/test_logicalmsg_hooks/Makefile new file mode 100644 index 00000000000..af129009b8a --- /dev/null +++ b/src/test/modules/test_logicalmsg_hooks/Makefile @@ -0,0 +1,20 @@ +# src/test/modules/test_logicalmsg_hooks/Makefile + +MODULE_big = test_logicalmsg_hooks +OBJS = \ + $(WIN32RES) \ + test_logicalmsg_hooks.o +PGFILEDESC = "test_logicalmsg_hooks - test logical replication message hooks" + +TAP_TESTS = 1 + +ifdef USE_PGXS +PG_CONFIG = pg_config +PGXS := $(shell $(PG_CONFIG) --pgxs) +include $(PGXS) +else +subdir = src/test/modules/test_logicalmsg_hooks +top_builddir = ../../../.. +include $(top_builddir)/src/Makefile.global +include $(top_srcdir)/contrib/contrib-global.mk +endif diff --git a/src/test/modules/test_logicalmsg_hooks/meson.build b/src/test/modules/test_logicalmsg_hooks/meson.build new file mode 100644 index 00000000000..8084d82e47b --- /dev/null +++ b/src/test/modules/test_logicalmsg_hooks/meson.build @@ -0,0 +1,28 @@ +# Copyright (c) 2022-2026, PostgreSQL Global Development Group + +test_logicalmsg_hooks_sources = files( + 'test_logicalmsg_hooks.c', +) + +if host_system == 'windows' + test_logicalmsg_hooks_sources += rc_lib_gen.process(win32ver_rc, extra_args: [ + '--NAME', 'test_logicalmsg_hooks', + '--FILEDESC', 'test_logicalmsg_hooks - test logical message hooks',]) +endif + +test_logicalmsg_hooks = shared_module('test_logicalmsg_hooks', + test_logicalmsg_hooks_sources, + kwargs: pg_test_mod_args, +) +test_install_libs += test_logicalmsg_hooks + +tests += { + 'name': 'test_logicalmsg_hooks', + 'sd': meson.current_source_dir(), + 'bd': meson.current_build_dir(), + 'tap': { + 'tests': [ + 't/001_basic.pl', + ], + }, +} diff --git a/src/test/modules/test_logicalmsg_hooks/t/001_basic.pl b/src/test/modules/test_logicalmsg_hooks/t/001_basic.pl new file mode 100644 index 00000000000..a86de1273ab --- /dev/null +++ b/src/test/modules/test_logicalmsg_hooks/t/001_basic.pl @@ -0,0 +1,318 @@ +# Copyright (c) 2026, PostgreSQL Global Development Group + +# Test LogicalRepMessageHandle_hook. The test module records every message it +# receives into public.test_logicalmsg_log on the subscriber, so most checks +# here are plain queries against that table. + +use strict; +use warnings FATAL => 'all'; + +use PostgreSQL::Test::Cluster; +use PostgreSQL::Test::Utils; +use Test::More; + +my $pub = PostgreSQL::Test::Cluster->new('publisher'); +$pub->init(allows_streaming => 'logical'); + +# Keep the memory limit low so that the transactions below are streamed. +$pub->append_conf( + 'postgresql.conf', q{ +logical_decoding_work_mem = 64kB +max_prepared_transactions = 10 +}); +$pub->start; + +my $sub = PostgreSQL::Test::Cluster->new('subscriber'); +$sub->init(allows_streaming => 'logical'); +$sub->append_conf( + 'postgresql.conf', q{ +shared_preload_libraries = 'test_logicalmsg_hooks' +max_parallel_apply_workers_per_subscription = 2 +max_prepared_transactions = 10 +}); +$sub->start; + +$pub->safe_psql( + 'postgres', qq{ +CREATE TABLE test (a int); +INSERT INTO test VALUES (1); +CREATE PUBLICATION pub FOR ALL TABLES; +}); + +my $pub_connstr = $pub->connstr . ' dbname=postgres'; + +$sub->safe_psql( + 'postgres', qq{ +CREATE TABLE test (a int primary key); +CREATE SUBSCRIPTION sub CONNECTION '$pub_connstr' PUBLICATION pub WITH (messages = true, disable_on_error = true) +}); + +$sub->wait_for_subscription_sync($pub, 'sub'); + +# Number of messages recorded under the given prefix. +sub logged_count +{ + my ($prefix) = @_; + + return $sub->safe_psql('postgres', + "SELECT count(*) FROM test_logicalmsg_log WHERE prefix = '$prefix'"); +} + +# Verify that a transactional message is passed to the hook, and that the +# message data given to the hook is correct. +$pub->safe_psql('postgres', + q{SELECT pg_logical_emit_message(true, 'basic', 'transactional message')} +); +$pub->wait_for_catchup('sub'); + +is( $sub->safe_psql( + 'postgres', + q{SELECT transactional, message_size, message FROM test_logicalmsg_log + WHERE prefix = 'basic'}), + 't|21|transactional message', + "transactional message is passed to the hook"); + +# Verify that a non-transactional message is applied even if the transaction +# that emitted it rolls back. +$pub->safe_psql( + 'postgres', q{ +BEGIN; +SELECT pg_logical_emit_message(false, 'nontransactional', 'emitted then rolled back'); +ROLLBACK; +}); + +# Commit another message to flush WAL, as ROLLBACK doesn't flush. +$pub->safe_psql('postgres', + q{SELECT pg_logical_emit_message(true, 'flush', 'flush')}); +$pub->wait_for_catchup('sub'); + +is( $sub->safe_psql( + 'postgres', + q{SELECT transactional, message FROM test_logicalmsg_log + WHERE prefix = 'nontransactional'}), + 'f|emitted then rolled back', + "non-transactional message of an aborted transaction is applied"); + +# Verify that a transactional message emitted in a subtransaction that is +# rolled back is not passed to the hook. +$pub->safe_psql( + 'postgres', q{ +BEGIN; +SAVEPOINT sp; +SELECT pg_logical_emit_message(true, 'subxact_aborted', 'not delivered'); +ROLLBACK TO SAVEPOINT sp; +SELECT pg_logical_emit_message(true, 'subxact_committed', 'delivered'); +COMMIT; +}); +$pub->wait_for_catchup('sub'); + +is(logged_count('subxact_aborted'), + 0, "message of an aborted subtransaction is not applied"); +is(logged_count('subxact_committed'), + 1, "message of a committed subtransaction is applied"); + +# Verify that transactional messages are applied in streaming transactions, +# both when the leader apply worker serializes the transaction to a file and +# when it hands the transaction to a parallel apply worker. +foreach my $mode ('on', 'parallel') +{ + $sub->safe_psql('postgres', + "ALTER SUBSCRIPTION sub SET (streaming = $mode)"); + + # Reconnect so that the new streaming mode takes effect immediately. + $sub->safe_psql('postgres', 'ALTER SUBSCRIPTION sub DISABLE'); + $sub->safe_psql('postgres', 'ALTER SUBSCRIPTION sub ENABLE'); + + $pub->safe_psql('postgres', + q{SELECT pg_stat_reset_replication_slot('sub')}); + + $pub->safe_psql( + 'postgres', qq{ +BEGIN; +SAVEPOINT sp; +SELECT pg_logical_emit_message(true, 'streaming_subxact_aborted_$mode', 'not delivered'); +INSERT INTO test SELECT generate_series(1000, 6000); +ROLLBACK TO SAVEPOINT sp; +SELECT pg_logical_emit_message(true, 'streaming_$mode', 'streamed message'); +COMMIT; +}); + $pub->wait_for_catchup('sub'); + + # Make sure the transaction really was streamed, so that this test cannot + # silently degrade into a non-streaming one. + $pub->poll_query_until('postgres', + q{SELECT stream_txns > 0 FROM pg_stat_replication_slots WHERE slot_name = 'sub'} + ) or die "transaction was not streamed"; + + is(logged_count("streaming_subxact_aborted_$mode"), 0, + "message of an aborted streamed subtransaction is not applied (streaming = $mode)" + ); + is(logged_count("streaming_$mode"), + 1, + "message in a streamed transaction is applied (streaming = $mode)"); + + $pub->safe_psql('postgres', 'TRUNCATE test'); + $pub->wait_for_catchup('sub'); +} + +# Verify that a transaction whose only change is a message is applied when it +# is streamed. There is no volume to trigger streaming, so force it. The +# setting is picked up by the walsender started when the subscription is +# re-enabled below. +$pub->safe_psql('postgres', + 'ALTER SYSTEM SET debug_logical_replication_streaming = immediate'); +$pub->reload; +$pub->wait_for_log( + qr/parameter "debug_logical_replication_streaming" changed to "immediate"/ +); + +foreach my $mode ('on', 'parallel') +{ + $sub->safe_psql('postgres', + "ALTER SUBSCRIPTION sub SET (streaming = $mode)"); + $sub->safe_psql('postgres', 'ALTER SUBSCRIPTION sub DISABLE'); + $sub->safe_psql('postgres', 'ALTER SUBSCRIPTION sub ENABLE'); + + $pub->safe_psql('postgres', + q{SELECT pg_stat_reset_replication_slot('sub')}); + + $pub->safe_psql( + 'postgres', qq{ +BEGIN; +SELECT pg_logical_emit_message(true, 'streaming_only_$mode', 'the only change'); +COMMIT; +}); + $pub->wait_for_catchup('sub'); + + $pub->poll_query_until('postgres', + q{SELECT stream_txns > 0 FROM pg_stat_replication_slots WHERE slot_name = 'sub'} + ) or die "transaction was not streamed"; + + is(logged_count("streaming_only_$mode"), 1, + "message that is the only change of a streamed transaction is applied (streaming = $mode)" + ); +} + +$pub->safe_psql('postgres', + 'ALTER SYSTEM RESET debug_logical_replication_streaming'); +$pub->reload; +$pub->wait_for_log( + qr/parameter "debug_logical_replication_streaming" removed from configuration file/ +); + +$sub->safe_psql('postgres', 'ALTER SUBSCRIPTION sub SET (streaming = off)'); + +# Verify that ALTER SUBSCRIPTION ... SKIP skips a transactional message along +# with the rest of its transaction. The message is emitted before the +# conflicting change, so the hook runs and records the message on the first +# attempt; that work must be rolled back with the failed transaction, and must +# not be redone once the transaction is skipped. +$pub->safe_psql('postgres', 'INSERT INTO test VALUES (1)'); +$pub->wait_for_catchup('sub'); + +my $log_location = -s $sub->logfile; + +$pub->safe_psql( + 'postgres', q{ +BEGIN; +SELECT pg_logical_emit_message(true, 'skipped', 'rolled back, then skipped'); +INSERT INTO test VALUES (1); +COMMIT; +}); + +# Wait until the conflict disables the subscription. +$sub->poll_query_until('postgres', + q{SELECT subenabled = FALSE FROM pg_subscription WHERE subname = 'sub'}); + +# The hook did run before the transaction failed. +ok( $sub->log_contains( + qr/LOG[^\n]+received message: [^\n]+prefix: skipped/, + $log_location), + "hook is called for a message in a transaction that later fails"); + +# ... but its work was rolled back with the transaction. +is(logged_count('skipped'), 0, + "work done by the hook is rolled back with the remote transaction"); + +# Get the finish LSN of the failed transaction. +my $contents = slurp_file($sub->logfile, $log_location); +$contents =~ + qr/conflict detected on relation "public.test".*\n.*DETAIL:.*Could not apply remote change.*\n.*Key already exists in unique index "test_pkey", modified in transaction \d+: key .*, local row .*\n.*CONTEXT:.* for replication target relation "public.test" in transaction \d+, finished at ([[:xdigit:]]+\/[[:xdigit:]]+)/m + or die "could not get error-LSN"; +my $lsn = $1; + +$log_location = -s $sub->logfile; + +# Set skip LSN and re-enable the subscription. +$sub->safe_psql('postgres', qq{ALTER SUBSCRIPTION sub SKIP (lsn = '$lsn')}); +$sub->safe_psql('postgres', 'ALTER SUBSCRIPTION sub ENABLE'); + +# Wait for the failed transaction to be skipped. +$sub->poll_query_until('postgres', + q{SELECT subskiplsn = '0/0' FROM pg_subscription WHERE subname = 'sub'}); + +ok( !$sub->log_contains( + qr/LOG[^\n]+received message: [^\n]+prefix: skipped/, + $log_location), + "hook is not called for a message in a skipped transaction"); +is(logged_count('skipped'), 0, + "message in a skipped transaction is not applied"); + +# Verify that a message in a prepared transaction is applied when the +# transaction is committed, and discarded when it is rolled back. +$sub->safe_psql('postgres', 'ALTER SUBSCRIPTION sub DISABLE'); +$sub->safe_psql('postgres', 'ALTER SUBSCRIPTION sub SET (two_phase = on)'); +$sub->safe_psql('postgres', 'ALTER SUBSCRIPTION sub ENABLE'); +$sub->poll_query_until('postgres', + q{SELECT subtwophasestate = 'e' FROM pg_subscription WHERE subname = 'sub'} +); + +$pub->safe_psql( + 'postgres', q{ +BEGIN; +SELECT pg_logical_emit_message(true, 'prepared', 'committed later'); +PREPARE TRANSACTION 'msg_commit'; +}); +$pub->wait_for_catchup('sub'); + +is(logged_count('prepared'), + 0, "message in a prepared transaction is not applied before commit"); + +$pub->safe_psql('postgres', q{COMMIT PREPARED 'msg_commit'}); +$pub->wait_for_catchup('sub'); + +is(logged_count('prepared'), + 1, "message in a prepared transaction is applied at commit"); + +$pub->safe_psql( + 'postgres', q{ +BEGIN; +SELECT pg_logical_emit_message(true, 'prepared_rollback', 'rolled back later'); +PREPARE TRANSACTION 'msg_rollback'; +}); +$pub->wait_for_catchup('sub'); +$pub->safe_psql('postgres', q{ROLLBACK PREPARED 'msg_rollback'}); +$pub->wait_for_catchup('sub'); + +is(logged_count('prepared_rollback'), + 0, "message in a prepared transaction is discarded at rollback"); + +# Verify that no message reaches the hook once the option is turned off, which +# is also its default. +$sub->safe_psql('postgres', 'ALTER SUBSCRIPTION sub SET (messages = false)'); + +# Reconnect so that the change takes effect immediately. +$sub->safe_psql('postgres', 'ALTER SUBSCRIPTION sub DISABLE'); +$sub->safe_psql('postgres', 'ALTER SUBSCRIPTION sub ENABLE'); + +$pub->safe_psql('postgres', + q{SELECT pg_logical_emit_message(true, 'not_requested', 'messages is off')} +); +$pub->wait_for_catchup('sub'); + +is(logged_count('not_requested'), + 0, "no message is passed to the hook when messages is off"); + +$sub->stop; +$pub->stop; +done_testing(); diff --git a/src/test/modules/test_logicalmsg_hooks/test_logicalmsg_hooks.c b/src/test/modules/test_logicalmsg_hooks/test_logicalmsg_hooks.c new file mode 100644 index 00000000000..d8d6fc7c619 --- /dev/null +++ b/src/test/modules/test_logicalmsg_hooks/test_logicalmsg_hooks.c @@ -0,0 +1,140 @@ +/*-------------------------------------------------------------------------- + * + * test_logicalmsg_hooks.c + * Code for testing LogicalRepMessageHandle_hook + * + * The handler records every logical decoding message it receives into the + * table public.test_logicalmsg_log, so that tests can assert on the contents + * of a table rather than on the server log. Recording the messages this way + * also exercises the transactional behavior of the hook: a transactional + * message is recorded as part of the remote transaction that emitted it, and + * therefore disappears together with it if that transaction is rolled back or + * skipped, while a non-transactional message is recorded independently. + * + * The handler creates that table itself on the first message, so a test does + * not have to set it up. For a transactional message the creation is part of + * the remote transaction too, and is rolled back with it, so the table does + * not exist until some message has been applied successfully; a test must not + * query it before then. + * + * The messages are also logged, which lets a test tell "the handler ran but + * its work was rolled back" apart from "the handler was never called". + * + * Portions Copyright (c) 1996-2026, PostgreSQL Global Development Group + * Portions Copyright (c) 1994, Regents of the University of California + * + * IDENTIFICATION + * src/test/modules/test_logicalmsg_hooks/test_logicalmsg_hooks.c + * + * ------------------------------------------------------------------------- + */ + +#include "postgres.h" + +#include "catalog/namespace.h" +#include "catalog/pg_type.h" +#include "executor/spi.h" +#include "replication/logicalproto.h" +#include "replication/logicalworker.h" +#include "utils/builtins.h" +#include "utils/lsyscache.h" +#include "utils/pg_lsn.h" + +PG_MODULE_MAGIC; + +#define LOG_SCHEMA "public" +#define LOG_TABLE "test_logicalmsg_log" + +static LogicalRepMessageHandle_hook_type prev_logical_message_handler = NULL; +static void test_logical_message_handler(const LogicalRepMessageData *msg); + +/* + * Module load callback + */ +void +_PG_init(void) +{ + prev_logical_message_handler = LogicalRepMessageHandle_hook; + LogicalRepMessageHandle_hook = &test_logical_message_handler; +} + +/* + * Create the log table if not exists. + */ +static void +ensure_log_table_exists(void) +{ + Oid relid; + int ret; + + relid = get_relname_relid(LOG_TABLE, get_namespace_oid(LOG_SCHEMA, true)); + if (OidIsValid(relid)) + return; + + SPI_connect(); + + ret = SPI_execute("CREATE TABLE " LOG_SCHEMA "." LOG_TABLE + "(lsn pg_lsn, transactional boolean, prefix text, message_size int, message text)", + false, 0); + if (ret != SPI_OK_UTILITY) + elog(ERROR, "could not create " LOG_SCHEMA "." LOG_TABLE); + + SPI_finish(); +} + +static void +test_logical_message_handler(const LogicalRepMessageData *msg) +{ + Oid argtypes[5] = {LSNOID, BOOLOID, TEXTOID, INT4OID, TEXTOID}; + Datum values[5]; + int ret; + + /* + * Give a handler installed before ours its turn, so that several + * extensions can act on the same message. + */ + if (prev_logical_message_handler) + (*prev_logical_message_handler) (msg); + + ereport(LOG, + (errmsg("received message: LSN %X/%08X, prefix: %s, size: %zu, transactional: %d", + LSN_FORMAT_ARGS(msg->lsn), msg->prefix, + msg->message_size, msg->transactional))); + + ensure_log_table_exists(); + + /* + * Pass the message as query parameters rather than interpolating it into + * the query text. Message contents come from the publisher and are not to + * be trusted, and the payload may contain characters that would need + * quoting. Note also that message_size, not strlen(), is authoritative + * for the payload length. + * + * The payload is recorded as text because the accompanying test only ever + * emits text messages. A handler must not assume that in general: a + * payload is an arbitrary string of bytes, which may contain embedded + * nulls and bytes that are not valid in the subscriber's encoding. See + * the comments on LogicalRepMessageHandle_hook_type. + */ + values[0] = LSNGetDatum(msg->lsn); + values[1] = BoolGetDatum(msg->transactional); + values[2] = CStringGetTextDatum(msg->prefix); + values[3] = Int32GetDatum((int32) msg->message_size); + values[4] = PointerGetDatum(cstring_to_text_with_len(msg->message, + msg->message_size)); + + SPI_connect(); + + /* + * The apply worker runs with an empty search_path, so the table name must + * be schema-qualified. + */ + ret = SPI_execute_with_args("INSERT INTO " LOG_SCHEMA "." LOG_TABLE + " (lsn, transactional, prefix, message_size, message)" + " VALUES ($1, $2, $3, $4, $5)", + 5, argtypes, values, NULL, false, 0); + if (ret != SPI_OK_INSERT) + elog(ERROR, "could not record logical message: SPI returned %d", ret); + + SPI_finish(); +} diff --git a/src/test/regress/expected/subscription.out b/src/test/regress/expected/subscription.out index 6b2825a6a14..118e1ee3ce8 100644 --- a/src/test/regress/expected/subscription.out +++ b/src/test/regress/expected/subscription.out @@ -139,18 +139,18 @@ CREATE SUBSCRIPTION regress_testsub4 CONNECTION 'dbname=regress_doesnotexist' PU WARNING: subscription was created, but is not connected HINT: To initiate replication, you must manually create the replication slot, enable the subscription, and alter the subscription to refresh publications. \dRs+ regress_testsub4 - List of subscriptions - Name | Owner | Enabled | Publication | Binary | Streaming | Two-phase commit | Disable on error | Origin | Password required | Run as owner? | Failover | Server | Retain dead tuples | Max retention duration | Retention active | Synchronous commit | Conninfo | Receiver timeout | Skip LSN | Description -------------------+---------------------------+---------+-------------+--------+-----------+------------------+------------------+--------+-------------------+---------------+----------+--------+--------------------+------------------------+------------------+--------------------+-----------------------------+------------------+------------+------------- - regress_testsub4 | regress_subscription_user | f | {testpub} | f | parallel | d | f | none | t | f | f | | f | 0 | f | off | dbname=regress_doesnotexist | -1 | 0/00000000 | + List of subscriptions + Name | Owner | Enabled | Publication | Binary | Streaming | Two-phase commit | Disable on error | Origin | Password required | Run as owner? | Failover | Server | Retain dead tuples | Max retention duration | Retention active | Messages | Synchronous commit | Conninfo | Receiver timeout | Skip LSN | Description +------------------+---------------------------+---------+-------------+--------+-----------+------------------+------------------+--------+-------------------+---------------+----------+--------+--------------------+------------------------+------------------+----------+--------------------+-----------------------------+------------------+------------+------------- + regress_testsub4 | regress_subscription_user | f | {testpub} | f | parallel | d | f | none | t | f | f | | f | 0 | f | f | off | dbname=regress_doesnotexist | -1 | 0/00000000 | (1 row) ALTER SUBSCRIPTION regress_testsub4 SET (origin = any); \dRs+ regress_testsub4 - List of subscriptions - Name | Owner | Enabled | Publication | Binary | Streaming | Two-phase commit | Disable on error | Origin | Password required | Run as owner? | Failover | Server | Retain dead tuples | Max retention duration | Retention active | Synchronous commit | Conninfo | Receiver timeout | Skip LSN | Description -------------------+---------------------------+---------+-------------+--------+-----------+------------------+------------------+--------+-------------------+---------------+----------+--------+--------------------+------------------------+------------------+--------------------+-----------------------------+------------------+------------+------------- - regress_testsub4 | regress_subscription_user | f | {testpub} | f | parallel | d | f | any | t | f | f | | f | 0 | f | off | dbname=regress_doesnotexist | -1 | 0/00000000 | + List of subscriptions + Name | Owner | Enabled | Publication | Binary | Streaming | Two-phase commit | Disable on error | Origin | Password required | Run as owner? | Failover | Server | Retain dead tuples | Max retention duration | Retention active | Messages | Synchronous commit | Conninfo | Receiver timeout | Skip LSN | Description +------------------+---------------------------+---------+-------------+--------+-----------+------------------+------------------+--------+-------------------+---------------+----------+--------+--------------------+------------------------+------------------+----------+--------------------+-----------------------------+------------------+------------+------------- + regress_testsub4 | regress_subscription_user | f | {testpub} | f | parallel | d | f | any | t | f | f | | f | 0 | f | f | off | dbname=regress_doesnotexist | -1 | 0/00000000 | (1 row) DROP SUBSCRIPTION regress_testsub3; @@ -277,10 +277,10 @@ ALTER SUBSCRIPTION regress_testsub CONNECTION 'foobar'; ERROR: invalid connection string syntax: missing "=" after "foobar" in connection info string \dRs+ - List of subscriptions - Name | Owner | Enabled | Publication | Binary | Streaming | Two-phase commit | Disable on error | Origin | Password required | Run as owner? | Failover | Server | Retain dead tuples | Max retention duration | Retention active | Synchronous commit | Conninfo | Receiver timeout | Skip LSN | Description ------------------+---------------------------+---------+-------------+--------+-----------+------------------+------------------+--------+-------------------+---------------+----------+--------+--------------------+------------------------+------------------+--------------------+-----------------------------+------------------+------------+------------------- - regress_testsub | regress_subscription_user | f | {testpub} | f | parallel | d | f | any | t | f | f | | f | 0 | f | off | dbname=regress_doesnotexist | -1 | 0/00000000 | test subscription + List of subscriptions + Name | Owner | Enabled | Publication | Binary | Streaming | Two-phase commit | Disable on error | Origin | Password required | Run as owner? | Failover | Server | Retain dead tuples | Max retention duration | Retention active | Messages | Synchronous commit | Conninfo | Receiver timeout | Skip LSN | Description +-----------------+---------------------------+---------+-------------+--------+-----------+------------------+------------------+--------+-------------------+---------------+----------+--------+--------------------+------------------------+------------------+----------+--------------------+-----------------------------+------------------+------------+------------------- + regress_testsub | regress_subscription_user | f | {testpub} | f | parallel | d | f | any | t | f | f | | f | 0 | f | f | off | dbname=regress_doesnotexist | -1 | 0/00000000 | test subscription (1 row) ALTER SUBSCRIPTION regress_testsub SET PUBLICATION testpub2, testpub3 WITH (refresh = false); @@ -289,10 +289,10 @@ ALTER SUBSCRIPTION regress_testsub SET (slot_name = 'newname'); ALTER SUBSCRIPTION regress_testsub SET (password_required = false); ALTER SUBSCRIPTION regress_testsub SET (run_as_owner = true); \dRs+ - List of subscriptions - Name | Owner | Enabled | Publication | Binary | Streaming | Two-phase commit | Disable on error | Origin | Password required | Run as owner? | Failover | Server | Retain dead tuples | Max retention duration | Retention active | Synchronous commit | Conninfo | Receiver timeout | Skip LSN | Description ------------------+---------------------------+---------+---------------------+--------+-----------+------------------+------------------+--------+-------------------+---------------+----------+--------+--------------------+------------------------+------------------+--------------------+------------------------------+------------------+------------+------------------- - regress_testsub | regress_subscription_user | f | {testpub2,testpub3} | f | parallel | d | f | any | f | t | f | | f | 0 | f | off | dbname=regress_doesnotexist2 | -1 | 0/00000000 | test subscription + List of subscriptions + Name | Owner | Enabled | Publication | Binary | Streaming | Two-phase commit | Disable on error | Origin | Password required | Run as owner? | Failover | Server | Retain dead tuples | Max retention duration | Retention active | Messages | Synchronous commit | Conninfo | Receiver timeout | Skip LSN | Description +-----------------+---------------------------+---------+---------------------+--------+-----------+------------------+------------------+--------+-------------------+---------------+----------+--------+--------------------+------------------------+------------------+----------+--------------------+------------------------------+------------------+------------+------------------- + regress_testsub | regress_subscription_user | f | {testpub2,testpub3} | f | parallel | d | f | any | f | t | f | | f | 0 | f | f | off | dbname=regress_doesnotexist2 | -1 | 0/00000000 | test subscription (1 row) ALTER SUBSCRIPTION regress_testsub SET (password_required = true); @@ -308,10 +308,10 @@ ERROR: unrecognized subscription parameter: "create_slot" -- ok ALTER SUBSCRIPTION regress_testsub SKIP (lsn = '0/12345'); \dRs+ - List of subscriptions - Name | Owner | Enabled | Publication | Binary | Streaming | Two-phase commit | Disable on error | Origin | Password required | Run as owner? | Failover | Server | Retain dead tuples | Max retention duration | Retention active | Synchronous commit | Conninfo | Receiver timeout | Skip LSN | Description ------------------+---------------------------+---------+---------------------+--------+-----------+------------------+------------------+--------+-------------------+---------------+----------+--------+--------------------+------------------------+------------------+--------------------+------------------------------+------------------+------------+------------------- - regress_testsub | regress_subscription_user | f | {testpub2,testpub3} | f | parallel | d | f | any | t | f | f | | f | 0 | f | off | dbname=regress_doesnotexist2 | -1 | 0/00012345 | test subscription + List of subscriptions + Name | Owner | Enabled | Publication | Binary | Streaming | Two-phase commit | Disable on error | Origin | Password required | Run as owner? | Failover | Server | Retain dead tuples | Max retention duration | Retention active | Messages | Synchronous commit | Conninfo | Receiver timeout | Skip LSN | Description +-----------------+---------------------------+---------+---------------------+--------+-----------+------------------+------------------+--------+-------------------+---------------+----------+--------+--------------------+------------------------+------------------+----------+--------------------+------------------------------+------------------+------------+------------------- + regress_testsub | regress_subscription_user | f | {testpub2,testpub3} | f | parallel | d | f | any | t | f | f | | f | 0 | f | f | off | dbname=regress_doesnotexist2 | -1 | 0/00012345 | test subscription (1 row) -- ok - with lsn = NONE @@ -320,10 +320,10 @@ ALTER SUBSCRIPTION regress_testsub SKIP (lsn = NONE); ALTER SUBSCRIPTION regress_testsub SKIP (lsn = '0/0'); ERROR: invalid WAL location (LSN): 0/0 \dRs+ - List of subscriptions - Name | Owner | Enabled | Publication | Binary | Streaming | Two-phase commit | Disable on error | Origin | Password required | Run as owner? | Failover | Server | Retain dead tuples | Max retention duration | Retention active | Synchronous commit | Conninfo | Receiver timeout | Skip LSN | Description ------------------+---------------------------+---------+---------------------+--------+-----------+------------------+------------------+--------+-------------------+---------------+----------+--------+--------------------+------------------------+------------------+--------------------+------------------------------+------------------+------------+------------------- - regress_testsub | regress_subscription_user | f | {testpub2,testpub3} | f | parallel | d | f | any | t | f | f | | f | 0 | f | off | dbname=regress_doesnotexist2 | -1 | 0/00000000 | test subscription + List of subscriptions + Name | Owner | Enabled | Publication | Binary | Streaming | Two-phase commit | Disable on error | Origin | Password required | Run as owner? | Failover | Server | Retain dead tuples | Max retention duration | Retention active | Messages | Synchronous commit | Conninfo | Receiver timeout | Skip LSN | Description +-----------------+---------------------------+---------+---------------------+--------+-----------+------------------+------------------+--------+-------------------+---------------+----------+--------+--------------------+------------------------+------------------+----------+--------------------+------------------------------+------------------+------------+------------------- + regress_testsub | regress_subscription_user | f | {testpub2,testpub3} | f | parallel | d | f | any | t | f | f | | f | 0 | f | f | off | dbname=regress_doesnotexist2 | -1 | 0/00000000 | test subscription (1 row) BEGIN; @@ -359,10 +359,10 @@ ALTER SUBSCRIPTION regress_testsub_foo SET (wal_receiver_timeout = '80s'); ALTER SUBSCRIPTION regress_testsub_foo SET (wal_receiver_timeout = 'foobar'); ERROR: invalid value for parameter "wal_receiver_timeout": "foobar" \dRs+ - List of subscriptions - Name | Owner | Enabled | Publication | Binary | Streaming | Two-phase commit | Disable on error | Origin | Password required | Run as owner? | Failover | Server | Retain dead tuples | Max retention duration | Retention active | Synchronous commit | Conninfo | Receiver timeout | Skip LSN | Description ----------------------+---------------------------+---------+---------------------+--------+-----------+------------------+------------------+--------+-------------------+---------------+----------+--------+--------------------+------------------------+------------------+--------------------+------------------------------+------------------+------------+------------------- - regress_testsub_foo | regress_subscription_user | f | {testpub2,testpub3} | f | parallel | d | f | any | t | f | f | | f | 0 | f | local | dbname=regress_doesnotexist2 | 80s | 0/00000000 | test subscription + List of subscriptions + Name | Owner | Enabled | Publication | Binary | Streaming | Two-phase commit | Disable on error | Origin | Password required | Run as owner? | Failover | Server | Retain dead tuples | Max retention duration | Retention active | Messages | Synchronous commit | Conninfo | Receiver timeout | Skip LSN | Description +---------------------+---------------------------+---------+---------------------+--------+-----------+------------------+------------------+--------+-------------------+---------------+----------+--------+--------------------+------------------------+------------------+----------+--------------------+------------------------------+------------------+------------+------------------- + regress_testsub_foo | regress_subscription_user | f | {testpub2,testpub3} | f | parallel | d | f | any | t | f | f | | f | 0 | f | f | local | dbname=regress_doesnotexist2 | 80s | 0/00000000 | test subscription (1 row) -- rename back to keep the rest simple @@ -391,19 +391,19 @@ CREATE SUBSCRIPTION regress_testsub CONNECTION 'dbname=regress_doesnotexist' PUB WARNING: subscription was created, but is not connected HINT: To initiate replication, you must manually create the replication slot, enable the subscription, and alter the subscription to refresh publications. \dRs+ - List of subscriptions - Name | Owner | Enabled | Publication | Binary | Streaming | Two-phase commit | Disable on error | Origin | Password required | Run as owner? | Failover | Server | Retain dead tuples | Max retention duration | Retention active | Synchronous commit | Conninfo | Receiver timeout | Skip LSN | Description ------------------+---------------------------+---------+-------------+--------+-----------+------------------+------------------+--------+-------------------+---------------+----------+--------+--------------------+------------------------+------------------+--------------------+-----------------------------+------------------+------------+------------- - regress_testsub | regress_subscription_user | f | {testpub} | t | parallel | d | f | any | t | f | f | | f | 0 | f | off | dbname=regress_doesnotexist | -1 | 0/00000000 | + List of subscriptions + Name | Owner | Enabled | Publication | Binary | Streaming | Two-phase commit | Disable on error | Origin | Password required | Run as owner? | Failover | Server | Retain dead tuples | Max retention duration | Retention active | Messages | Synchronous commit | Conninfo | Receiver timeout | Skip LSN | Description +-----------------+---------------------------+---------+-------------+--------+-----------+------------------+------------------+--------+-------------------+---------------+----------+--------+--------------------+------------------------+------------------+----------+--------------------+-----------------------------+------------------+------------+------------- + regress_testsub | regress_subscription_user | f | {testpub} | t | parallel | d | f | any | t | f | f | | f | 0 | f | f | off | dbname=regress_doesnotexist | -1 | 0/00000000 | (1 row) ALTER SUBSCRIPTION regress_testsub SET (binary = false); ALTER SUBSCRIPTION regress_testsub SET (slot_name = NONE); \dRs+ - List of subscriptions - Name | Owner | Enabled | Publication | Binary | Streaming | Two-phase commit | Disable on error | Origin | Password required | Run as owner? | Failover | Server | Retain dead tuples | Max retention duration | Retention active | Synchronous commit | Conninfo | Receiver timeout | Skip LSN | Description ------------------+---------------------------+---------+-------------+--------+-----------+------------------+------------------+--------+-------------------+---------------+----------+--------+--------------------+------------------------+------------------+--------------------+-----------------------------+------------------+------------+------------- - regress_testsub | regress_subscription_user | f | {testpub} | f | parallel | d | f | any | t | f | f | | f | 0 | f | off | dbname=regress_doesnotexist | -1 | 0/00000000 | + List of subscriptions + Name | Owner | Enabled | Publication | Binary | Streaming | Two-phase commit | Disable on error | Origin | Password required | Run as owner? | Failover | Server | Retain dead tuples | Max retention duration | Retention active | Messages | Synchronous commit | Conninfo | Receiver timeout | Skip LSN | Description +-----------------+---------------------------+---------+-------------+--------+-----------+------------------+------------------+--------+-------------------+---------------+----------+--------+--------------------+------------------------+------------------+----------+--------------------+-----------------------------+------------------+------------+------------- + regress_testsub | regress_subscription_user | f | {testpub} | f | parallel | d | f | any | t | f | f | | f | 0 | f | f | off | dbname=regress_doesnotexist | -1 | 0/00000000 | (1 row) DROP SUBSCRIPTION regress_testsub; @@ -415,27 +415,27 @@ CREATE SUBSCRIPTION regress_testsub CONNECTION 'dbname=regress_doesnotexist' PUB WARNING: subscription was created, but is not connected HINT: To initiate replication, you must manually create the replication slot, enable the subscription, and alter the subscription to refresh publications. \dRs+ - List of subscriptions - Name | Owner | Enabled | Publication | Binary | Streaming | Two-phase commit | Disable on error | Origin | Password required | Run as owner? | Failover | Server | Retain dead tuples | Max retention duration | Retention active | Synchronous commit | Conninfo | Receiver timeout | Skip LSN | Description ------------------+---------------------------+---------+-------------+--------+-----------+------------------+------------------+--------+-------------------+---------------+----------+--------+--------------------+------------------------+------------------+--------------------+-----------------------------+------------------+------------+------------- - regress_testsub | regress_subscription_user | f | {testpub} | f | on | d | f | any | t | f | f | | f | 0 | f | off | dbname=regress_doesnotexist | -1 | 0/00000000 | + List of subscriptions + Name | Owner | Enabled | Publication | Binary | Streaming | Two-phase commit | Disable on error | Origin | Password required | Run as owner? | Failover | Server | Retain dead tuples | Max retention duration | Retention active | Messages | Synchronous commit | Conninfo | Receiver timeout | Skip LSN | Description +-----------------+---------------------------+---------+-------------+--------+-----------+------------------+------------------+--------+-------------------+---------------+----------+--------+--------------------+------------------------+------------------+----------+--------------------+-----------------------------+------------------+------------+------------- + regress_testsub | regress_subscription_user | f | {testpub} | f | on | d | f | any | t | f | f | | f | 0 | f | f | off | dbname=regress_doesnotexist | -1 | 0/00000000 | (1 row) ALTER SUBSCRIPTION regress_testsub SET (streaming = parallel); \dRs+ - List of subscriptions - Name | Owner | Enabled | Publication | Binary | Streaming | Two-phase commit | Disable on error | Origin | Password required | Run as owner? | Failover | Server | Retain dead tuples | Max retention duration | Retention active | Synchronous commit | Conninfo | Receiver timeout | Skip LSN | Description ------------------+---------------------------+---------+-------------+--------+-----------+------------------+------------------+--------+-------------------+---------------+----------+--------+--------------------+------------------------+------------------+--------------------+-----------------------------+------------------+------------+------------- - regress_testsub | regress_subscription_user | f | {testpub} | f | parallel | d | f | any | t | f | f | | f | 0 | f | off | dbname=regress_doesnotexist | -1 | 0/00000000 | + List of subscriptions + Name | Owner | Enabled | Publication | Binary | Streaming | Two-phase commit | Disable on error | Origin | Password required | Run as owner? | Failover | Server | Retain dead tuples | Max retention duration | Retention active | Messages | Synchronous commit | Conninfo | Receiver timeout | Skip LSN | Description +-----------------+---------------------------+---------+-------------+--------+-----------+------------------+------------------+--------+-------------------+---------------+----------+--------+--------------------+------------------------+------------------+----------+--------------------+-----------------------------+------------------+------------+------------- + regress_testsub | regress_subscription_user | f | {testpub} | f | parallel | d | f | any | t | f | f | | f | 0 | f | f | off | dbname=regress_doesnotexist | -1 | 0/00000000 | (1 row) ALTER SUBSCRIPTION regress_testsub SET (streaming = false); ALTER SUBSCRIPTION regress_testsub SET (slot_name = NONE); \dRs+ - List of subscriptions - Name | Owner | Enabled | Publication | Binary | Streaming | Two-phase commit | Disable on error | Origin | Password required | Run as owner? | Failover | Server | Retain dead tuples | Max retention duration | Retention active | Synchronous commit | Conninfo | Receiver timeout | Skip LSN | Description ------------------+---------------------------+---------+-------------+--------+-----------+------------------+------------------+--------+-------------------+---------------+----------+--------+--------------------+------------------------+------------------+--------------------+-----------------------------+------------------+------------+------------- - regress_testsub | regress_subscription_user | f | {testpub} | f | off | d | f | any | t | f | f | | f | 0 | f | off | dbname=regress_doesnotexist | -1 | 0/00000000 | + List of subscriptions + Name | Owner | Enabled | Publication | Binary | Streaming | Two-phase commit | Disable on error | Origin | Password required | Run as owner? | Failover | Server | Retain dead tuples | Max retention duration | Retention active | Messages | Synchronous commit | Conninfo | Receiver timeout | Skip LSN | Description +-----------------+---------------------------+---------+-------------+--------+-----------+------------------+------------------+--------+-------------------+---------------+----------+--------+--------------------+------------------------+------------------+----------+--------------------+-----------------------------+------------------+------------+------------- + regress_testsub | regress_subscription_user | f | {testpub} | f | off | d | f | any | t | f | f | | f | 0 | f | f | off | dbname=regress_doesnotexist | -1 | 0/00000000 | (1 row) -- fail - publication already exists @@ -450,10 +450,10 @@ ALTER SUBSCRIPTION regress_testsub ADD PUBLICATION testpub1, testpub2 WITH (refr ALTER SUBSCRIPTION regress_testsub ADD PUBLICATION testpub1, testpub2 WITH (refresh = false); ERROR: publication "testpub1" is already in subscription "regress_testsub" \dRs+ - List of subscriptions - Name | Owner | Enabled | Publication | Binary | Streaming | Two-phase commit | Disable on error | Origin | Password required | Run as owner? | Failover | Server | Retain dead tuples | Max retention duration | Retention active | Synchronous commit | Conninfo | Receiver timeout | Skip LSN | Description ------------------+---------------------------+---------+-----------------------------+--------+-----------+------------------+------------------+--------+-------------------+---------------+----------+--------+--------------------+------------------------+------------------+--------------------+-----------------------------+------------------+------------+------------- - regress_testsub | regress_subscription_user | f | {testpub,testpub1,testpub2} | f | off | d | f | any | t | f | f | | f | 0 | f | off | dbname=regress_doesnotexist | -1 | 0/00000000 | + List of subscriptions + Name | Owner | Enabled | Publication | Binary | Streaming | Two-phase commit | Disable on error | Origin | Password required | Run as owner? | Failover | Server | Retain dead tuples | Max retention duration | Retention active | Messages | Synchronous commit | Conninfo | Receiver timeout | Skip LSN | Description +-----------------+---------------------------+---------+-----------------------------+--------+-----------+------------------+------------------+--------+-------------------+---------------+----------+--------+--------------------+------------------------+------------------+----------+--------------------+-----------------------------+------------------+------------+------------- + regress_testsub | regress_subscription_user | f | {testpub,testpub1,testpub2} | f | off | d | f | any | t | f | f | | f | 0 | f | f | off | dbname=regress_doesnotexist | -1 | 0/00000000 | (1 row) -- fail - publication used more than once @@ -468,10 +468,10 @@ ERROR: publication "testpub3" is not in subscription "regress_testsub" -- ok - delete publications ALTER SUBSCRIPTION regress_testsub DROP PUBLICATION testpub1, testpub2 WITH (refresh = false); \dRs+ - List of subscriptions - Name | Owner | Enabled | Publication | Binary | Streaming | Two-phase commit | Disable on error | Origin | Password required | Run as owner? | Failover | Server | Retain dead tuples | Max retention duration | Retention active | Synchronous commit | Conninfo | Receiver timeout | Skip LSN | Description ------------------+---------------------------+---------+-------------+--------+-----------+------------------+------------------+--------+-------------------+---------------+----------+--------+--------------------+------------------------+------------------+--------------------+-----------------------------+------------------+------------+------------- - regress_testsub | regress_subscription_user | f | {testpub} | f | off | d | f | any | t | f | f | | f | 0 | f | off | dbname=regress_doesnotexist | -1 | 0/00000000 | + List of subscriptions + Name | Owner | Enabled | Publication | Binary | Streaming | Two-phase commit | Disable on error | Origin | Password required | Run as owner? | Failover | Server | Retain dead tuples | Max retention duration | Retention active | Messages | Synchronous commit | Conninfo | Receiver timeout | Skip LSN | Description +-----------------+---------------------------+---------+-------------+--------+-----------+------------------+------------------+--------+-------------------+---------------+----------+--------+--------------------+------------------------+------------------+----------+--------------------+-----------------------------+------------------+------------+------------- + regress_testsub | regress_subscription_user | f | {testpub} | f | off | d | f | any | t | f | f | | f | 0 | f | f | off | dbname=regress_doesnotexist | -1 | 0/00000000 | (1 row) DROP SUBSCRIPTION regress_testsub; @@ -507,19 +507,19 @@ CREATE SUBSCRIPTION regress_testsub CONNECTION 'dbname=regress_doesnotexist' PUB WARNING: subscription was created, but is not connected HINT: To initiate replication, you must manually create the replication slot, enable the subscription, and alter the subscription to refresh publications. \dRs+ - List of subscriptions - Name | Owner | Enabled | Publication | Binary | Streaming | Two-phase commit | Disable on error | Origin | Password required | Run as owner? | Failover | Server | Retain dead tuples | Max retention duration | Retention active | Synchronous commit | Conninfo | Receiver timeout | Skip LSN | Description ------------------+---------------------------+---------+-------------+--------+-----------+------------------+------------------+--------+-------------------+---------------+----------+--------+--------------------+------------------------+------------------+--------------------+-----------------------------+------------------+------------+------------- - regress_testsub | regress_subscription_user | f | {testpub} | f | parallel | p | f | any | t | f | f | | f | 0 | f | off | dbname=regress_doesnotexist | -1 | 0/00000000 | + List of subscriptions + Name | Owner | Enabled | Publication | Binary | Streaming | Two-phase commit | Disable on error | Origin | Password required | Run as owner? | Failover | Server | Retain dead tuples | Max retention duration | Retention active | Messages | Synchronous commit | Conninfo | Receiver timeout | Skip LSN | Description +-----------------+---------------------------+---------+-------------+--------+-----------+------------------+------------------+--------+-------------------+---------------+----------+--------+--------------------+------------------------+------------------+----------+--------------------+-----------------------------+------------------+------------+------------- + regress_testsub | regress_subscription_user | f | {testpub} | f | parallel | p | f | any | t | f | f | | f | 0 | f | f | off | dbname=regress_doesnotexist | -1 | 0/00000000 | (1 row) -- we can alter streaming when two_phase enabled ALTER SUBSCRIPTION regress_testsub SET (streaming = true); \dRs+ - List of subscriptions - Name | Owner | Enabled | Publication | Binary | Streaming | Two-phase commit | Disable on error | Origin | Password required | Run as owner? | Failover | Server | Retain dead tuples | Max retention duration | Retention active | Synchronous commit | Conninfo | Receiver timeout | Skip LSN | Description ------------------+---------------------------+---------+-------------+--------+-----------+------------------+------------------+--------+-------------------+---------------+----------+--------+--------------------+------------------------+------------------+--------------------+-----------------------------+------------------+------------+------------- - regress_testsub | regress_subscription_user | f | {testpub} | f | on | p | f | any | t | f | f | | f | 0 | f | off | dbname=regress_doesnotexist | -1 | 0/00000000 | + List of subscriptions + Name | Owner | Enabled | Publication | Binary | Streaming | Two-phase commit | Disable on error | Origin | Password required | Run as owner? | Failover | Server | Retain dead tuples | Max retention duration | Retention active | Messages | Synchronous commit | Conninfo | Receiver timeout | Skip LSN | Description +-----------------+---------------------------+---------+-------------+--------+-----------+------------------+------------------+--------+-------------------+---------------+----------+--------+--------------------+------------------------+------------------+----------+--------------------+-----------------------------+------------------+------------+------------- + regress_testsub | regress_subscription_user | f | {testpub} | f | on | p | f | any | t | f | f | | f | 0 | f | f | off | dbname=regress_doesnotexist | -1 | 0/00000000 | (1 row) ALTER SUBSCRIPTION regress_testsub SET (slot_name = NONE); @@ -529,10 +529,10 @@ CREATE SUBSCRIPTION regress_testsub CONNECTION 'dbname=regress_doesnotexist' PUB WARNING: subscription was created, but is not connected HINT: To initiate replication, you must manually create the replication slot, enable the subscription, and alter the subscription to refresh publications. \dRs+ - List of subscriptions - Name | Owner | Enabled | Publication | Binary | Streaming | Two-phase commit | Disable on error | Origin | Password required | Run as owner? | Failover | Server | Retain dead tuples | Max retention duration | Retention active | Synchronous commit | Conninfo | Receiver timeout | Skip LSN | Description ------------------+---------------------------+---------+-------------+--------+-----------+------------------+------------------+--------+-------------------+---------------+----------+--------+--------------------+------------------------+------------------+--------------------+-----------------------------+------------------+------------+------------- - regress_testsub | regress_subscription_user | f | {testpub} | f | on | p | f | any | t | f | f | | f | 0 | f | off | dbname=regress_doesnotexist | -1 | 0/00000000 | + List of subscriptions + Name | Owner | Enabled | Publication | Binary | Streaming | Two-phase commit | Disable on error | Origin | Password required | Run as owner? | Failover | Server | Retain dead tuples | Max retention duration | Retention active | Messages | Synchronous commit | Conninfo | Receiver timeout | Skip LSN | Description +-----------------+---------------------------+---------+-------------+--------+-----------+------------------+------------------+--------+-------------------+---------------+----------+--------+--------------------+------------------------+------------------+----------+--------------------+-----------------------------+------------------+------------+------------- + regress_testsub | regress_subscription_user | f | {testpub} | f | on | p | f | any | t | f | f | | f | 0 | f | f | off | dbname=regress_doesnotexist | -1 | 0/00000000 | (1 row) ALTER SUBSCRIPTION regress_testsub SET (slot_name = NONE); @@ -545,18 +545,18 @@ CREATE SUBSCRIPTION regress_testsub CONNECTION 'dbname=regress_doesnotexist' PUB WARNING: subscription was created, but is not connected HINT: To initiate replication, you must manually create the replication slot, enable the subscription, and alter the subscription to refresh publications. \dRs+ - List of subscriptions - Name | Owner | Enabled | Publication | Binary | Streaming | Two-phase commit | Disable on error | Origin | Password required | Run as owner? | Failover | Server | Retain dead tuples | Max retention duration | Retention active | Synchronous commit | Conninfo | Receiver timeout | Skip LSN | Description ------------------+---------------------------+---------+-------------+--------+-----------+------------------+------------------+--------+-------------------+---------------+----------+--------+--------------------+------------------------+------------------+--------------------+-----------------------------+------------------+------------+------------- - regress_testsub | regress_subscription_user | f | {testpub} | f | parallel | d | f | any | t | f | f | | f | 0 | f | off | dbname=regress_doesnotexist | -1 | 0/00000000 | + List of subscriptions + Name | Owner | Enabled | Publication | Binary | Streaming | Two-phase commit | Disable on error | Origin | Password required | Run as owner? | Failover | Server | Retain dead tuples | Max retention duration | Retention active | Messages | Synchronous commit | Conninfo | Receiver timeout | Skip LSN | Description +-----------------+---------------------------+---------+-------------+--------+-----------+------------------+------------------+--------+-------------------+---------------+----------+--------+--------------------+------------------------+------------------+----------+--------------------+-----------------------------+------------------+------------+------------- + regress_testsub | regress_subscription_user | f | {testpub} | f | parallel | d | f | any | t | f | f | | f | 0 | f | f | off | dbname=regress_doesnotexist | -1 | 0/00000000 | (1 row) ALTER SUBSCRIPTION regress_testsub SET (disable_on_error = true); \dRs+ - List of subscriptions - Name | Owner | Enabled | Publication | Binary | Streaming | Two-phase commit | Disable on error | Origin | Password required | Run as owner? | Failover | Server | Retain dead tuples | Max retention duration | Retention active | Synchronous commit | Conninfo | Receiver timeout | Skip LSN | Description ------------------+---------------------------+---------+-------------+--------+-----------+------------------+------------------+--------+-------------------+---------------+----------+--------+--------------------+------------------------+------------------+--------------------+-----------------------------+------------------+------------+------------- - regress_testsub | regress_subscription_user | f | {testpub} | f | parallel | d | t | any | t | f | f | | f | 0 | f | off | dbname=regress_doesnotexist | -1 | 0/00000000 | + List of subscriptions + Name | Owner | Enabled | Publication | Binary | Streaming | Two-phase commit | Disable on error | Origin | Password required | Run as owner? | Failover | Server | Retain dead tuples | Max retention duration | Retention active | Messages | Synchronous commit | Conninfo | Receiver timeout | Skip LSN | Description +-----------------+---------------------------+---------+-------------+--------+-----------+------------------+------------------+--------+-------------------+---------------+----------+--------+--------------------+------------------------+------------------+----------+--------------------+-----------------------------+------------------+------------+------------- + regress_testsub | regress_subscription_user | f | {testpub} | f | parallel | d | t | any | t | f | f | | f | 0 | f | f | off | dbname=regress_doesnotexist | -1 | 0/00000000 | (1 row) ALTER SUBSCRIPTION regress_testsub SET (slot_name = NONE); @@ -569,10 +569,10 @@ CREATE SUBSCRIPTION regress_testsub CONNECTION 'dbname=regress_doesnotexist' PUB WARNING: subscription was created, but is not connected HINT: To initiate replication, you must manually create the replication slot, enable the subscription, and alter the subscription to refresh publications. \dRs+ - List of subscriptions - Name | Owner | Enabled | Publication | Binary | Streaming | Two-phase commit | Disable on error | Origin | Password required | Run as owner? | Failover | Server | Retain dead tuples | Max retention duration | Retention active | Synchronous commit | Conninfo | Receiver timeout | Skip LSN | Description ------------------+---------------------------+---------+-------------+--------+-----------+------------------+------------------+--------+-------------------+---------------+----------+--------+--------------------+------------------------+------------------+--------------------+-----------------------------+------------------+------------+------------- - regress_testsub | regress_subscription_user | f | {testpub} | f | parallel | d | f | any | t | f | f | | f | 0 | f | off | dbname=regress_doesnotexist | -1 | 0/00000000 | + List of subscriptions + Name | Owner | Enabled | Publication | Binary | Streaming | Two-phase commit | Disable on error | Origin | Password required | Run as owner? | Failover | Server | Retain dead tuples | Max retention duration | Retention active | Messages | Synchronous commit | Conninfo | Receiver timeout | Skip LSN | Description +-----------------+---------------------------+---------+-------------+--------+-----------+------------------+------------------+--------+-------------------+---------------+----------+--------+--------------------+------------------------+------------------+----------+--------------------+-----------------------------+------------------+------------+------------- + regress_testsub | regress_subscription_user | f | {testpub} | f | parallel | d | f | any | t | f | f | | f | 0 | f | f | off | dbname=regress_doesnotexist | -1 | 0/00000000 | (1 row) ALTER SUBSCRIPTION regress_testsub SET (slot_name = NONE); @@ -589,10 +589,10 @@ NOTICE: max_retention_duration is ineffective when retain_dead_tuples is disabl WARNING: subscription was created, but is not connected HINT: To initiate replication, you must manually create the replication slot, enable the subscription, and alter the subscription to refresh publications. \dRs+ - List of subscriptions - Name | Owner | Enabled | Publication | Binary | Streaming | Two-phase commit | Disable on error | Origin | Password required | Run as owner? | Failover | Server | Retain dead tuples | Max retention duration | Retention active | Synchronous commit | Conninfo | Receiver timeout | Skip LSN | Description ------------------+---------------------------+---------+-------------+--------+-----------+------------------+------------------+--------+-------------------+---------------+----------+--------+--------------------+------------------------+------------------+--------------------+-----------------------------+------------------+------------+------------- - regress_testsub | regress_subscription_user | f | {testpub} | f | parallel | d | f | any | t | f | f | | f | 1000 | f | off | dbname=regress_doesnotexist | -1 | 0/00000000 | + List of subscriptions + Name | Owner | Enabled | Publication | Binary | Streaming | Two-phase commit | Disable on error | Origin | Password required | Run as owner? | Failover | Server | Retain dead tuples | Max retention duration | Retention active | Messages | Synchronous commit | Conninfo | Receiver timeout | Skip LSN | Description +-----------------+---------------------------+---------+-------------+--------+-----------+------------------+------------------+--------+-------------------+---------------+----------+--------+--------------------+------------------------+------------------+----------+--------------------+-----------------------------+------------------+------------+------------- + regress_testsub | regress_subscription_user | f | {testpub} | f | parallel | d | f | any | t | f | f | | f | 1000 | f | f | off | dbname=regress_doesnotexist | -1 | 0/00000000 | (1 row) -- fail - max_retention_duration must be non-negative @@ -601,12 +601,38 @@ ERROR: option "max_retention_duration" cannot be negative -- ok ALTER SUBSCRIPTION regress_testsub SET (max_retention_duration = 0); \dRs+ - List of subscriptions - Name | Owner | Enabled | Publication | Binary | Streaming | Two-phase commit | Disable on error | Origin | Password required | Run as owner? | Failover | Server | Retain dead tuples | Max retention duration | Retention active | Synchronous commit | Conninfo | Receiver timeout | Skip LSN | Description ------------------+---------------------------+---------+-------------+--------+-----------+------------------+------------------+--------+-------------------+---------------+----------+--------+--------------------+------------------------+------------------+--------------------+-----------------------------+------------------+------------+------------- - regress_testsub | regress_subscription_user | f | {testpub} | f | parallel | d | f | any | t | f | f | | f | 0 | f | off | dbname=regress_doesnotexist | -1 | 0/00000000 | + List of subscriptions + Name | Owner | Enabled | Publication | Binary | Streaming | Two-phase commit | Disable on error | Origin | Password required | Run as owner? | Failover | Server | Retain dead tuples | Max retention duration | Retention active | Messages | Synchronous commit | Conninfo | Receiver timeout | Skip LSN | Description +-----------------+---------------------------+---------+-------------+--------+-----------+------------------+------------------+--------+-------------------+---------------+----------+--------+--------------------+------------------------+------------------+----------+--------------------+-----------------------------+------------------+------------+------------- + regress_testsub | regress_subscription_user | f | {testpub} | f | parallel | d | f | any | t | f | f | | f | 0 | f | f | off | dbname=regress_doesnotexist | -1 | 0/00000000 | +(1 row) + +ALTER SUBSCRIPTION regress_testsub SET (slot_name = NONE); +DROP SUBSCRIPTION regress_testsub; +-- fail - messages must be boolean +CREATE SUBSCRIPTION regress_testsub CONNECTION 'dbname=regress_doesnotexist' PUBLICATION testpub WITH (connect = false, messages = foo); +ERROR: messages requires a Boolean value +-- ok +CREATE SUBSCRIPTION regress_testsub CONNECTION 'dbname=regress_doesnotexist' PUBLICATION testpub WITH (connect = false, messages = true); +WARNING: subscription was created, but is not connected +HINT: To initiate replication, you must manually create the replication slot, enable the subscription, and alter the subscription to refresh publications. +\dRs+ + List of subscriptions + Name | Owner | Enabled | Publication | Binary | Streaming | Two-phase commit | Disable on error | Origin | Password required | Run as owner? | Failover | Server | Retain dead tuples | Max retention duration | Retention active | Messages | Synchronous commit | Conninfo | Receiver timeout | Skip LSN | Description +-----------------+---------------------------+---------+-------------+--------+-----------+------------------+------------------+--------+-------------------+---------------+----------+--------+--------------------+------------------------+------------------+----------+--------------------+-----------------------------+------------------+------------+------------- + regress_testsub | regress_subscription_user | f | {testpub} | f | parallel | d | f | any | t | f | f | | f | 0 | f | t | off | dbname=regress_doesnotexist | -1 | 0/00000000 | +(1 row) + +-- ok +ALTER SUBSCRIPTION regress_testsub SET (messages = false); +\dRs+ + List of subscriptions + Name | Owner | Enabled | Publication | Binary | Streaming | Two-phase commit | Disable on error | Origin | Password required | Run as owner? | Failover | Server | Retain dead tuples | Max retention duration | Retention active | Messages | Synchronous commit | Conninfo | Receiver timeout | Skip LSN | Description +-----------------+---------------------------+---------+-------------+--------+-----------+------------------+------------------+--------+-------------------+---------------+----------+--------+--------------------+------------------------+------------------+----------+--------------------+-----------------------------+------------------+------------+------------- + regress_testsub | regress_subscription_user | f | {testpub} | f | parallel | d | f | any | t | f | f | | f | 0 | f | f | off | dbname=regress_doesnotexist | -1 | 0/00000000 | (1 row) +-- cleanup ALTER SUBSCRIPTION regress_testsub SET (slot_name = NONE); DROP SUBSCRIPTION regress_testsub; -- let's do some tests with pg_create_subscription rather than superuser diff --git a/src/test/regress/sql/subscription.sql b/src/test/regress/sql/subscription.sql index 1f34889a9a5..823b9318ead 100644 --- a/src/test/regress/sql/subscription.sql +++ b/src/test/regress/sql/subscription.sql @@ -457,6 +457,23 @@ ALTER SUBSCRIPTION regress_testsub SET (max_retention_duration = 0); ALTER SUBSCRIPTION regress_testsub SET (slot_name = NONE); DROP SUBSCRIPTION regress_testsub; +-- fail - messages must be boolean +CREATE SUBSCRIPTION regress_testsub CONNECTION 'dbname=regress_doesnotexist' PUBLICATION testpub WITH (connect = false, messages = foo); + +-- ok +CREATE SUBSCRIPTION regress_testsub CONNECTION 'dbname=regress_doesnotexist' PUBLICATION testpub WITH (connect = false, messages = true); + +\dRs+ + +-- ok +ALTER SUBSCRIPTION regress_testsub SET (messages = false); + +\dRs+ + +-- cleanup +ALTER SUBSCRIPTION regress_testsub SET (slot_name = NONE); +DROP SUBSCRIPTION regress_testsub; + -- let's do some tests with pg_create_subscription rather than superuser SET SESSION AUTHORIZATION regress_subscription_user3; diff --git a/src/tools/pgindent/typedefs.list b/src/tools/pgindent/typedefs.list index c21457c4887..179225ada82 100644 --- a/src/tools/pgindent/typedefs.list +++ b/src/tools/pgindent/typedefs.list @@ -1684,6 +1684,8 @@ LogicalRepBeginData LogicalRepCommitData LogicalRepCommitPreparedTxnData LogicalRepCtxStruct +LogicalRepMessageData +LogicalRepMessageHandle_hook_type LogicalRepMsgType LogicalRepPartMapEntry LogicalRepPreparedTxnData -- 2.56.0
^ permalink raw reply [nested|flat] 44+ messages in thread

* Re: Add a hook for handling logical decoding messages on subscribers.
@ 2026-11-11 22:40  Masahiko Sawada <sawada.mshk@gmail.com>
  parent: Masahiko Sawada <sawada.mshk@gmail.com>
  0 siblings, 1 reply; 44+ messages in thread

From: Masahiko Sawada @ 2026-11-11 22:40 UTC (permalink / raw)
  To: Amit Kapila <amit.kapila16@gmail.com>; +Cc: Zhijie Hou <houzhijie22@gmail.com>; Bharath Rupireddy <bharath.rupireddyforpostgres@gmail.com>; Chao Li <li.evan.chao@gmail.com>; Fujii Masao <masao.fujii@gmail.com>; pgsql-hackers

On Mon, Nov 9, 2026 at 5:21 PM Masahiko Sawada <sawada.mshk@gmail.com> wrote:
>
> On Mon, Nov 9, 2026 at 11:02 AM Masahiko Sawada <sawada.mshk@gmail.com> wrote:
> >
> > On Mon, Nov 2, 2026 at 4:47 AM Amit Kapila <amit.kapila16@gmail.com> wrote:
> > >
> > > On Sat, Oct 31, 2026 at 6:05 PM Zhijie Hou <houzhijie22@gmail.com> wrote:
> > > >
> > > > On Sat, Oct 31, 2026 at 06:34 AM Masahiko Sawada wrote:
> > > > >
> > > > > It seems to me worth adding a check for tablesync workers since both
> > > > > the apply workers and the tablesync workers call
> > > > > walrcv_startstreaming() and it would prevent mistakes by future
> > > > > changes.
> > > > >
> > > > > I've attached the updated patch.
> > > >
> > > > While reviewing how transactional logical messages interact with
> > > > subtransaction aborts, I noticed an issue with streamed transactions. In
> > > > non-streaming mode, it works as expected (i.e., the message is not
> > > > applied if the subtransaction is aborted). However, in streaming mode,
> > > > if a transactional message is emitted in a subtransaction that is later
> > > > aborted, the STREAM ABORT message does not discard or roll back that
> > > > logical message, so it still gets applied in both streaming=on and
> > > > streaming=parallel modes.
> > > >
> > > > I think the reason is that ReorderBufferApplyMessage() passes the
> > > > top-level txn rather than change->txn to rb->stream_message(), so
> > > > logical decoding always sends the top-level XID instead of the
> > > > subtransaction XID when outputting a streamed message. Thus the apply
> > > > worker cannot associate the message with the subtransaction and cannot
> > > > roll it back properly.
> > > >
> > > > Strictly speaking, this is an existing issue, but it becomes more
> > > > visible once subscribers can handle logical messages via this hook. So
> > > > it may be worth considering whether to fix it along with this patch or
> > > > separately.
> > > >
> > >
> > > Right, it didn't have any impact till now for native replication as we
> > > were just ignoring these messages on subscribers. I think this could
> > > have impacted output-plugins if they used this functionality. I think
> > > we can fix this separately but it is better to fix it before this
> > > patch as this patch can expose a hidden issue.
> >
> > Agreed that we should fix it before this patch.
> >
> > The Hou's patch looks good. I prefer to apply it to only master as
> > external output plugin might require changes similar to what the patch
> > does for test_decoding.
> >
> > And thank you for the 0002 patch. I'll include it in the next version patch.
>
> I've attached the new patches. I just updated the commit message for
> the 0001 patch, and added some fixes and regression tests for the 0002
> patch. Please review them.

I think the 0001 patch can be pushed independently. So I'm going to
push it to master, barring objections.

Regards,

-- 
Masahiko Sawada
Amazon Web Services: https://aws.amazon.com






^ permalink  raw  reply  [nested|flat] 44+ messages in thread

* Re: Add a hook for handling logical decoding messages on subscribers.
@ 2026-11-12 09:32  Amit Kapila <amit.kapila16@gmail.com>
  parent: Masahiko Sawada <sawada.mshk@gmail.com>
  0 siblings, 1 reply; 44+ messages in thread

From: Amit Kapila @ 2026-11-12 09:32 UTC (permalink / raw)
  To: Masahiko Sawada <sawada.mshk@gmail.com>; +Cc: Zhijie Hou <houzhijie22@gmail.com>; Bharath Rupireddy <bharath.rupireddyforpostgres@gmail.com>; Chao Li <li.evan.chao@gmail.com>; Fujii Masao <masao.fujii@gmail.com>; pgsql-hackers

On Thu, Nov 12, 2026 at 4:10 AM Masahiko Sawada <sawada.mshk@gmail.com> wrote:
>
> On Mon, Nov 9, 2026 at 5:21 PM Masahiko Sawada <sawada.mshk@gmail.com> wrote:
> >
> > I've attached the new patches. I just updated the commit message for
> > the 0001 patch, and added some fixes and regression tests for the 0002
> > patch. Please review them.
>
> I think the 0001 patch can be pushed independently. So I'm going to
> push it to master, barring objections.
>

Few comments on 0001:
* As the parameter to stream_message callback could be a
sub-transaction which is different from other similar callbacks like
truncate and change, it would be better to add that in docs where we
describe LogicalDecodeStreamMessageCB. The one more thing which we may
consider is how to get the top txn if required by callback as the
patch did in test_decoding.
* As we are planning this as a master-only change, will it be better
if we just pass subxid as a separate parameter to keep the interface
consistent with other similar interfaces and avoid the need for
plugins to retrieve toptxn separately.
* Similar to pg_decode_stream_message, shouldn't there be a call to
stream_start if (!txndata->stream_wrote_changes). This is just a code
observation, I haven't tested it.

-- 
With Regards,
Amit Kapila.






^ permalink  raw  reply  [nested|flat] 44+ messages in thread

* Re: Add a hook for handling logical decoding messages on subscribers.
@ 2026-11-12 10:06  Bertrand Drouvot <bertranddrouvot.pg@gmail.com>
  parent: Amit Kapila <amit.kapila16@gmail.com>
  0 siblings, 1 reply; 44+ messages in thread

From: Bertrand Drouvot @ 2026-11-12 10:06 UTC (permalink / raw)
  To: Amit Kapila <amit.kapila16@gmail.com>; +Cc: Masahiko Sawada <sawada.mshk@gmail.com>; Zhijie Hou <houzhijie22@gmail.com>; Bharath Rupireddy <bharath.rupireddyforpostgres@gmail.com>; Chao Li <li.evan.chao@gmail.com>; Fujii Masao <masao.fujii@gmail.com>; pgsql-hackers

Hi,

Sorry for joining this discussion so late.

On Thu, Nov 12, 2026 at 03:02:44PM +0530, Amit Kapila wrote:
> On Thu, Nov 12, 2026 at 4:10 AM Masahiko Sawada <sawada.mshk@gmail.com> wrote:
> >
> > On Mon, Nov 9, 2026 at 5:21 PM Masahiko Sawada <sawada.mshk@gmail.com> wrote:
> > >
> > > I've attached the new patches. I just updated the commit message for
> > > the 0001 patch, and added some fixes and regression tests for the 0002
> > > patch. Please review them.
> >
> > I think the 0001 patch can be pushed independently. So I'm going to
> > push it to master, barring objections.
> >
> 
> Few comments on 0001:
> * As we are planning this as a master-only change, will it be better
> if we just pass subxid as a separate parameter to keep the interface
> consistent with other similar interfaces and avoid the need for
> plugins to retrieve toptxn separately.

+1. That would also prevent the crash mentioned in the commit message, as txn
would remain the top level transaction.

Regards,

-- 
Bertrand Drouvot
PostgreSQL Contributors Team
RDS Open Source Databases
Amazon Web Services: https://aws.amazon.com






^ permalink  raw  reply  [nested|flat] 44+ messages in thread

* Re: Add a hook for handling logical decoding messages on subscribers.
@ 2026-11-13 23:35  Masahiko Sawada <sawada.mshk@gmail.com>
  parent: Bertrand Drouvot <bertranddrouvot.pg@gmail.com>
  0 siblings, 1 reply; 44+ messages in thread

From: Masahiko Sawada @ 2026-11-13 23:35 UTC (permalink / raw)
  To: Bertrand Drouvot <bertranddrouvot.pg@gmail.com>; +Cc: Amit Kapila <amit.kapila16@gmail.com>; Zhijie Hou <houzhijie22@gmail.com>; Bharath Rupireddy <bharath.rupireddyforpostgres@gmail.com>; Chao Li <li.evan.chao@gmail.com>; Fujii Masao <masao.fujii@gmail.com>; pgsql-hackers

On Thu, Nov 12, 2026 at 2:06 AM Bertrand Drouvot
<bertranddrouvot.pg@gmail.com> wrote:
>
> Hi,
>
> Sorry for joining this discussion so late.
>
> On Thu, Nov 12, 2026 at 03:02:44PM +0530, Amit Kapila wrote:
> > On Thu, Nov 12, 2026 at 4:10 AM Masahiko Sawada <sawada.mshk@gmail.com> wrote:
> > >
> > > On Mon, Nov 9, 2026 at 5:21 PM Masahiko Sawada <sawada.mshk@gmail.com> wrote:
> > > >
> > > > I've attached the new patches. I just updated the commit message for
> > > > the 0001 patch, and added some fixes and regression tests for the 0002
> > > > patch. Please review them.
> > >
> > > I think the 0001 patch can be pushed independently. So I'm going to
> > > push it to master, barring objections.
> > >
> >
> > Few comments on 0001:
> > * As we are planning this as a master-only change, will it be better
> > if we just pass subxid as a separate parameter to keep the interface
> > consistent with other similar interfaces and avoid the need for
> > plugins to retrieve toptxn separately.
>
> +1. That would also prevent the crash mentioned in the commit message, as txn
> would remain the top level transaction.
>
> Regards,

Agreed. It's better for callbacks to get both top-level and
subtransaction's ids.

I've updated the patches (the 0002 patch unchanged).

Regards,

-- 
Masahiko Sawada
Amazon Web Services: https://aws.amazon.com

Attachments: download all (2 files, .tar.gz)

v8-0002-Add-a-hook-for-handling-logical-messages-on-subsc.patch (text/x-patch, 121.3K) download From 3e913a370ca90ed708473e638b8445984d39441a Mon Sep 17 00:00:00 2001 From: Masahiko Sawada <sawada.mshk@gmail.com> Date: Mon, 8 Jun 2026 11:25:47 -0700 Subject: [PATCH v8 2/2] Add a hook for handling logical messages on subscribers. Previously, pgoutput could emit logical decoding messages, those written with pg_logical_emit_message(), but the built-in subscriber never requested them from the publisher, and apply_handle_message() was only a placeholder that discarded the MESSAGE message type. Acting on such messages therefore meant writing a separate logical decoding client, and giving up the apply worker's streaming, conflict handling and progress tracking. Logical messages are a general-purpose side channel carried in band with the data, useful for replicating deparsed DDL, for carrying change-data-capture context that the row data does not have, and for cache invalidation delivered in commit order relative to the rows it refers to. This commit adds a new "messages" subscription option, which asks the publisher for the pgoutput "messages" option, and dispatches each received message to a new LogicalRepMessageHandle_hook. Behavior is unchanged when the option is off or when no handler is installed. A transactional message is applied as a step of the remote transaction it belongs to, so the handler's work commits atomically with that transaction, and transaction streaming and ALTER SUBSCRIPTION ... SKIP apply to it as they do to any other change. A non-transactional message belongs to no remote transaction and is committed on its own. A message may still reach the handler more than once, for example after an apply worker restart, so handlers must be idempotent. Note also that messages are not scoped by publication: every message emitted in the publisher database is sent, subject only to origin filtering, and pg_logical_emit_message() requires no special privileges. A handler must therefore treat the payload as untrusted input, and as an arbitrary string of bytes rather than as text in the subscriber's encoding. The publisher must be running PostgreSQL 14 or later, where pgoutput gained the "messages" option; against an older publisher the option is accepted but no message is ever sent. Bump catalog version. Reviewed-by: Bharath Rupireddy <bharath.rupireddyforpostgres@gmail.com> Reviewed-by: Fujii Masao <masao.fujii@gmail.com> Reviewed-by: Amit Kapila <amit.kapila16@gmail.com> Reviewed-by: Chao Li <li.evan.chao@gmail.com> Discussion: https://postgr.es/m/CAD21AoCTNGiddikkUcDKj5QLnsg-51bpr-o6L-GTHWZL4ZFYtQ@mail.gmail.com --- doc/src/sgml/catalogs.sgml | 11 + doc/src/sgml/func/func-admin.sgml | 7 +- doc/src/sgml/logical-replication.sgml | 35 ++ doc/src/sgml/ref/alter_subscription.sgml | 5 +- doc/src/sgml/ref/create_subscription.sgml | 43 +++ src/backend/catalog/pg_subscription.c | 1 + src/backend/catalog/system_views.sql | 5 +- src/backend/commands/subscriptioncmds.c | 27 +- .../libpqwalreceiver/libpqwalreceiver.c | 4 + src/backend/replication/logical/proto.c | 37 ++ src/backend/replication/logical/worker.c | 100 +++++- src/bin/pg_dump/pg_dump.c | 16 +- src/bin/pg_dump/pg_dump.h | 1 + src/bin/pg_dump/t/002_pg_dump.pl | 4 +- src/bin/psql/describe.c | 7 +- src/bin/psql/tab-complete.in.c | 6 +- src/include/catalog/pg_subscription.h | 5 + src/include/replication/logicalproto.h | 13 + src/include/replication/logicalworker.h | 49 +++ src/include/replication/walreceiver.h | 1 + src/test/modules/Makefile | 1 + src/test/modules/meson.build | 1 + .../modules/test_logicalmsg_hooks/.gitignore | 4 + .../modules/test_logicalmsg_hooks/Makefile | 20 ++ .../modules/test_logicalmsg_hooks/meson.build | 28 ++ .../test_logicalmsg_hooks/t/001_basic.pl | 318 ++++++++++++++++++ .../test_logicalmsg_hooks.c | 140 ++++++++ src/test/regress/expected/subscription.out | 202 ++++++----- src/test/regress/sql/subscription.sql | 17 + src/tools/pgindent/typedefs.list | 2 + 30 files changed, 1000 insertions(+), 110 deletions(-) create mode 100644 src/test/modules/test_logicalmsg_hooks/.gitignore create mode 100644 src/test/modules/test_logicalmsg_hooks/Makefile create mode 100644 src/test/modules/test_logicalmsg_hooks/meson.build create mode 100644 src/test/modules/test_logicalmsg_hooks/t/001_basic.pl create mode 100644 src/test/modules/test_logicalmsg_hooks/test_logicalmsg_hooks.c diff --git a/doc/src/sgml/catalogs.sgml b/doc/src/sgml/catalogs.sgml index 07ba05dccd9..5e7f3032baa 100644 --- a/doc/src/sgml/catalogs.sgml +++ b/doc/src/sgml/catalogs.sgml @@ -8250,6 +8250,17 @@ SCRAM-SHA-256$<replaceable>&lt;iteration count&gt;</replaceable>:<replaceable>&l </para></entry> </row> + <row> + <entry role="catalog_table_entry"><para role="column_definition"> + <structfield>submessages</structfield> <type>bool</type> + </para> + <para> + If true, the subscription requests the publisher to send logical + decoding messages, which are passed to a handler registered by an + extension on the subscriber. + </para></entry> + </row> + <row> <entry role="catalog_table_entry"><para role="column_definition"> <structfield>subconninfo</structfield> <type>text</type> diff --git a/doc/src/sgml/func/func-admin.sgml b/doc/src/sgml/func/func-admin.sgml index ed872264f02..e63d72f20ff 100644 --- a/doc/src/sgml/func/func-admin.sgml +++ b/doc/src/sgml/func/func-admin.sgml @@ -1479,7 +1479,12 @@ postgres=# SELECT '0/0'::pg_lsn + pd.segment_number * ps.setting::int + :offset <literal>false</literal>) controls if the message is immediately flushed to WAL or not. <parameter>flush</parameter> has no effect with <parameter>transactional</parameter>, as the message's WAL - record is flushed along with its transaction. + record is flushed along with its transaction. In addition to logical + decoding clients, messages emitted by this function are delivered to + subscribers whose subscriptions have + <link linkend="sql-createsubscription-params-with-messages"><literal>messages</literal></link> + enabled, where they are passed to a handler registered by an extension + on the subscriber. </para></entry> </row> diff --git a/doc/src/sgml/logical-replication.sgml b/doc/src/sgml/logical-replication.sgml index 4701a3d9d18..849724efc43 100644 --- a/doc/src/sgml/logical-replication.sgml +++ b/doc/src/sgml/logical-replication.sgml @@ -2661,6 +2661,41 @@ CONTEXT: processing remote data for replication origin "pg_16395" during "INSER not need privileges to <literal>SET ROLE</literal> to the sequence owner. </para> + <para> + Logical decoding messages are an exception to the user switching described + above. If the subscription has <link linkend="sql-createsubscription-params-with-messages"><literal>messages</literal></link> + enabled, the handler registered for them runs with the privileges of the + subscription owner. Messages are not restricted to the tables or schemas + of the subscribed publications either, and by default any role may call + <link linkend="pg-logical-emit-message"><function>pg_logical_emit_message</function></link>. + Any role that can connect to the publisher database can therefore choose + what the handler is given, and a handler that acts on what it receives does + so with the subscription owner's privileges. Restrict who can emit messages + on the publisher. Revoke the default privilege and grant it to a role created + for this purpose: +<programlisting> +REVOKE EXECUTE ON FUNCTION pg_logical_emit_message(boolean, text, text, boolean) FROM PUBLIC; +REVOKE EXECUTE ON FUNCTION pg_logical_emit_message(boolean, text, bytea, boolean) FROM PUBLIC; +GRANT EXECUTE ON FUNCTION pg_logical_emit_message(boolean, text, text, boolean) TO message_emitter; +GRANT EXECUTE ON FUNCTION pg_logical_emit_message(boolean, text, bytea, boolean) TO message_emitter; +</programlisting> + Function privileges are per database, so this has to be done in each database + that such a subscription replicates from, and in <literal>template1</literal> + if databases created later are to inherit it. No role that is not trusted to + drive the handler should be a member of the new role, since a role that can + <literal>SET ROLE</literal> to it can emit whatever it likes. Where a message + has to be emitted on behalf of an ordinary user, for example from an event + trigger that captures that user's DDL, use a <literal>SECURITY DEFINER</literal> + function owned by that role rather than granting the privilege more widely. + </para> + + <para> + This is a control on the publisher, and the subscriber has no way to verify + that it is in place. If you do not administer both the publisher and the + subscriber, do not install a message handler that does anything you would + not let every role on the publisher do. + </para> + <para> On the publisher, privileges are only checked once at the start of a replication connection and are not re-checked as each change record is read. diff --git a/doc/src/sgml/ref/alter_subscription.sgml b/doc/src/sgml/ref/alter_subscription.sgml index 545264e8a0a..72b1b0fcd83 100644 --- a/doc/src/sgml/ref/alter_subscription.sgml +++ b/doc/src/sgml/ref/alter_subscription.sgml @@ -302,8 +302,9 @@ ALTER SUBSCRIPTION <replaceable class="parameter">name</replaceable> RENAME TO < <link linkend="sql-createsubscription-params-with-two-phase"><literal>two_phase</literal></link>, <link linkend="sql-createsubscription-params-with-retain-dead-tuples"><literal>retain_dead_tuples</literal></link>, <link linkend="sql-createsubscription-params-with-max-retention-duration"><literal>max_retention_duration</literal></link>, - <link linkend="sql-createsubscription-params-with-wal-receiver-timeout"><literal>wal_receiver_timeout</literal></link>, and - <link linkend="sql-createsubscription-params-with-conflict-log-destination"><literal>conflict_log_destination</literal></link>. + <link linkend="sql-createsubscription-params-with-wal-receiver-timeout"><literal>wal_receiver_timeout</literal></link>, + <link linkend="sql-createsubscription-params-with-conflict-log-destination"><literal>conflict_log_destination</literal></link>, and + <link linkend="sql-createsubscription-params-with-messages"><literal>messages</literal></link>. Only a superuser can set <literal>password_required = false</literal>. </para> diff --git a/doc/src/sgml/ref/create_subscription.sgml b/doc/src/sgml/ref/create_subscription.sgml index 6248c228a64..1c38daaf434 100644 --- a/doc/src/sgml/ref/create_subscription.sgml +++ b/doc/src/sgml/ref/create_subscription.sgml @@ -478,6 +478,49 @@ CREATE SUBSCRIPTION <replaceable class="parameter">subscription_name</replaceabl </listitem> </varlistentry> + <varlistentry id="sql-createsubscription-params-with-messages"> + <term><literal>messages</literal> (<type>boolean</type>)</term> + <listitem> + <para> + Specifies whether the subscription requests the publisher to send + logical decoding messages, that is, those emitted with + <link linkend="pg-logical-emit-message"><function>pg_logical_emit_message</function></link>. + The default is <literal>false</literal>. + </para> + <para> + Received messages are only acted upon if an extension installed on + the subscriber has registered a handler for them; otherwise they are + received and discarded. A message may be passed to the handler more + than once, for example after an apply worker restart, so a handler + must be prepared to process the same message repeatedly. + </para> + <para> + A transactional message is applied as part of the remote transaction + that emitted it, and is therefore delivered in commit order with + respect to other transactions, not in the order in which it was written. + Note that + <link linkend="sql-createsubscription-params-with-streaming"><literal>streaming</literal></link> + and <link linkend="sql-altersubscription-params-skip"><literal>SKIP</literal></link> + apply to transactional messages just as they do to other changes. A + non-transactional message is delivered as soon as it is decoded, even + if the transaction that emitted it later aborted. + </para> + <para> + Messages are not restricted to the tables or schemas of the + subscribed publications: every message emitted in the publisher + database is sent, subject only to + <link linkend="sql-createsubscription-params-with-origin"><literal>origin</literal></link> + filtering. Message contents are supplied by whoever calls + <function>pg_logical_emit_message</function> on the publisher, which + by default any role may do, so a handler should treat them as + untrusted input. A message is an arbitrary string of bytes, and is + transmitted without character set conversion; it may therefore + contain embedded nulls, and need not be valid text in the encoding + of either database. + </para> + </listitem> + </varlistentry> + <varlistentry id="sql-createsubscription-params-with-origin"> <term><literal>origin</literal> (<type>string</type>)</term> <listitem> diff --git a/src/backend/catalog/pg_subscription.c b/src/backend/catalog/pg_subscription.c index 60c15b22194..138cdecd436 100644 --- a/src/backend/catalog/pg_subscription.c +++ b/src/backend/catalog/pg_subscription.c @@ -133,6 +133,7 @@ GetSubscription(Oid subid, bool missing_ok) sub->maxretention = subform->submaxretention; sub->retentionactive = subform->subretentionactive; sub->conflictlogrelid = subform->subconflictlogrelid; + sub->messages = subform->submessages; /* Get slotname */ datum = SysCacheGetAttr(SUBSCRIPTIONOID, diff --git a/src/backend/catalog/system_views.sql b/src/backend/catalog/system_views.sql index 6dce9192ba0..19b96bba1fd 100644 --- a/src/backend/catalog/system_views.sql +++ b/src/backend/catalog/system_views.sql @@ -1545,8 +1545,9 @@ GRANT SELECT (oid, subdbid, subskiplsn, subname, subowner, subenabled, subbinary, substream, subtwophasestate, subdisableonerr, subpasswordrequired, subrunasowner, subfailover, subretaindeadtuples, submaxretention, subretentionactive, - subserver, subconflictlogrelid, subconflictlogdest, subslotname, - subsynccommit, subwalrcvtimeout, subpublications, suborigin) + subserver, subconflictlogrelid, subconflictlogdest, submessages, + subslotname, subsynccommit, subwalrcvtimeout, subpublications, + suborigin) ON pg_subscription TO public; CREATE VIEW pg_stat_subscription_stats AS diff --git a/src/backend/commands/subscriptioncmds.c b/src/backend/commands/subscriptioncmds.c index ce45f7e16d4..9dc8d608b8d 100644 --- a/src/backend/commands/subscriptioncmds.c +++ b/src/backend/commands/subscriptioncmds.c @@ -84,6 +84,7 @@ #define SUBOPT_LSN 0x00020000 #define SUBOPT_ORIGIN 0x00040000 #define SUBOPT_CONFLICT_LOG_DEST 0x00080000 +#define SUBOPT_MESSAGES 0x00100000 /* check if the 'val' has 'bits' set */ #define IsSet(val, bits) (((val) & (bits)) == (bits)) @@ -110,6 +111,7 @@ typedef struct SubOpts bool runasowner; bool failover; bool retaindeadtuples; + bool messages; int32 maxretention; char *origin; ConflictLogDest conflictlogdest; @@ -208,6 +210,8 @@ parse_subscription_options(ParseState *pstate, List *stmt_options, opts->origin = pstrdup(LOGICALREP_ORIGIN_ANY); if (IsSet(supported_opts, SUBOPT_CONFLICT_LOG_DEST)) opts->conflictlogdest = CONFLICT_LOG_DEST_LOG; + if (IsSet(supported_opts, SUBOPT_MESSAGES)) + opts->messages = false; /* Parse options */ foreach(lc, stmt_options) @@ -460,6 +464,15 @@ parse_subscription_options(ParseState *pstate, List *stmt_options, opts->conflictlogdest = GetConflictLogDest(val); opts->specified_opts |= SUBOPT_CONFLICT_LOG_DEST; } + else if (IsSet(supported_opts, SUBOPT_MESSAGES) && + strcmp(defel->defname, "messages") == 0) + { + if (IsSet(opts->specified_opts, SUBOPT_MESSAGES)) + errorConflictingDefElem(defel, pstate); + + opts->specified_opts |= SUBOPT_MESSAGES; + opts->messages = defGetBoolean(defel); + } else ereport(ERROR, (errcode(ERRCODE_SYNTAX_ERROR), @@ -700,7 +713,7 @@ CreateSubscription(ParseState *pstate, CreateSubscriptionStmt *stmt, SUBOPT_RETAIN_DEAD_TUPLES | SUBOPT_MAX_RETENTION_DURATION | SUBOPT_WAL_RECEIVER_TIMEOUT | SUBOPT_ORIGIN | - SUBOPT_CONFLICT_LOG_DEST); + SUBOPT_CONFLICT_LOG_DEST | SUBOPT_MESSAGES); parse_subscription_options(pstate, stmt->options, supported_opts, &opts); /* @@ -872,6 +885,8 @@ CreateSubscription(ParseState *pstate, CreateSubscriptionStmt *stmt, values[Anum_pg_subscription_subretentionactive - 1] = BoolGetDatum(opts.retaindeadtuples); values[Anum_pg_subscription_subserver - 1] = ObjectIdGetDatum(serverid); + values[Anum_pg_subscription_submessages - 1] = + BoolGetDatum(opts.messages); if (stmt->conninfo) { Assert(stmt->conninfo == conninfo && !OidIsValid(serverid)); @@ -1714,7 +1729,8 @@ AlterSubscription(ParseState *pstate, AlterSubscriptionStmt *stmt, SUBOPT_MAX_RETENTION_DURATION | SUBOPT_WAL_RECEIVER_TIMEOUT | SUBOPT_ORIGIN | - SUBOPT_CONFLICT_LOG_DEST); + SUBOPT_CONFLICT_LOG_DEST | + SUBOPT_MESSAGES); break; case ALTER_SUBSCRIPTION_ENABLED: @@ -2128,6 +2144,13 @@ AlterSubscription(ParseState *pstate, AlterSubscriptionStmt *stmt, } } + if (IsSet(opts.specified_opts, SUBOPT_MESSAGES)) + { + values[Anum_pg_subscription_submessages - 1] = + BoolGetDatum(opts.messages); + replaces[Anum_pg_subscription_submessages - 1] = true; + } + update_tuple = true; break; } diff --git a/src/backend/replication/libpqwalreceiver/libpqwalreceiver.c b/src/backend/replication/libpqwalreceiver/libpqwalreceiver.c index dcc738f4e38..ffd03909ab4 100644 --- a/src/backend/replication/libpqwalreceiver/libpqwalreceiver.c +++ b/src/backend/replication/libpqwalreceiver/libpqwalreceiver.c @@ -664,6 +664,10 @@ libpqrcv_startstreaming(WalReceiverConn *conn, appendQuotedLiteral(&cmd, options->proto.logical.origin); } + if (options->proto.logical.messages && + PQserverVersion(conn->streamConn) >= 140000) + appendStringInfo(&cmd, ", messages 'on'"); + pubnames = options->proto.logical.publication_names; pubnames_str = stringlist_to_identifierstr(pubnames); appendStringInfoString(&cmd, ", publication_names "); diff --git a/src/backend/replication/logical/proto.c b/src/backend/replication/logical/proto.c index 86ad97cd937..6461361d68d 100644 --- a/src/backend/replication/logical/proto.c +++ b/src/backend/replication/logical/proto.c @@ -660,6 +660,43 @@ logicalrep_write_message(StringInfo out, TransactionId xid, XLogRecPtr lsn, pq_sendbytes(out, message, sz); } +/* + * Read MESSAGE from stream. + */ +void +logicalrep_read_message(StringInfo in, LogicalRepMessageData *msg_data) +{ + Size len; + char *msg; + uint8 flags; + + /* read and decode flags */ + flags = pq_getmsgint(in, 1); + msg_data->transactional = (flags & MESSAGE_TRANSACTIONAL) != 0; + + msg_data->lsn = pq_getmsgint64(in); + msg_data->prefix = pstrdup(pq_getmsgstring(in)); + + /* read message length */ + len = pq_getmsgint(in, 4); + msg_data->message_size = len; + + /* and data */ + msg = palloc(len + 1); + pq_copymsgbytes(in, msg, len); + + /* + * The payload is an arbitrary string of bytes, so message_size, not + * strlen(), is what gives its length. NULL-terminate it anyway, as + * logicalrep_read_tuple() does for column values: that way it can be + * handed to code expecting a C string without reading past the end, which + * is convenient for logging and debugging. Such code will of course stop + * at the first embedded null, if any. + */ + msg[len] = '\0'; + msg_data->message = msg; +} + /* * Write relation description to the output stream. */ diff --git a/src/backend/replication/logical/worker.c b/src/backend/replication/logical/worker.c index 95044620e68..166fdc5b2fb 100644 --- a/src/backend/replication/logical/worker.c +++ b/src/backend/replication/logical/worker.c @@ -561,6 +561,9 @@ typedef struct ApplySubXactData static ApplySubXactData subxact_data = {0, 0, InvalidTransactionId, NULL}; +/* Hook for plugins to get control in handling logical decoding messages */ +LogicalRepMessageHandle_hook_type LogicalRepMessageHandle_hook = NULL; + static inline void subxact_filename(char *path, Oid subid, TransactionId xid); static inline void changes_filename(char *path, Oid subid, TransactionId xid); @@ -1709,6 +1712,84 @@ apply_handle_origin(StringInfo s) errmsg_internal("ORIGIN message sent out of order"))); } +/* + * Handle MESSAGE message. + * + * Invoke a hook function if set. + */ +static void +apply_handle_message(StringInfo s) +{ + LogicalRepMessageData msg; + + /* + * The publisher must not send a message that this worker did not ask for; + * see set_stream_options(). + */ + if (!MySubscription->messages || am_tablesync_worker()) + ereport(ERROR, + (errcode(ERRCODE_PROTOCOL_VIOLATION), + errmsg_internal("MESSAGE message received but messages are not enabled for this worker"))); + + if (!LogicalRepMessageHandle_hook) + return; + + if (is_skipping_changes() || + handle_streamed_transaction(LOGICAL_REP_MSG_MESSAGE, s)) + return; + + begin_replication_step(); + + logicalrep_read_message(s, &msg); + + /* + * The publisher sends a non-transactional message on its own, never + * inside a transaction or a streamed block; see + * ReorderBufferQueueMessage(). Reject one that arrives inside a + * transaction before calling the handler, as the commit below would + * otherwise split the remote transaction in two. + */ + if (!msg.transactional && + (in_remote_transaction || in_streamed_transaction)) + ereport(ERROR, + (errcode(ERRCODE_PROTOCOL_VIOLATION), + errmsg_internal("non-transactional MESSAGE message received inside a transaction"))); + + (*LogicalRepMessageHandle_hook) (&msg); + + end_replication_step(); + + /* + * A transactional message is applied as a step of the remote transaction + * that emitted it, and is committed together with it when applying the + * commit message. A non-transactional message belongs to no remote + * transaction, so commit it here. + */ + if (!msg.transactional) + { + /* + * The message doesn't belong to any remote transaction, so there is + * no remote commit LSN nor timestamp to record. Clear the state left + * over by the previously applied transaction so that this commit + * doesn't inherit it. + */ + replorigin_xact_clear(false); + + CommitTransactionCommand(); + + pgstat_report_stat(false); + + /* + * Report the flush position only after the local commit is durable. + * Otherwise send_feedback() would report everything received so far + * as flushed, the publisher would advance confirmed_flush_lsn past + * this message, and a crash before the commit reaches disk would lose + * the handler's work without the message ever being sent again. + */ + store_flush_position(msg.lsn, XactLastCommitEnd); + } +} + /* * Initialize fileset (if not already done). * @@ -3894,12 +3975,7 @@ apply_dispatch(StringInfo s) break; case LOGICAL_REP_MSG_MESSAGE: - - /* - * Logical replication does not use generic logical messages yet. - * Although, it could be used by other applications that use this - * output plugin. - */ + apply_handle_message(s); break; case LOGICAL_REP_MSG_STREAM_START: @@ -5185,7 +5261,8 @@ maybe_reread_subscription(void) newsub->passwordrequired != MySubscription->passwordrequired || strcmp(newsub->origin, MySubscription->origin) != 0 || newsub->owner != MySubscription->owner || - !equal(newsub->publications, MySubscription->publications)) + !equal(newsub->publications, MySubscription->publications) || + newsub->messages != MySubscription->messages) { if (am_parallel_apply_worker()) ereport(LOG, @@ -5676,6 +5753,15 @@ set_stream_options(WalRcvStreamOptions *options, options->proto.logical.twophase = false; options->proto.logical.origin = pstrdup(MySubscription->origin); + + /* + * Logical messages are relation-agnostic, so they don't map cleanly onto + * the tablesync worker of a particular relation, and there would be no + * well-defined ordering between a message and the initial copy. Leave + * them to the (parallel) apply workers. + */ + options->proto.logical.messages = (MySubscription->messages && + !am_tablesync_worker()); } /* diff --git a/src/bin/pg_dump/pg_dump.c b/src/bin/pg_dump/pg_dump.c index 4fc74e6e26e..efc60fdefb0 100644 --- a/src/bin/pg_dump/pg_dump.c +++ b/src/bin/pg_dump/pg_dump.c @@ -5142,6 +5142,7 @@ getSubscriptions(Archive *fout) int i_subfailover; int i_subretaindeadtuples; int i_submaxretention; + int i_submessages; int i, ntups; @@ -5240,9 +5241,14 @@ getSubscriptions(Archive *fout) " '-1' AS subwalrcvtimeout,\n"); if (fout->remoteVersion >= 190000) - appendPQExpBufferStr(query, " fs.srvname AS subservername\n"); + appendPQExpBufferStr(query, " fs.srvname AS subservername,\n"); else - appendPQExpBufferStr(query, " NULL AS subservername\n"); + appendPQExpBufferStr(query, " NULL AS subservername,\n"); + + if (fout->remoteVersion >= 200000) + appendPQExpBufferStr(query, " s.submessages\n"); + else + appendPQExpBufferStr(query, " false AS submessages\n"); appendPQExpBufferStr(query, "FROM pg_subscription s\n"); @@ -5291,6 +5297,7 @@ getSubscriptions(Archive *fout) i_subpublications = PQfnumber(res, "subpublications"); i_suborigin = PQfnumber(res, "suborigin"); i_suboriginremotelsn = PQfnumber(res, "suboriginremotelsn"); + i_submessages = PQfnumber(res, "submessages"); subinfo = pg_malloc_array(SubscriptionInfo, ntups); @@ -5348,6 +5355,8 @@ getSubscriptions(Archive *fout) else subinfo[i].suboriginremotelsn = pg_strdup(PQgetvalue(res, i, i_suboriginremotelsn)); + subinfo[i].submessages = + (strcmp(PQgetvalue(res, i, i_submessages), "t") == 0); /* Decide whether we want to dump it */ selectDumpableObject(&(subinfo[i].dobj), fout); @@ -5600,6 +5609,9 @@ dumpSubscription(Archive *fout, const SubscriptionInfo *subinfo) if (subinfo->subretaindeadtuples) appendPQExpBufferStr(query, ", retain_dead_tuples = true"); + if (subinfo->submessages) + appendPQExpBufferStr(query, ", messages = true"); + if (subinfo->submaxretention) appendPQExpBuffer(query, ", max_retention_duration = %d", subinfo->submaxretention); diff --git a/src/bin/pg_dump/pg_dump.h b/src/bin/pg_dump/pg_dump.h index 41d932f003b..1df8a5252a4 100644 --- a/src/bin/pg_dump/pg_dump.h +++ b/src/bin/pg_dump/pg_dump.h @@ -721,6 +721,7 @@ typedef struct _SubscriptionInfo bool subrunasowner; bool subfailover; bool subretaindeadtuples; + bool submessages; int submaxretention; char *subservername; char *subconninfo; diff --git a/src/bin/pg_dump/t/002_pg_dump.pl b/src/bin/pg_dump/t/002_pg_dump.pl index ec73c4d5be9..a32710d085a 100644 --- a/src/bin/pg_dump/t/002_pg_dump.pl +++ b/src/bin/pg_dump/t/002_pg_dump.pl @@ -3424,9 +3424,9 @@ my %tests = ( create_order => 50, create_sql => 'CREATE SUBSCRIPTION sub3 CONNECTION \'dbname=doesnotexist\' PUBLICATION pub1 - WITH (connect = false, origin = any, streaming = on);', + WITH (connect = false, origin = any, streaming = on, messages = on);', regexp => qr/^ - \QCREATE SUBSCRIPTION sub3 CONNECTION 'dbname=doesnotexist' PUBLICATION pub1 WITH (connect = false, slot_name = 'sub3', streaming = on);\E + \QCREATE SUBSCRIPTION sub3 CONNECTION 'dbname=doesnotexist' PUBLICATION pub1 WITH (connect = false, slot_name = 'sub3', streaming = on, messages = true);\E /xm, like => { %full_runs, section_post_data => 1, }, unlike => { diff --git a/src/bin/psql/describe.c b/src/bin/psql/describe.c index e4a4ba47280..e6e48a9c66c 100644 --- a/src/bin/psql/describe.c +++ b/src/bin/psql/describe.c @@ -6774,7 +6774,7 @@ describeSubscriptions(const char *pattern, bool verbose) printQueryOpt myopt = pset.popt; static const bool translate_columns[] = {false, false, false, false, false, false, false, false, false, false, false, false, false, false, - false, false, false, false, false, false, false}; + false, false, false, false, false, false, false, false}; initPQExpBuffer(&buf); @@ -6852,6 +6852,11 @@ describeSubscriptions(const char *pattern, bool verbose) gettext_noop("Retention active")); } + if (pset.sversion >= 200000) + appendPQExpBuffer(&buf, + ", submessages AS \"%s\"\n", + gettext_noop("Messages")); + appendPQExpBuffer(&buf, ", subsynccommit AS \"%s\"\n" ", subconninfo AS \"%s\"\n", diff --git a/src/bin/psql/tab-complete.in.c b/src/bin/psql/tab-complete.in.c index 96de78cd087..2346c8b969f 100644 --- a/src/bin/psql/tab-complete.in.c +++ b/src/bin/psql/tab-complete.in.c @@ -2360,8 +2360,8 @@ match_previous_words(int pattern_id, /* ALTER SUBSCRIPTION <name> SET ( */ else if (Matches("ALTER", "SUBSCRIPTION", MatchAny, MatchAnyN, "SET", "(")) COMPLETE_WITH("binary", "conflict_log_destination", "disable_on_error", - "failover", "max_retention_duration", "origin", - "password_required", "retain_dead_tuples", + "failover", "max_retention_duration", "messages", + "origin", "password_required", "retain_dead_tuples", "run_as_owner", "slot_name", "streaming", "synchronous_commit", "two_phase", "wal_receiver_timeout"); @@ -3897,7 +3897,7 @@ match_previous_words(int pattern_id, else if (Matches("CREATE", "SUBSCRIPTION", MatchAnyN, "WITH", "(")) COMPLETE_WITH("binary", "conflict_log_destination", "connect", "copy_data", "create_slot", "disable_on_error", "enabled", "failover", - "max_retention_duration", "origin", + "max_retention_duration", "messages", "origin", "password_required", "retain_dead_tuples", "run_as_owner", "slot_name", "streaming", "synchronous_commit", "two_phase", diff --git a/src/include/catalog/pg_subscription.h b/src/include/catalog/pg_subscription.h index b860f02d166..07b28f86e0f 100644 --- a/src/include/catalog/pg_subscription.h +++ b/src/include/catalog/pg_subscription.h @@ -96,6 +96,9 @@ CATALOG(pg_subscription,6100,SubscriptionRelationId) BKI_SHARED_RELATION BKI_ROW * server */ Oid subconflictlogrelid; /* Relid of the conflict log table. */ + + bool submessages; /* True if the subscription wants to receive + * logical messages. */ #ifdef CATALOG_VARLEN /* variable-length fields start here */ /* @@ -173,6 +176,8 @@ typedef struct Subscription * exceeded max_retention_duration, when * defined */ Oid conflictlogrelid; /* conflict log table Oid */ + bool messages; /* True if the subscription wants to receive + * logical messages */ char *slotname; /* Name of the replication slot */ char *synccommit; /* Synchronous commit setting for worker */ char *walrcvtimeout; /* wal_receiver_timeout setting for worker */ diff --git a/src/include/replication/logicalproto.h b/src/include/replication/logicalproto.h index 058a955e20c..83741a2080d 100644 --- a/src/include/replication/logicalproto.h +++ b/src/include/replication/logicalproto.h @@ -191,6 +191,18 @@ typedef struct LogicalRepStreamAbortData TimestampTz abort_time; } LogicalRepStreamAbortData; +/* + * Logical decoding message information + */ +typedef struct LogicalRepMessageData +{ + XLogRecPtr lsn; + bool transactional; + const char *prefix; + Size message_size; /* length of message in bytes */ + const char *message; /* payload; may contain embedded nulls */ +} LogicalRepMessageData; + extern void logicalrep_write_begin(StringInfo out, ReorderBufferTXN *txn); extern void logicalrep_read_begin(StringInfo in, LogicalRepBeginData *begin_data); @@ -248,6 +260,7 @@ extern List *logicalrep_read_truncate(StringInfo in, bool *cascade, bool *restart_seqs); extern void logicalrep_write_message(StringInfo out, TransactionId xid, XLogRecPtr lsn, bool transactional, const char *prefix, Size sz, const char *message); +extern void logicalrep_read_message(StringInfo in, LogicalRepMessageData *msg_data); extern void logicalrep_write_rel(StringInfo out, TransactionId xid, Relation rel, Bitmapset *columns, PublishGencolsType include_gencols_type); diff --git a/src/include/replication/logicalworker.h b/src/include/replication/logicalworker.h index 7d748a28da8..3f745201c1a 100644 --- a/src/include/replication/logicalworker.h +++ b/src/include/replication/logicalworker.h @@ -16,6 +16,55 @@ extern PGDLLIMPORT volatile sig_atomic_t ParallelApplyMessagePending; +/* + * Forward-declared so that this header, which is included by several files + * unrelated to logical replication, does not have to include logicalproto.h. + * A handler that dereferences the struct must include that header itself. + */ +struct LogicalRepMessageData; + +/* + * Hook for extensions to handle logical decoding messages (see + * pg_logical_emit_message()) received from the publisher. Called only when + * the subscription has the "messages" option enabled. + * + * The handler runs in the apply worker, inside a transaction, with an active + * snapshot pushed. For a transactional message the handler's work is part of + * the remote transaction and commits with it; a non-transactional message is + * committed as an independent unit. The same message may be delivered more + * than once, so the handler must be idempotent. + * + * The payload is an arbitrary string of bytes. The WAL record makes no + * distinction between the text and the bytea form of + * pg_logical_emit_message(), and the bytes are transmitted without character + * set conversion, so the payload may contain embedded nulls and bytes that + * are not valid in the subscriber's encoding. Its length is given by + * msg->message_size, not by strlen(); the buffer is null-terminated only for + * convenience, and that null is not part of the payload. + * + * The handler always runs with the privileges of the subscription owner. + * The payload is chosen by whoever called pg_logical_emit_message() on the + * publisher, which by default any role may do, so a handler that acts on the + * payload gives every role on the publisher the subscription owner's + * privileges. A handler that needs less than that can switch to a less + * privileged role with SwitchToUntrustedUser(). It must take that role + * from its own configuration and never from the payload, which is written + * by the very user the switch defends against, and it should check the + * role with member_can_set_role() first: SwitchToUntrustedUser() throws an + * error if the subscription owner cannot SET ROLE to it, and that error + * loops as described below. MySubscription, declared in + * worker_internal.h, is available where the choice differs per + * subscription. + * + * Note that the apply worker runs with an empty search_path. An ERROR thrown + * by the handler restarts the apply worker, or disables the subscription if + * disable_on_error is set. Because the message is delivered again after the + * restart, a handler cannot reject a message by throwing an error; that only + * loops. A message the handler does not want must simply be ignored. + */ +typedef void (*LogicalRepMessageHandle_hook_type) (const struct LogicalRepMessageData *msg); +extern PGDLLIMPORT LogicalRepMessageHandle_hook_type LogicalRepMessageHandle_hook; + extern void ApplyWorkerMain(Datum main_arg); extern void ParallelApplyWorkerMain(Datum main_arg); extern void TableSyncWorkerMain(Datum main_arg); diff --git a/src/include/replication/walreceiver.h b/src/include/replication/walreceiver.h index 5cfe90f9989..42a1dab24b2 100644 --- a/src/include/replication/walreceiver.h +++ b/src/include/replication/walreceiver.h @@ -188,6 +188,7 @@ typedef struct * prepare time */ char *origin; /* Only publish data originating from the * specified origin */ + bool messages; /* Ask publisher to send logical messages */ } logical; } proto; } WalRcvStreamOptions; diff --git a/src/test/modules/Makefile b/src/test/modules/Makefile index 6c1504ac932..a8a31f2b23e 100644 --- a/src/test/modules/Makefile +++ b/src/test/modules/Makefile @@ -37,6 +37,7 @@ SUBDIRS = \ test_integerset \ test_json_parser \ test_lfind \ + test_logicalmsg_hooks \ test_lwlock_tranches \ test_misc \ test_oat_hooks \ diff --git a/src/test/modules/meson.build b/src/test/modules/meson.build index ef3cdb8faf9..f1aaeae71ab 100644 --- a/src/test/modules/meson.build +++ b/src/test/modules/meson.build @@ -39,6 +39,7 @@ subdir('test_int128') subdir('test_integerset') subdir('test_json_parser') subdir('test_lfind') +subdir('test_logicalmsg_hooks') subdir('test_lwlock_tranches') subdir('test_misc') subdir('test_oat_hooks') diff --git a/src/test/modules/test_logicalmsg_hooks/.gitignore b/src/test/modules/test_logicalmsg_hooks/.gitignore new file mode 100644 index 00000000000..5dcb3ff9723 --- /dev/null +++ b/src/test/modules/test_logicalmsg_hooks/.gitignore @@ -0,0 +1,4 @@ +# Generated subdirectories +/log/ +/results/ +/tmp_check/ diff --git a/src/test/modules/test_logicalmsg_hooks/Makefile b/src/test/modules/test_logicalmsg_hooks/Makefile new file mode 100644 index 00000000000..af129009b8a --- /dev/null +++ b/src/test/modules/test_logicalmsg_hooks/Makefile @@ -0,0 +1,20 @@ +# src/test/modules/test_logicalmsg_hooks/Makefile + +MODULE_big = test_logicalmsg_hooks +OBJS = \ + $(WIN32RES) \ + test_logicalmsg_hooks.o +PGFILEDESC = "test_logicalmsg_hooks - test logical replication message hooks" + +TAP_TESTS = 1 + +ifdef USE_PGXS +PG_CONFIG = pg_config +PGXS := $(shell $(PG_CONFIG) --pgxs) +include $(PGXS) +else +subdir = src/test/modules/test_logicalmsg_hooks +top_builddir = ../../../.. +include $(top_builddir)/src/Makefile.global +include $(top_srcdir)/contrib/contrib-global.mk +endif diff --git a/src/test/modules/test_logicalmsg_hooks/meson.build b/src/test/modules/test_logicalmsg_hooks/meson.build new file mode 100644 index 00000000000..8084d82e47b --- /dev/null +++ b/src/test/modules/test_logicalmsg_hooks/meson.build @@ -0,0 +1,28 @@ +# Copyright (c) 2022-2026, PostgreSQL Global Development Group + +test_logicalmsg_hooks_sources = files( + 'test_logicalmsg_hooks.c', +) + +if host_system == 'windows' + test_logicalmsg_hooks_sources += rc_lib_gen.process(win32ver_rc, extra_args: [ + '--NAME', 'test_logicalmsg_hooks', + '--FILEDESC', 'test_logicalmsg_hooks - test logical message hooks',]) +endif + +test_logicalmsg_hooks = shared_module('test_logicalmsg_hooks', + test_logicalmsg_hooks_sources, + kwargs: pg_test_mod_args, +) +test_install_libs += test_logicalmsg_hooks + +tests += { + 'name': 'test_logicalmsg_hooks', + 'sd': meson.current_source_dir(), + 'bd': meson.current_build_dir(), + 'tap': { + 'tests': [ + 't/001_basic.pl', + ], + }, +} diff --git a/src/test/modules/test_logicalmsg_hooks/t/001_basic.pl b/src/test/modules/test_logicalmsg_hooks/t/001_basic.pl new file mode 100644 index 00000000000..a86de1273ab --- /dev/null +++ b/src/test/modules/test_logicalmsg_hooks/t/001_basic.pl @@ -0,0 +1,318 @@ +# Copyright (c) 2026, PostgreSQL Global Development Group + +# Test LogicalRepMessageHandle_hook. The test module records every message it +# receives into public.test_logicalmsg_log on the subscriber, so most checks +# here are plain queries against that table. + +use strict; +use warnings FATAL => 'all'; + +use PostgreSQL::Test::Cluster; +use PostgreSQL::Test::Utils; +use Test::More; + +my $pub = PostgreSQL::Test::Cluster->new('publisher'); +$pub->init(allows_streaming => 'logical'); + +# Keep the memory limit low so that the transactions below are streamed. +$pub->append_conf( + 'postgresql.conf', q{ +logical_decoding_work_mem = 64kB +max_prepared_transactions = 10 +}); +$pub->start; + +my $sub = PostgreSQL::Test::Cluster->new('subscriber'); +$sub->init(allows_streaming => 'logical'); +$sub->append_conf( + 'postgresql.conf', q{ +shared_preload_libraries = 'test_logicalmsg_hooks' +max_parallel_apply_workers_per_subscription = 2 +max_prepared_transactions = 10 +}); +$sub->start; + +$pub->safe_psql( + 'postgres', qq{ +CREATE TABLE test (a int); +INSERT INTO test VALUES (1); +CREATE PUBLICATION pub FOR ALL TABLES; +}); + +my $pub_connstr = $pub->connstr . ' dbname=postgres'; + +$sub->safe_psql( + 'postgres', qq{ +CREATE TABLE test (a int primary key); +CREATE SUBSCRIPTION sub CONNECTION '$pub_connstr' PUBLICATION pub WITH (messages = true, disable_on_error = true) +}); + +$sub->wait_for_subscription_sync($pub, 'sub'); + +# Number of messages recorded under the given prefix. +sub logged_count +{ + my ($prefix) = @_; + + return $sub->safe_psql('postgres', + "SELECT count(*) FROM test_logicalmsg_log WHERE prefix = '$prefix'"); +} + +# Verify that a transactional message is passed to the hook, and that the +# message data given to the hook is correct. +$pub->safe_psql('postgres', + q{SELECT pg_logical_emit_message(true, 'basic', 'transactional message')} +); +$pub->wait_for_catchup('sub'); + +is( $sub->safe_psql( + 'postgres', + q{SELECT transactional, message_size, message FROM test_logicalmsg_log + WHERE prefix = 'basic'}), + 't|21|transactional message', + "transactional message is passed to the hook"); + +# Verify that a non-transactional message is applied even if the transaction +# that emitted it rolls back. +$pub->safe_psql( + 'postgres', q{ +BEGIN; +SELECT pg_logical_emit_message(false, 'nontransactional', 'emitted then rolled back'); +ROLLBACK; +}); + +# Commit another message to flush WAL, as ROLLBACK doesn't flush. +$pub->safe_psql('postgres', + q{SELECT pg_logical_emit_message(true, 'flush', 'flush')}); +$pub->wait_for_catchup('sub'); + +is( $sub->safe_psql( + 'postgres', + q{SELECT transactional, message FROM test_logicalmsg_log + WHERE prefix = 'nontransactional'}), + 'f|emitted then rolled back', + "non-transactional message of an aborted transaction is applied"); + +# Verify that a transactional message emitted in a subtransaction that is +# rolled back is not passed to the hook. +$pub->safe_psql( + 'postgres', q{ +BEGIN; +SAVEPOINT sp; +SELECT pg_logical_emit_message(true, 'subxact_aborted', 'not delivered'); +ROLLBACK TO SAVEPOINT sp; +SELECT pg_logical_emit_message(true, 'subxact_committed', 'delivered'); +COMMIT; +}); +$pub->wait_for_catchup('sub'); + +is(logged_count('subxact_aborted'), + 0, "message of an aborted subtransaction is not applied"); +is(logged_count('subxact_committed'), + 1, "message of a committed subtransaction is applied"); + +# Verify that transactional messages are applied in streaming transactions, +# both when the leader apply worker serializes the transaction to a file and +# when it hands the transaction to a parallel apply worker. +foreach my $mode ('on', 'parallel') +{ + $sub->safe_psql('postgres', + "ALTER SUBSCRIPTION sub SET (streaming = $mode)"); + + # Reconnect so that the new streaming mode takes effect immediately. + $sub->safe_psql('postgres', 'ALTER SUBSCRIPTION sub DISABLE'); + $sub->safe_psql('postgres', 'ALTER SUBSCRIPTION sub ENABLE'); + + $pub->safe_psql('postgres', + q{SELECT pg_stat_reset_replication_slot('sub')}); + + $pub->safe_psql( + 'postgres', qq{ +BEGIN; +SAVEPOINT sp; +SELECT pg_logical_emit_message(true, 'streaming_subxact_aborted_$mode', 'not delivered'); +INSERT INTO test SELECT generate_series(1000, 6000); +ROLLBACK TO SAVEPOINT sp; +SELECT pg_logical_emit_message(true, 'streaming_$mode', 'streamed message'); +COMMIT; +}); + $pub->wait_for_catchup('sub'); + + # Make sure the transaction really was streamed, so that this test cannot + # silently degrade into a non-streaming one. + $pub->poll_query_until('postgres', + q{SELECT stream_txns > 0 FROM pg_stat_replication_slots WHERE slot_name = 'sub'} + ) or die "transaction was not streamed"; + + is(logged_count("streaming_subxact_aborted_$mode"), 0, + "message of an aborted streamed subtransaction is not applied (streaming = $mode)" + ); + is(logged_count("streaming_$mode"), + 1, + "message in a streamed transaction is applied (streaming = $mode)"); + + $pub->safe_psql('postgres', 'TRUNCATE test'); + $pub->wait_for_catchup('sub'); +} + +# Verify that a transaction whose only change is a message is applied when it +# is streamed. There is no volume to trigger streaming, so force it. The +# setting is picked up by the walsender started when the subscription is +# re-enabled below. +$pub->safe_psql('postgres', + 'ALTER SYSTEM SET debug_logical_replication_streaming = immediate'); +$pub->reload; +$pub->wait_for_log( + qr/parameter "debug_logical_replication_streaming" changed to "immediate"/ +); + +foreach my $mode ('on', 'parallel') +{ + $sub->safe_psql('postgres', + "ALTER SUBSCRIPTION sub SET (streaming = $mode)"); + $sub->safe_psql('postgres', 'ALTER SUBSCRIPTION sub DISABLE'); + $sub->safe_psql('postgres', 'ALTER SUBSCRIPTION sub ENABLE'); + + $pub->safe_psql('postgres', + q{SELECT pg_stat_reset_replication_slot('sub')}); + + $pub->safe_psql( + 'postgres', qq{ +BEGIN; +SELECT pg_logical_emit_message(true, 'streaming_only_$mode', 'the only change'); +COMMIT; +}); + $pub->wait_for_catchup('sub'); + + $pub->poll_query_until('postgres', + q{SELECT stream_txns > 0 FROM pg_stat_replication_slots WHERE slot_name = 'sub'} + ) or die "transaction was not streamed"; + + is(logged_count("streaming_only_$mode"), 1, + "message that is the only change of a streamed transaction is applied (streaming = $mode)" + ); +} + +$pub->safe_psql('postgres', + 'ALTER SYSTEM RESET debug_logical_replication_streaming'); +$pub->reload; +$pub->wait_for_log( + qr/parameter "debug_logical_replication_streaming" removed from configuration file/ +); + +$sub->safe_psql('postgres', 'ALTER SUBSCRIPTION sub SET (streaming = off)'); + +# Verify that ALTER SUBSCRIPTION ... SKIP skips a transactional message along +# with the rest of its transaction. The message is emitted before the +# conflicting change, so the hook runs and records the message on the first +# attempt; that work must be rolled back with the failed transaction, and must +# not be redone once the transaction is skipped. +$pub->safe_psql('postgres', 'INSERT INTO test VALUES (1)'); +$pub->wait_for_catchup('sub'); + +my $log_location = -s $sub->logfile; + +$pub->safe_psql( + 'postgres', q{ +BEGIN; +SELECT pg_logical_emit_message(true, 'skipped', 'rolled back, then skipped'); +INSERT INTO test VALUES (1); +COMMIT; +}); + +# Wait until the conflict disables the subscription. +$sub->poll_query_until('postgres', + q{SELECT subenabled = FALSE FROM pg_subscription WHERE subname = 'sub'}); + +# The hook did run before the transaction failed. +ok( $sub->log_contains( + qr/LOG[^\n]+received message: [^\n]+prefix: skipped/, + $log_location), + "hook is called for a message in a transaction that later fails"); + +# ... but its work was rolled back with the transaction. +is(logged_count('skipped'), 0, + "work done by the hook is rolled back with the remote transaction"); + +# Get the finish LSN of the failed transaction. +my $contents = slurp_file($sub->logfile, $log_location); +$contents =~ + qr/conflict detected on relation "public.test".*\n.*DETAIL:.*Could not apply remote change.*\n.*Key already exists in unique index "test_pkey", modified in transaction \d+: key .*, local row .*\n.*CONTEXT:.* for replication target relation "public.test" in transaction \d+, finished at ([[:xdigit:]]+\/[[:xdigit:]]+)/m + or die "could not get error-LSN"; +my $lsn = $1; + +$log_location = -s $sub->logfile; + +# Set skip LSN and re-enable the subscription. +$sub->safe_psql('postgres', qq{ALTER SUBSCRIPTION sub SKIP (lsn = '$lsn')}); +$sub->safe_psql('postgres', 'ALTER SUBSCRIPTION sub ENABLE'); + +# Wait for the failed transaction to be skipped. +$sub->poll_query_until('postgres', + q{SELECT subskiplsn = '0/0' FROM pg_subscription WHERE subname = 'sub'}); + +ok( !$sub->log_contains( + qr/LOG[^\n]+received message: [^\n]+prefix: skipped/, + $log_location), + "hook is not called for a message in a skipped transaction"); +is(logged_count('skipped'), 0, + "message in a skipped transaction is not applied"); + +# Verify that a message in a prepared transaction is applied when the +# transaction is committed, and discarded when it is rolled back. +$sub->safe_psql('postgres', 'ALTER SUBSCRIPTION sub DISABLE'); +$sub->safe_psql('postgres', 'ALTER SUBSCRIPTION sub SET (two_phase = on)'); +$sub->safe_psql('postgres', 'ALTER SUBSCRIPTION sub ENABLE'); +$sub->poll_query_until('postgres', + q{SELECT subtwophasestate = 'e' FROM pg_subscription WHERE subname = 'sub'} +); + +$pub->safe_psql( + 'postgres', q{ +BEGIN; +SELECT pg_logical_emit_message(true, 'prepared', 'committed later'); +PREPARE TRANSACTION 'msg_commit'; +}); +$pub->wait_for_catchup('sub'); + +is(logged_count('prepared'), + 0, "message in a prepared transaction is not applied before commit"); + +$pub->safe_psql('postgres', q{COMMIT PREPARED 'msg_commit'}); +$pub->wait_for_catchup('sub'); + +is(logged_count('prepared'), + 1, "message in a prepared transaction is applied at commit"); + +$pub->safe_psql( + 'postgres', q{ +BEGIN; +SELECT pg_logical_emit_message(true, 'prepared_rollback', 'rolled back later'); +PREPARE TRANSACTION 'msg_rollback'; +}); +$pub->wait_for_catchup('sub'); +$pub->safe_psql('postgres', q{ROLLBACK PREPARED 'msg_rollback'}); +$pub->wait_for_catchup('sub'); + +is(logged_count('prepared_rollback'), + 0, "message in a prepared transaction is discarded at rollback"); + +# Verify that no message reaches the hook once the option is turned off, which +# is also its default. +$sub->safe_psql('postgres', 'ALTER SUBSCRIPTION sub SET (messages = false)'); + +# Reconnect so that the change takes effect immediately. +$sub->safe_psql('postgres', 'ALTER SUBSCRIPTION sub DISABLE'); +$sub->safe_psql('postgres', 'ALTER SUBSCRIPTION sub ENABLE'); + +$pub->safe_psql('postgres', + q{SELECT pg_logical_emit_message(true, 'not_requested', 'messages is off')} +); +$pub->wait_for_catchup('sub'); + +is(logged_count('not_requested'), + 0, "no message is passed to the hook when messages is off"); + +$sub->stop; +$pub->stop; +done_testing(); diff --git a/src/test/modules/test_logicalmsg_hooks/test_logicalmsg_hooks.c b/src/test/modules/test_logicalmsg_hooks/test_logicalmsg_hooks.c new file mode 100644 index 00000000000..d8d6fc7c619 --- /dev/null +++ b/src/test/modules/test_logicalmsg_hooks/test_logicalmsg_hooks.c @@ -0,0 +1,140 @@ +/*-------------------------------------------------------------------------- + * + * test_logicalmsg_hooks.c + * Code for testing LogicalRepMessageHandle_hook + * + * The handler records every logical decoding message it receives into the + * table public.test_logicalmsg_log, so that tests can assert on the contents + * of a table rather than on the server log. Recording the messages this way + * also exercises the transactional behavior of the hook: a transactional + * message is recorded as part of the remote transaction that emitted it, and + * therefore disappears together with it if that transaction is rolled back or + * skipped, while a non-transactional message is recorded independently. + * + * The handler creates that table itself on the first message, so a test does + * not have to set it up. For a transactional message the creation is part of + * the remote transaction too, and is rolled back with it, so the table does + * not exist until some message has been applied successfully; a test must not + * query it before then. + * + * The messages are also logged, which lets a test tell "the handler ran but + * its work was rolled back" apart from "the handler was never called". + * + * Portions Copyright (c) 1996-2026, PostgreSQL Global Development Group + * Portions Copyright (c) 1994, Regents of the University of California + * + * IDENTIFICATION + * src/test/modules/test_logicalmsg_hooks/test_logicalmsg_hooks.c + * + * ------------------------------------------------------------------------- + */ + +#include "postgres.h" + +#include "catalog/namespace.h" +#include "catalog/pg_type.h" +#include "executor/spi.h" +#include "replication/logicalproto.h" +#include "replication/logicalworker.h" +#include "utils/builtins.h" +#include "utils/lsyscache.h" +#include "utils/pg_lsn.h" + +PG_MODULE_MAGIC; + +#define LOG_SCHEMA "public" +#define LOG_TABLE "test_logicalmsg_log" + +static LogicalRepMessageHandle_hook_type prev_logical_message_handler = NULL; +static void test_logical_message_handler(const LogicalRepMessageData *msg); + +/* + * Module load callback + */ +void +_PG_init(void) +{ + prev_logical_message_handler = LogicalRepMessageHandle_hook; + LogicalRepMessageHandle_hook = &test_logical_message_handler; +} + +/* + * Create the log table if not exists. + */ +static void +ensure_log_table_exists(void) +{ + Oid relid; + int ret; + + relid = get_relname_relid(LOG_TABLE, get_namespace_oid(LOG_SCHEMA, true)); + if (OidIsValid(relid)) + return; + + SPI_connect(); + + ret = SPI_execute("CREATE TABLE " LOG_SCHEMA "." LOG_TABLE + "(lsn pg_lsn, transactional boolean, prefix text, message_size int, message text)", + false, 0); + if (ret != SPI_OK_UTILITY) + elog(ERROR, "could not create " LOG_SCHEMA "." LOG_TABLE); + + SPI_finish(); +} + +static void +test_logical_message_handler(const LogicalRepMessageData *msg) +{ + Oid argtypes[5] = {LSNOID, BOOLOID, TEXTOID, INT4OID, TEXTOID}; + Datum values[5]; + int ret; + + /* + * Give a handler installed before ours its turn, so that several + * extensions can act on the same message. + */ + if (prev_logical_message_handler) + (*prev_logical_message_handler) (msg); + + ereport(LOG, + (errmsg("received message: LSN %X/%08X, prefix: %s, size: %zu, transactional: %d", + LSN_FORMAT_ARGS(msg->lsn), msg->prefix, + msg->message_size, msg->transactional))); + + ensure_log_table_exists(); + + /* + * Pass the message as query parameters rather than interpolating it into + * the query text. Message contents come from the publisher and are not to + * be trusted, and the payload may contain characters that would need + * quoting. Note also that message_size, not strlen(), is authoritative + * for the payload length. + * + * The payload is recorded as text because the accompanying test only ever + * emits text messages. A handler must not assume that in general: a + * payload is an arbitrary string of bytes, which may contain embedded + * nulls and bytes that are not valid in the subscriber's encoding. See + * the comments on LogicalRepMessageHandle_hook_type. + */ + values[0] = LSNGetDatum(msg->lsn); + values[1] = BoolGetDatum(msg->transactional); + values[2] = CStringGetTextDatum(msg->prefix); + values[3] = Int32GetDatum((int32) msg->message_size); + values[4] = PointerGetDatum(cstring_to_text_with_len(msg->message, + msg->message_size)); + + SPI_connect(); + + /* + * The apply worker runs with an empty search_path, so the table name must + * be schema-qualified. + */ + ret = SPI_execute_with_args("INSERT INTO " LOG_SCHEMA "." LOG_TABLE + " (lsn, transactional, prefix, message_size, message)" + " VALUES ($1, $2, $3, $4, $5)", + 5, argtypes, values, NULL, false, 0); + if (ret != SPI_OK_INSERT) + elog(ERROR, "could not record logical message: SPI returned %d", ret); + + SPI_finish(); +} diff --git a/src/test/regress/expected/subscription.out b/src/test/regress/expected/subscription.out index 6b2825a6a14..118e1ee3ce8 100644 --- a/src/test/regress/expected/subscription.out +++ b/src/test/regress/expected/subscription.out @@ -139,18 +139,18 @@ CREATE SUBSCRIPTION regress_testsub4 CONNECTION 'dbname=regress_doesnotexist' PU WARNING: subscription was created, but is not connected HINT: To initiate replication, you must manually create the replication slot, enable the subscription, and alter the subscription to refresh publications. \dRs+ regress_testsub4 - List of subscriptions - Name | Owner | Enabled | Publication | Binary | Streaming | Two-phase commit | Disable on error | Origin | Password required | Run as owner? | Failover | Server | Retain dead tuples | Max retention duration | Retention active | Synchronous commit | Conninfo | Receiver timeout | Skip LSN | Description -------------------+---------------------------+---------+-------------+--------+-----------+------------------+------------------+--------+-------------------+---------------+----------+--------+--------------------+------------------------+------------------+--------------------+-----------------------------+------------------+------------+------------- - regress_testsub4 | regress_subscription_user | f | {testpub} | f | parallel | d | f | none | t | f | f | | f | 0 | f | off | dbname=regress_doesnotexist | -1 | 0/00000000 | + List of subscriptions + Name | Owner | Enabled | Publication | Binary | Streaming | Two-phase commit | Disable on error | Origin | Password required | Run as owner? | Failover | Server | Retain dead tuples | Max retention duration | Retention active | Messages | Synchronous commit | Conninfo | Receiver timeout | Skip LSN | Description +------------------+---------------------------+---------+-------------+--------+-----------+------------------+------------------+--------+-------------------+---------------+----------+--------+--------------------+------------------------+------------------+----------+--------------------+-----------------------------+------------------+------------+------------- + regress_testsub4 | regress_subscription_user | f | {testpub} | f | parallel | d | f | none | t | f | f | | f | 0 | f | f | off | dbname=regress_doesnotexist | -1 | 0/00000000 | (1 row) ALTER SUBSCRIPTION regress_testsub4 SET (origin = any); \dRs+ regress_testsub4 - List of subscriptions - Name | Owner | Enabled | Publication | Binary | Streaming | Two-phase commit | Disable on error | Origin | Password required | Run as owner? | Failover | Server | Retain dead tuples | Max retention duration | Retention active | Synchronous commit | Conninfo | Receiver timeout | Skip LSN | Description -------------------+---------------------------+---------+-------------+--------+-----------+------------------+------------------+--------+-------------------+---------------+----------+--------+--------------------+------------------------+------------------+--------------------+-----------------------------+------------------+------------+------------- - regress_testsub4 | regress_subscription_user | f | {testpub} | f | parallel | d | f | any | t | f | f | | f | 0 | f | off | dbname=regress_doesnotexist | -1 | 0/00000000 | + List of subscriptions + Name | Owner | Enabled | Publication | Binary | Streaming | Two-phase commit | Disable on error | Origin | Password required | Run as owner? | Failover | Server | Retain dead tuples | Max retention duration | Retention active | Messages | Synchronous commit | Conninfo | Receiver timeout | Skip LSN | Description +------------------+---------------------------+---------+-------------+--------+-----------+------------------+------------------+--------+-------------------+---------------+----------+--------+--------------------+------------------------+------------------+----------+--------------------+-----------------------------+------------------+------------+------------- + regress_testsub4 | regress_subscription_user | f | {testpub} | f | parallel | d | f | any | t | f | f | | f | 0 | f | f | off | dbname=regress_doesnotexist | -1 | 0/00000000 | (1 row) DROP SUBSCRIPTION regress_testsub3; @@ -277,10 +277,10 @@ ALTER SUBSCRIPTION regress_testsub CONNECTION 'foobar'; ERROR: invalid connection string syntax: missing "=" after "foobar" in connection info string \dRs+ - List of subscriptions - Name | Owner | Enabled | Publication | Binary | Streaming | Two-phase commit | Disable on error | Origin | Password required | Run as owner? | Failover | Server | Retain dead tuples | Max retention duration | Retention active | Synchronous commit | Conninfo | Receiver timeout | Skip LSN | Description ------------------+---------------------------+---------+-------------+--------+-----------+------------------+------------------+--------+-------------------+---------------+----------+--------+--------------------+------------------------+------------------+--------------------+-----------------------------+------------------+------------+------------------- - regress_testsub | regress_subscription_user | f | {testpub} | f | parallel | d | f | any | t | f | f | | f | 0 | f | off | dbname=regress_doesnotexist | -1 | 0/00000000 | test subscription + List of subscriptions + Name | Owner | Enabled | Publication | Binary | Streaming | Two-phase commit | Disable on error | Origin | Password required | Run as owner? | Failover | Server | Retain dead tuples | Max retention duration | Retention active | Messages | Synchronous commit | Conninfo | Receiver timeout | Skip LSN | Description +-----------------+---------------------------+---------+-------------+--------+-----------+------------------+------------------+--------+-------------------+---------------+----------+--------+--------------------+------------------------+------------------+----------+--------------------+-----------------------------+------------------+------------+------------------- + regress_testsub | regress_subscription_user | f | {testpub} | f | parallel | d | f | any | t | f | f | | f | 0 | f | f | off | dbname=regress_doesnotexist | -1 | 0/00000000 | test subscription (1 row) ALTER SUBSCRIPTION regress_testsub SET PUBLICATION testpub2, testpub3 WITH (refresh = false); @@ -289,10 +289,10 @@ ALTER SUBSCRIPTION regress_testsub SET (slot_name = 'newname'); ALTER SUBSCRIPTION regress_testsub SET (password_required = false); ALTER SUBSCRIPTION regress_testsub SET (run_as_owner = true); \dRs+ - List of subscriptions - Name | Owner | Enabled | Publication | Binary | Streaming | Two-phase commit | Disable on error | Origin | Password required | Run as owner? | Failover | Server | Retain dead tuples | Max retention duration | Retention active | Synchronous commit | Conninfo | Receiver timeout | Skip LSN | Description ------------------+---------------------------+---------+---------------------+--------+-----------+------------------+------------------+--------+-------------------+---------------+----------+--------+--------------------+------------------------+------------------+--------------------+------------------------------+------------------+------------+------------------- - regress_testsub | regress_subscription_user | f | {testpub2,testpub3} | f | parallel | d | f | any | f | t | f | | f | 0 | f | off | dbname=regress_doesnotexist2 | -1 | 0/00000000 | test subscription + List of subscriptions + Name | Owner | Enabled | Publication | Binary | Streaming | Two-phase commit | Disable on error | Origin | Password required | Run as owner? | Failover | Server | Retain dead tuples | Max retention duration | Retention active | Messages | Synchronous commit | Conninfo | Receiver timeout | Skip LSN | Description +-----------------+---------------------------+---------+---------------------+--------+-----------+------------------+------------------+--------+-------------------+---------------+----------+--------+--------------------+------------------------+------------------+----------+--------------------+------------------------------+------------------+------------+------------------- + regress_testsub | regress_subscription_user | f | {testpub2,testpub3} | f | parallel | d | f | any | f | t | f | | f | 0 | f | f | off | dbname=regress_doesnotexist2 | -1 | 0/00000000 | test subscription (1 row) ALTER SUBSCRIPTION regress_testsub SET (password_required = true); @@ -308,10 +308,10 @@ ERROR: unrecognized subscription parameter: "create_slot" -- ok ALTER SUBSCRIPTION regress_testsub SKIP (lsn = '0/12345'); \dRs+ - List of subscriptions - Name | Owner | Enabled | Publication | Binary | Streaming | Two-phase commit | Disable on error | Origin | Password required | Run as owner? | Failover | Server | Retain dead tuples | Max retention duration | Retention active | Synchronous commit | Conninfo | Receiver timeout | Skip LSN | Description ------------------+---------------------------+---------+---------------------+--------+-----------+------------------+------------------+--------+-------------------+---------------+----------+--------+--------------------+------------------------+------------------+--------------------+------------------------------+------------------+------------+------------------- - regress_testsub | regress_subscription_user | f | {testpub2,testpub3} | f | parallel | d | f | any | t | f | f | | f | 0 | f | off | dbname=regress_doesnotexist2 | -1 | 0/00012345 | test subscription + List of subscriptions + Name | Owner | Enabled | Publication | Binary | Streaming | Two-phase commit | Disable on error | Origin | Password required | Run as owner? | Failover | Server | Retain dead tuples | Max retention duration | Retention active | Messages | Synchronous commit | Conninfo | Receiver timeout | Skip LSN | Description +-----------------+---------------------------+---------+---------------------+--------+-----------+------------------+------------------+--------+-------------------+---------------+----------+--------+--------------------+------------------------+------------------+----------+--------------------+------------------------------+------------------+------------+------------------- + regress_testsub | regress_subscription_user | f | {testpub2,testpub3} | f | parallel | d | f | any | t | f | f | | f | 0 | f | f | off | dbname=regress_doesnotexist2 | -1 | 0/00012345 | test subscription (1 row) -- ok - with lsn = NONE @@ -320,10 +320,10 @@ ALTER SUBSCRIPTION regress_testsub SKIP (lsn = NONE); ALTER SUBSCRIPTION regress_testsub SKIP (lsn = '0/0'); ERROR: invalid WAL location (LSN): 0/0 \dRs+ - List of subscriptions - Name | Owner | Enabled | Publication | Binary | Streaming | Two-phase commit | Disable on error | Origin | Password required | Run as owner? | Failover | Server | Retain dead tuples | Max retention duration | Retention active | Synchronous commit | Conninfo | Receiver timeout | Skip LSN | Description ------------------+---------------------------+---------+---------------------+--------+-----------+------------------+------------------+--------+-------------------+---------------+----------+--------+--------------------+------------------------+------------------+--------------------+------------------------------+------------------+------------+------------------- - regress_testsub | regress_subscription_user | f | {testpub2,testpub3} | f | parallel | d | f | any | t | f | f | | f | 0 | f | off | dbname=regress_doesnotexist2 | -1 | 0/00000000 | test subscription + List of subscriptions + Name | Owner | Enabled | Publication | Binary | Streaming | Two-phase commit | Disable on error | Origin | Password required | Run as owner? | Failover | Server | Retain dead tuples | Max retention duration | Retention active | Messages | Synchronous commit | Conninfo | Receiver timeout | Skip LSN | Description +-----------------+---------------------------+---------+---------------------+--------+-----------+------------------+------------------+--------+-------------------+---------------+----------+--------+--------------------+------------------------+------------------+----------+--------------------+------------------------------+------------------+------------+------------------- + regress_testsub | regress_subscription_user | f | {testpub2,testpub3} | f | parallel | d | f | any | t | f | f | | f | 0 | f | f | off | dbname=regress_doesnotexist2 | -1 | 0/00000000 | test subscription (1 row) BEGIN; @@ -359,10 +359,10 @@ ALTER SUBSCRIPTION regress_testsub_foo SET (wal_receiver_timeout = '80s'); ALTER SUBSCRIPTION regress_testsub_foo SET (wal_receiver_timeout = 'foobar'); ERROR: invalid value for parameter "wal_receiver_timeout": "foobar" \dRs+ - List of subscriptions - Name | Owner | Enabled | Publication | Binary | Streaming | Two-phase commit | Disable on error | Origin | Password required | Run as owner? | Failover | Server | Retain dead tuples | Max retention duration | Retention active | Synchronous commit | Conninfo | Receiver timeout | Skip LSN | Description ----------------------+---------------------------+---------+---------------------+--------+-----------+------------------+------------------+--------+-------------------+---------------+----------+--------+--------------------+------------------------+------------------+--------------------+------------------------------+------------------+------------+------------------- - regress_testsub_foo | regress_subscription_user | f | {testpub2,testpub3} | f | parallel | d | f | any | t | f | f | | f | 0 | f | local | dbname=regress_doesnotexist2 | 80s | 0/00000000 | test subscription + List of subscriptions + Name | Owner | Enabled | Publication | Binary | Streaming | Two-phase commit | Disable on error | Origin | Password required | Run as owner? | Failover | Server | Retain dead tuples | Max retention duration | Retention active | Messages | Synchronous commit | Conninfo | Receiver timeout | Skip LSN | Description +---------------------+---------------------------+---------+---------------------+--------+-----------+------------------+------------------+--------+-------------------+---------------+----------+--------+--------------------+------------------------+------------------+----------+--------------------+------------------------------+------------------+------------+------------------- + regress_testsub_foo | regress_subscription_user | f | {testpub2,testpub3} | f | parallel | d | f | any | t | f | f | | f | 0 | f | f | local | dbname=regress_doesnotexist2 | 80s | 0/00000000 | test subscription (1 row) -- rename back to keep the rest simple @@ -391,19 +391,19 @@ CREATE SUBSCRIPTION regress_testsub CONNECTION 'dbname=regress_doesnotexist' PUB WARNING: subscription was created, but is not connected HINT: To initiate replication, you must manually create the replication slot, enable the subscription, and alter the subscription to refresh publications. \dRs+ - List of subscriptions - Name | Owner | Enabled | Publication | Binary | Streaming | Two-phase commit | Disable on error | Origin | Password required | Run as owner? | Failover | Server | Retain dead tuples | Max retention duration | Retention active | Synchronous commit | Conninfo | Receiver timeout | Skip LSN | Description ------------------+---------------------------+---------+-------------+--------+-----------+------------------+------------------+--------+-------------------+---------------+----------+--------+--------------------+------------------------+------------------+--------------------+-----------------------------+------------------+------------+------------- - regress_testsub | regress_subscription_user | f | {testpub} | t | parallel | d | f | any | t | f | f | | f | 0 | f | off | dbname=regress_doesnotexist | -1 | 0/00000000 | + List of subscriptions + Name | Owner | Enabled | Publication | Binary | Streaming | Two-phase commit | Disable on error | Origin | Password required | Run as owner? | Failover | Server | Retain dead tuples | Max retention duration | Retention active | Messages | Synchronous commit | Conninfo | Receiver timeout | Skip LSN | Description +-----------------+---------------------------+---------+-------------+--------+-----------+------------------+------------------+--------+-------------------+---------------+----------+--------+--------------------+------------------------+------------------+----------+--------------------+-----------------------------+------------------+------------+------------- + regress_testsub | regress_subscription_user | f | {testpub} | t | parallel | d | f | any | t | f | f | | f | 0 | f | f | off | dbname=regress_doesnotexist | -1 | 0/00000000 | (1 row) ALTER SUBSCRIPTION regress_testsub SET (binary = false); ALTER SUBSCRIPTION regress_testsub SET (slot_name = NONE); \dRs+ - List of subscriptions - Name | Owner | Enabled | Publication | Binary | Streaming | Two-phase commit | Disable on error | Origin | Password required | Run as owner? | Failover | Server | Retain dead tuples | Max retention duration | Retention active | Synchronous commit | Conninfo | Receiver timeout | Skip LSN | Description ------------------+---------------------------+---------+-------------+--------+-----------+------------------+------------------+--------+-------------------+---------------+----------+--------+--------------------+------------------------+------------------+--------------------+-----------------------------+------------------+------------+------------- - regress_testsub | regress_subscription_user | f | {testpub} | f | parallel | d | f | any | t | f | f | | f | 0 | f | off | dbname=regress_doesnotexist | -1 | 0/00000000 | + List of subscriptions + Name | Owner | Enabled | Publication | Binary | Streaming | Two-phase commit | Disable on error | Origin | Password required | Run as owner? | Failover | Server | Retain dead tuples | Max retention duration | Retention active | Messages | Synchronous commit | Conninfo | Receiver timeout | Skip LSN | Description +-----------------+---------------------------+---------+-------------+--------+-----------+------------------+------------------+--------+-------------------+---------------+----------+--------+--------------------+------------------------+------------------+----------+--------------------+-----------------------------+------------------+------------+------------- + regress_testsub | regress_subscription_user | f | {testpub} | f | parallel | d | f | any | t | f | f | | f | 0 | f | f | off | dbname=regress_doesnotexist | -1 | 0/00000000 | (1 row) DROP SUBSCRIPTION regress_testsub; @@ -415,27 +415,27 @@ CREATE SUBSCRIPTION regress_testsub CONNECTION 'dbname=regress_doesnotexist' PUB WARNING: subscription was created, but is not connected HINT: To initiate replication, you must manually create the replication slot, enable the subscription, and alter the subscription to refresh publications. \dRs+ - List of subscriptions - Name | Owner | Enabled | Publication | Binary | Streaming | Two-phase commit | Disable on error | Origin | Password required | Run as owner? | Failover | Server | Retain dead tuples | Max retention duration | Retention active | Synchronous commit | Conninfo | Receiver timeout | Skip LSN | Description ------------------+---------------------------+---------+-------------+--------+-----------+------------------+------------------+--------+-------------------+---------------+----------+--------+--------------------+------------------------+------------------+--------------------+-----------------------------+------------------+------------+------------- - regress_testsub | regress_subscription_user | f | {testpub} | f | on | d | f | any | t | f | f | | f | 0 | f | off | dbname=regress_doesnotexist | -1 | 0/00000000 | + List of subscriptions + Name | Owner | Enabled | Publication | Binary | Streaming | Two-phase commit | Disable on error | Origin | Password required | Run as owner? | Failover | Server | Retain dead tuples | Max retention duration | Retention active | Messages | Synchronous commit | Conninfo | Receiver timeout | Skip LSN | Description +-----------------+---------------------------+---------+-------------+--------+-----------+------------------+------------------+--------+-------------------+---------------+----------+--------+--------------------+------------------------+------------------+----------+--------------------+-----------------------------+------------------+------------+------------- + regress_testsub | regress_subscription_user | f | {testpub} | f | on | d | f | any | t | f | f | | f | 0 | f | f | off | dbname=regress_doesnotexist | -1 | 0/00000000 | (1 row) ALTER SUBSCRIPTION regress_testsub SET (streaming = parallel); \dRs+ - List of subscriptions - Name | Owner | Enabled | Publication | Binary | Streaming | Two-phase commit | Disable on error | Origin | Password required | Run as owner? | Failover | Server | Retain dead tuples | Max retention duration | Retention active | Synchronous commit | Conninfo | Receiver timeout | Skip LSN | Description ------------------+---------------------------+---------+-------------+--------+-----------+------------------+------------------+--------+-------------------+---------------+----------+--------+--------------------+------------------------+------------------+--------------------+-----------------------------+------------------+------------+------------- - regress_testsub | regress_subscription_user | f | {testpub} | f | parallel | d | f | any | t | f | f | | f | 0 | f | off | dbname=regress_doesnotexist | -1 | 0/00000000 | + List of subscriptions + Name | Owner | Enabled | Publication | Binary | Streaming | Two-phase commit | Disable on error | Origin | Password required | Run as owner? | Failover | Server | Retain dead tuples | Max retention duration | Retention active | Messages | Synchronous commit | Conninfo | Receiver timeout | Skip LSN | Description +-----------------+---------------------------+---------+-------------+--------+-----------+------------------+------------------+--------+-------------------+---------------+----------+--------+--------------------+------------------------+------------------+----------+--------------------+-----------------------------+------------------+------------+------------- + regress_testsub | regress_subscription_user | f | {testpub} | f | parallel | d | f | any | t | f | f | | f | 0 | f | f | off | dbname=regress_doesnotexist | -1 | 0/00000000 | (1 row) ALTER SUBSCRIPTION regress_testsub SET (streaming = false); ALTER SUBSCRIPTION regress_testsub SET (slot_name = NONE); \dRs+ - List of subscriptions - Name | Owner | Enabled | Publication | Binary | Streaming | Two-phase commit | Disable on error | Origin | Password required | Run as owner? | Failover | Server | Retain dead tuples | Max retention duration | Retention active | Synchronous commit | Conninfo | Receiver timeout | Skip LSN | Description ------------------+---------------------------+---------+-------------+--------+-----------+------------------+------------------+--------+-------------------+---------------+----------+--------+--------------------+------------------------+------------------+--------------------+-----------------------------+------------------+------------+------------- - regress_testsub | regress_subscription_user | f | {testpub} | f | off | d | f | any | t | f | f | | f | 0 | f | off | dbname=regress_doesnotexist | -1 | 0/00000000 | + List of subscriptions + Name | Owner | Enabled | Publication | Binary | Streaming | Two-phase commit | Disable on error | Origin | Password required | Run as owner? | Failover | Server | Retain dead tuples | Max retention duration | Retention active | Messages | Synchronous commit | Conninfo | Receiver timeout | Skip LSN | Description +-----------------+---------------------------+---------+-------------+--------+-----------+------------------+------------------+--------+-------------------+---------------+----------+--------+--------------------+------------------------+------------------+----------+--------------------+-----------------------------+------------------+------------+------------- + regress_testsub | regress_subscription_user | f | {testpub} | f | off | d | f | any | t | f | f | | f | 0 | f | f | off | dbname=regress_doesnotexist | -1 | 0/00000000 | (1 row) -- fail - publication already exists @@ -450,10 +450,10 @@ ALTER SUBSCRIPTION regress_testsub ADD PUBLICATION testpub1, testpub2 WITH (refr ALTER SUBSCRIPTION regress_testsub ADD PUBLICATION testpub1, testpub2 WITH (refresh = false); ERROR: publication "testpub1" is already in subscription "regress_testsub" \dRs+ - List of subscriptions - Name | Owner | Enabled | Publication | Binary | Streaming | Two-phase commit | Disable on error | Origin | Password required | Run as owner? | Failover | Server | Retain dead tuples | Max retention duration | Retention active | Synchronous commit | Conninfo | Receiver timeout | Skip LSN | Description ------------------+---------------------------+---------+-----------------------------+--------+-----------+------------------+------------------+--------+-------------------+---------------+----------+--------+--------------------+------------------------+------------------+--------------------+-----------------------------+------------------+------------+------------- - regress_testsub | regress_subscription_user | f | {testpub,testpub1,testpub2} | f | off | d | f | any | t | f | f | | f | 0 | f | off | dbname=regress_doesnotexist | -1 | 0/00000000 | + List of subscriptions + Name | Owner | Enabled | Publication | Binary | Streaming | Two-phase commit | Disable on error | Origin | Password required | Run as owner? | Failover | Server | Retain dead tuples | Max retention duration | Retention active | Messages | Synchronous commit | Conninfo | Receiver timeout | Skip LSN | Description +-----------------+---------------------------+---------+-----------------------------+--------+-----------+------------------+------------------+--------+-------------------+---------------+----------+--------+--------------------+------------------------+------------------+----------+--------------------+-----------------------------+------------------+------------+------------- + regress_testsub | regress_subscription_user | f | {testpub,testpub1,testpub2} | f | off | d | f | any | t | f | f | | f | 0 | f | f | off | dbname=regress_doesnotexist | -1 | 0/00000000 | (1 row) -- fail - publication used more than once @@ -468,10 +468,10 @@ ERROR: publication "testpub3" is not in subscription "regress_testsub" -- ok - delete publications ALTER SUBSCRIPTION regress_testsub DROP PUBLICATION testpub1, testpub2 WITH (refresh = false); \dRs+ - List of subscriptions - Name | Owner | Enabled | Publication | Binary | Streaming | Two-phase commit | Disable on error | Origin | Password required | Run as owner? | Failover | Server | Retain dead tuples | Max retention duration | Retention active | Synchronous commit | Conninfo | Receiver timeout | Skip LSN | Description ------------------+---------------------------+---------+-------------+--------+-----------+------------------+------------------+--------+-------------------+---------------+----------+--------+--------------------+------------------------+------------------+--------------------+-----------------------------+------------------+------------+------------- - regress_testsub | regress_subscription_user | f | {testpub} | f | off | d | f | any | t | f | f | | f | 0 | f | off | dbname=regress_doesnotexist | -1 | 0/00000000 | + List of subscriptions + Name | Owner | Enabled | Publication | Binary | Streaming | Two-phase commit | Disable on error | Origin | Password required | Run as owner? | Failover | Server | Retain dead tuples | Max retention duration | Retention active | Messages | Synchronous commit | Conninfo | Receiver timeout | Skip LSN | Description +-----------------+---------------------------+---------+-------------+--------+-----------+------------------+------------------+--------+-------------------+---------------+----------+--------+--------------------+------------------------+------------------+----------+--------------------+-----------------------------+------------------+------------+------------- + regress_testsub | regress_subscription_user | f | {testpub} | f | off | d | f | any | t | f | f | | f | 0 | f | f | off | dbname=regress_doesnotexist | -1 | 0/00000000 | (1 row) DROP SUBSCRIPTION regress_testsub; @@ -507,19 +507,19 @@ CREATE SUBSCRIPTION regress_testsub CONNECTION 'dbname=regress_doesnotexist' PUB WARNING: subscription was created, but is not connected HINT: To initiate replication, you must manually create the replication slot, enable the subscription, and alter the subscription to refresh publications. \dRs+ - List of subscriptions - Name | Owner | Enabled | Publication | Binary | Streaming | Two-phase commit | Disable on error | Origin | Password required | Run as owner? | Failover | Server | Retain dead tuples | Max retention duration | Retention active | Synchronous commit | Conninfo | Receiver timeout | Skip LSN | Description ------------------+---------------------------+---------+-------------+--------+-----------+------------------+------------------+--------+-------------------+---------------+----------+--------+--------------------+------------------------+------------------+--------------------+-----------------------------+------------------+------------+------------- - regress_testsub | regress_subscription_user | f | {testpub} | f | parallel | p | f | any | t | f | f | | f | 0 | f | off | dbname=regress_doesnotexist | -1 | 0/00000000 | + List of subscriptions + Name | Owner | Enabled | Publication | Binary | Streaming | Two-phase commit | Disable on error | Origin | Password required | Run as owner? | Failover | Server | Retain dead tuples | Max retention duration | Retention active | Messages | Synchronous commit | Conninfo | Receiver timeout | Skip LSN | Description +-----------------+---------------------------+---------+-------------+--------+-----------+------------------+------------------+--------+-------------------+---------------+----------+--------+--------------------+------------------------+------------------+----------+--------------------+-----------------------------+------------------+------------+------------- + regress_testsub | regress_subscription_user | f | {testpub} | f | parallel | p | f | any | t | f | f | | f | 0 | f | f | off | dbname=regress_doesnotexist | -1 | 0/00000000 | (1 row) -- we can alter streaming when two_phase enabled ALTER SUBSCRIPTION regress_testsub SET (streaming = true); \dRs+ - List of subscriptions - Name | Owner | Enabled | Publication | Binary | Streaming | Two-phase commit | Disable on error | Origin | Password required | Run as owner? | Failover | Server | Retain dead tuples | Max retention duration | Retention active | Synchronous commit | Conninfo | Receiver timeout | Skip LSN | Description ------------------+---------------------------+---------+-------------+--------+-----------+------------------+------------------+--------+-------------------+---------------+----------+--------+--------------------+------------------------+------------------+--------------------+-----------------------------+------------------+------------+------------- - regress_testsub | regress_subscription_user | f | {testpub} | f | on | p | f | any | t | f | f | | f | 0 | f | off | dbname=regress_doesnotexist | -1 | 0/00000000 | + List of subscriptions + Name | Owner | Enabled | Publication | Binary | Streaming | Two-phase commit | Disable on error | Origin | Password required | Run as owner? | Failover | Server | Retain dead tuples | Max retention duration | Retention active | Messages | Synchronous commit | Conninfo | Receiver timeout | Skip LSN | Description +-----------------+---------------------------+---------+-------------+--------+-----------+------------------+------------------+--------+-------------------+---------------+----------+--------+--------------------+------------------------+------------------+----------+--------------------+-----------------------------+------------------+------------+------------- + regress_testsub | regress_subscription_user | f | {testpub} | f | on | p | f | any | t | f | f | | f | 0 | f | f | off | dbname=regress_doesnotexist | -1 | 0/00000000 | (1 row) ALTER SUBSCRIPTION regress_testsub SET (slot_name = NONE); @@ -529,10 +529,10 @@ CREATE SUBSCRIPTION regress_testsub CONNECTION 'dbname=regress_doesnotexist' PUB WARNING: subscription was created, but is not connected HINT: To initiate replication, you must manually create the replication slot, enable the subscription, and alter the subscription to refresh publications. \dRs+ - List of subscriptions - Name | Owner | Enabled | Publication | Binary | Streaming | Two-phase commit | Disable on error | Origin | Password required | Run as owner? | Failover | Server | Retain dead tuples | Max retention duration | Retention active | Synchronous commit | Conninfo | Receiver timeout | Skip LSN | Description ------------------+---------------------------+---------+-------------+--------+-----------+------------------+------------------+--------+-------------------+---------------+----------+--------+--------------------+------------------------+------------------+--------------------+-----------------------------+------------------+------------+------------- - regress_testsub | regress_subscription_user | f | {testpub} | f | on | p | f | any | t | f | f | | f | 0 | f | off | dbname=regress_doesnotexist | -1 | 0/00000000 | + List of subscriptions + Name | Owner | Enabled | Publication | Binary | Streaming | Two-phase commit | Disable on error | Origin | Password required | Run as owner? | Failover | Server | Retain dead tuples | Max retention duration | Retention active | Messages | Synchronous commit | Conninfo | Receiver timeout | Skip LSN | Description +-----------------+---------------------------+---------+-------------+--------+-----------+------------------+------------------+--------+-------------------+---------------+----------+--------+--------------------+------------------------+------------------+----------+--------------------+-----------------------------+------------------+------------+------------- + regress_testsub | regress_subscription_user | f | {testpub} | f | on | p | f | any | t | f | f | | f | 0 | f | f | off | dbname=regress_doesnotexist | -1 | 0/00000000 | (1 row) ALTER SUBSCRIPTION regress_testsub SET (slot_name = NONE); @@ -545,18 +545,18 @@ CREATE SUBSCRIPTION regress_testsub CONNECTION 'dbname=regress_doesnotexist' PUB WARNING: subscription was created, but is not connected HINT: To initiate replication, you must manually create the replication slot, enable the subscription, and alter the subscription to refresh publications. \dRs+ - List of subscriptions - Name | Owner | Enabled | Publication | Binary | Streaming | Two-phase commit | Disable on error | Origin | Password required | Run as owner? | Failover | Server | Retain dead tuples | Max retention duration | Retention active | Synchronous commit | Conninfo | Receiver timeout | Skip LSN | Description ------------------+---------------------------+---------+-------------+--------+-----------+------------------+------------------+--------+-------------------+---------------+----------+--------+--------------------+------------------------+------------------+--------------------+-----------------------------+------------------+------------+------------- - regress_testsub | regress_subscription_user | f | {testpub} | f | parallel | d | f | any | t | f | f | | f | 0 | f | off | dbname=regress_doesnotexist | -1 | 0/00000000 | + List of subscriptions + Name | Owner | Enabled | Publication | Binary | Streaming | Two-phase commit | Disable on error | Origin | Password required | Run as owner? | Failover | Server | Retain dead tuples | Max retention duration | Retention active | Messages | Synchronous commit | Conninfo | Receiver timeout | Skip LSN | Description +-----------------+---------------------------+---------+-------------+--------+-----------+------------------+------------------+--------+-------------------+---------------+----------+--------+--------------------+------------------------+------------------+----------+--------------------+-----------------------------+------------------+------------+------------- + regress_testsub | regress_subscription_user | f | {testpub} | f | parallel | d | f | any | t | f | f | | f | 0 | f | f | off | dbname=regress_doesnotexist | -1 | 0/00000000 | (1 row) ALTER SUBSCRIPTION regress_testsub SET (disable_on_error = true); \dRs+ - List of subscriptions - Name | Owner | Enabled | Publication | Binary | Streaming | Two-phase commit | Disable on error | Origin | Password required | Run as owner? | Failover | Server | Retain dead tuples | Max retention duration | Retention active | Synchronous commit | Conninfo | Receiver timeout | Skip LSN | Description ------------------+---------------------------+---------+-------------+--------+-----------+------------------+------------------+--------+-------------------+---------------+----------+--------+--------------------+------------------------+------------------+--------------------+-----------------------------+------------------+------------+------------- - regress_testsub | regress_subscription_user | f | {testpub} | f | parallel | d | t | any | t | f | f | | f | 0 | f | off | dbname=regress_doesnotexist | -1 | 0/00000000 | + List of subscriptions + Name | Owner | Enabled | Publication | Binary | Streaming | Two-phase commit | Disable on error | Origin | Password required | Run as owner? | Failover | Server | Retain dead tuples | Max retention duration | Retention active | Messages | Synchronous commit | Conninfo | Receiver timeout | Skip LSN | Description +-----------------+---------------------------+---------+-------------+--------+-----------+------------------+------------------+--------+-------------------+---------------+----------+--------+--------------------+------------------------+------------------+----------+--------------------+-----------------------------+------------------+------------+------------- + regress_testsub | regress_subscription_user | f | {testpub} | f | parallel | d | t | any | t | f | f | | f | 0 | f | f | off | dbname=regress_doesnotexist | -1 | 0/00000000 | (1 row) ALTER SUBSCRIPTION regress_testsub SET (slot_name = NONE); @@ -569,10 +569,10 @@ CREATE SUBSCRIPTION regress_testsub CONNECTION 'dbname=regress_doesnotexist' PUB WARNING: subscription was created, but is not connected HINT: To initiate replication, you must manually create the replication slot, enable the subscription, and alter the subscription to refresh publications. \dRs+ - List of subscriptions - Name | Owner | Enabled | Publication | Binary | Streaming | Two-phase commit | Disable on error | Origin | Password required | Run as owner? | Failover | Server | Retain dead tuples | Max retention duration | Retention active | Synchronous commit | Conninfo | Receiver timeout | Skip LSN | Description ------------------+---------------------------+---------+-------------+--------+-----------+------------------+------------------+--------+-------------------+---------------+----------+--------+--------------------+------------------------+------------------+--------------------+-----------------------------+------------------+------------+------------- - regress_testsub | regress_subscription_user | f | {testpub} | f | parallel | d | f | any | t | f | f | | f | 0 | f | off | dbname=regress_doesnotexist | -1 | 0/00000000 | + List of subscriptions + Name | Owner | Enabled | Publication | Binary | Streaming | Two-phase commit | Disable on error | Origin | Password required | Run as owner? | Failover | Server | Retain dead tuples | Max retention duration | Retention active | Messages | Synchronous commit | Conninfo | Receiver timeout | Skip LSN | Description +-----------------+---------------------------+---------+-------------+--------+-----------+------------------+------------------+--------+-------------------+---------------+----------+--------+--------------------+------------------------+------------------+----------+--------------------+-----------------------------+------------------+------------+------------- + regress_testsub | regress_subscription_user | f | {testpub} | f | parallel | d | f | any | t | f | f | | f | 0 | f | f | off | dbname=regress_doesnotexist | -1 | 0/00000000 | (1 row) ALTER SUBSCRIPTION regress_testsub SET (slot_name = NONE); @@ -589,10 +589,10 @@ NOTICE: max_retention_duration is ineffective when retain_dead_tuples is disabl WARNING: subscription was created, but is not connected HINT: To initiate replication, you must manually create the replication slot, enable the subscription, and alter the subscription to refresh publications. \dRs+ - List of subscriptions - Name | Owner | Enabled | Publication | Binary | Streaming | Two-phase commit | Disable on error | Origin | Password required | Run as owner? | Failover | Server | Retain dead tuples | Max retention duration | Retention active | Synchronous commit | Conninfo | Receiver timeout | Skip LSN | Description ------------------+---------------------------+---------+-------------+--------+-----------+------------------+------------------+--------+-------------------+---------------+----------+--------+--------------------+------------------------+------------------+--------------------+-----------------------------+------------------+------------+------------- - regress_testsub | regress_subscription_user | f | {testpub} | f | parallel | d | f | any | t | f | f | | f | 1000 | f | off | dbname=regress_doesnotexist | -1 | 0/00000000 | + List of subscriptions + Name | Owner | Enabled | Publication | Binary | Streaming | Two-phase commit | Disable on error | Origin | Password required | Run as owner? | Failover | Server | Retain dead tuples | Max retention duration | Retention active | Messages | Synchronous commit | Conninfo | Receiver timeout | Skip LSN | Description +-----------------+---------------------------+---------+-------------+--------+-----------+------------------+------------------+--------+-------------------+---------------+----------+--------+--------------------+------------------------+------------------+----------+--------------------+-----------------------------+------------------+------------+------------- + regress_testsub | regress_subscription_user | f | {testpub} | f | parallel | d | f | any | t | f | f | | f | 1000 | f | f | off | dbname=regress_doesnotexist | -1 | 0/00000000 | (1 row) -- fail - max_retention_duration must be non-negative @@ -601,12 +601,38 @@ ERROR: option "max_retention_duration" cannot be negative -- ok ALTER SUBSCRIPTION regress_testsub SET (max_retention_duration = 0); \dRs+ - List of subscriptions - Name | Owner | Enabled | Publication | Binary | Streaming | Two-phase commit | Disable on error | Origin | Password required | Run as owner? | Failover | Server | Retain dead tuples | Max retention duration | Retention active | Synchronous commit | Conninfo | Receiver timeout | Skip LSN | Description ------------------+---------------------------+---------+-------------+--------+-----------+------------------+------------------+--------+-------------------+---------------+----------+--------+--------------------+------------------------+------------------+--------------------+-----------------------------+------------------+------------+------------- - regress_testsub | regress_subscription_user | f | {testpub} | f | parallel | d | f | any | t | f | f | | f | 0 | f | off | dbname=regress_doesnotexist | -1 | 0/00000000 | + List of subscriptions + Name | Owner | Enabled | Publication | Binary | Streaming | Two-phase commit | Disable on error | Origin | Password required | Run as owner? | Failover | Server | Retain dead tuples | Max retention duration | Retention active | Messages | Synchronous commit | Conninfo | Receiver timeout | Skip LSN | Description +-----------------+---------------------------+---------+-------------+--------+-----------+------------------+------------------+--------+-------------------+---------------+----------+--------+--------------------+------------------------+------------------+----------+--------------------+-----------------------------+------------------+------------+------------- + regress_testsub | regress_subscription_user | f | {testpub} | f | parallel | d | f | any | t | f | f | | f | 0 | f | f | off | dbname=regress_doesnotexist | -1 | 0/00000000 | +(1 row) + +ALTER SUBSCRIPTION regress_testsub SET (slot_name = NONE); +DROP SUBSCRIPTION regress_testsub; +-- fail - messages must be boolean +CREATE SUBSCRIPTION regress_testsub CONNECTION 'dbname=regress_doesnotexist' PUBLICATION testpub WITH (connect = false, messages = foo); +ERROR: messages requires a Boolean value +-- ok +CREATE SUBSCRIPTION regress_testsub CONNECTION 'dbname=regress_doesnotexist' PUBLICATION testpub WITH (connect = false, messages = true); +WARNING: subscription was created, but is not connected +HINT: To initiate replication, you must manually create the replication slot, enable the subscription, and alter the subscription to refresh publications. +\dRs+ + List of subscriptions + Name | Owner | Enabled | Publication | Binary | Streaming | Two-phase commit | Disable on error | Origin | Password required | Run as owner? | Failover | Server | Retain dead tuples | Max retention duration | Retention active | Messages | Synchronous commit | Conninfo | Receiver timeout | Skip LSN | Description +-----------------+---------------------------+---------+-------------+--------+-----------+------------------+------------------+--------+-------------------+---------------+----------+--------+--------------------+------------------------+------------------+----------+--------------------+-----------------------------+------------------+------------+------------- + regress_testsub | regress_subscription_user | f | {testpub} | f | parallel | d | f | any | t | f | f | | f | 0 | f | t | off | dbname=regress_doesnotexist | -1 | 0/00000000 | +(1 row) + +-- ok +ALTER SUBSCRIPTION regress_testsub SET (messages = false); +\dRs+ + List of subscriptions + Name | Owner | Enabled | Publication | Binary | Streaming | Two-phase commit | Disable on error | Origin | Password required | Run as owner? | Failover | Server | Retain dead tuples | Max retention duration | Retention active | Messages | Synchronous commit | Conninfo | Receiver timeout | Skip LSN | Description +-----------------+---------------------------+---------+-------------+--------+-----------+------------------+------------------+--------+-------------------+---------------+----------+--------+--------------------+------------------------+------------------+----------+--------------------+-----------------------------+------------------+------------+------------- + regress_testsub | regress_subscription_user | f | {testpub} | f | parallel | d | f | any | t | f | f | | f | 0 | f | f | off | dbname=regress_doesnotexist | -1 | 0/00000000 | (1 row) +-- cleanup ALTER SUBSCRIPTION regress_testsub SET (slot_name = NONE); DROP SUBSCRIPTION regress_testsub; -- let's do some tests with pg_create_subscription rather than superuser diff --git a/src/test/regress/sql/subscription.sql b/src/test/regress/sql/subscription.sql index 1f34889a9a5..823b9318ead 100644 --- a/src/test/regress/sql/subscription.sql +++ b/src/test/regress/sql/subscription.sql @@ -457,6 +457,23 @@ ALTER SUBSCRIPTION regress_testsub SET (max_retention_duration = 0); ALTER SUBSCRIPTION regress_testsub SET (slot_name = NONE); DROP SUBSCRIPTION regress_testsub; +-- fail - messages must be boolean +CREATE SUBSCRIPTION regress_testsub CONNECTION 'dbname=regress_doesnotexist' PUBLICATION testpub WITH (connect = false, messages = foo); + +-- ok +CREATE SUBSCRIPTION regress_testsub CONNECTION 'dbname=regress_doesnotexist' PUBLICATION testpub WITH (connect = false, messages = true); + +\dRs+ + +-- ok +ALTER SUBSCRIPTION regress_testsub SET (messages = false); + +\dRs+ + +-- cleanup +ALTER SUBSCRIPTION regress_testsub SET (slot_name = NONE); +DROP SUBSCRIPTION regress_testsub; + -- let's do some tests with pg_create_subscription rather than superuser SET SESSION AUTHORIZATION regress_subscription_user3; diff --git a/src/tools/pgindent/typedefs.list b/src/tools/pgindent/typedefs.list index c21457c4887..179225ada82 100644 --- a/src/tools/pgindent/typedefs.list +++ b/src/tools/pgindent/typedefs.list @@ -1684,6 +1684,8 @@ LogicalRepBeginData LogicalRepCommitData LogicalRepCommitPreparedTxnData LogicalRepCtxStruct +LogicalRepMessageData +LogicalRepMessageHandle_hook_type LogicalRepMsgType LogicalRepPartMapEntry LogicalRepPreparedTxnData -- 2.56.0
v8-0001-Add-an-xid-argument-to-the-logical-decoding-messa.patch (text/x-patch, 19.3K) download From 587f0c596dbd392f78e2db433cd3c3fbb01793ef Mon Sep 17 00:00:00 2001 From: Zhijie Hou <houzhijie22@gmail.com> Date: Sat, 31 Oct 2026 19:55:25 +0800 Subject: [PATCH v8 1/2] Add an xid argument to the logical decoding message callbacks. Previously, when streaming an in-progress transaction, a transactional logical decoding message emitted in a subtransaction was passed to the stream_message callback with the top-level transaction only, and the callback had no way to know which subtransaction had emitted it. The change callbacks get that through ReorderBufferChange, but a message has none. pgoutput therefore sent the message with the top-level XID, and when the subtransaction was later rolled back and STREAM ABORT arrived for it, the subscriber could not discard the message with it. Oversight in commit 7259736a6e5. To fix this, we add the XID of the (sub)transaction that emitted the message as a new argument of both the message and stream_message callbacks, and have pgoutput send it. Output plugins can now see which subtransaction a message belongs to, as they already can for the other change types, while still getting the top-level transaction as before. Both callbacks get the argument so that a plugin can keep implementing them with one function, as pgoutput does. We considered fixing this in a backpatchable way, such as passing the subtransaction's ReorderBufferTXN to the callback instead of the top-level one, but that silently changes the meaning of an existing argument, and an output plugin that keeps its state on the top-level transaction would break. Adding an argument breaks the ABI, but it is the cleaner interface, so we chose it and don't backpatch. The bug exists in the back branches too, but there have been no reports, and the built-in logical replication doesn't use logical messages there. Author: Zhijie Hou <houzhijie22@gmail.com> Co-authored-by: Masahiko Sawada <sawada.mshk@gmail.com> Reviewed-by: Amit Kapila <amit.kapila16@gmail.com> Discussion: https://postgr.es/m/b9b5bd27-6416-48b4-8f73-c5a60e2c97e2@gmail.com --- contrib/test_decoding/expected/stream.out | 28 +++++++++++++++++++ contrib/test_decoding/sql/stream.sql | 20 +++++++++++++ contrib/test_decoding/test_decoding.c | 22 ++++++++------- doc/src/sgml/logicaldecoding.sgml | 14 +++++++++- src/backend/replication/logical/logical.c | 14 +++++----- .../replication/logical/reorderbuffer.c | 11 ++++---- src/backend/replication/pgoutput/pgoutput.c | 17 +++++------ src/include/replication/output_plugin.h | 10 +++++-- src/include/replication/reorderbuffer.h | 2 ++ 9 files changed, 105 insertions(+), 33 deletions(-) diff --git a/contrib/test_decoding/expected/stream.out b/contrib/test_decoding/expected/stream.out index 0ec5c933610..f34a07ba761 100644 --- a/contrib/test_decoding/expected/stream.out +++ b/contrib/test_decoding/expected/stream.out @@ -157,6 +157,34 @@ SELECT data FROM pg_logical_slot_get_changes('regression_slot', NULL, NULL, 'inc COMMIT (3 rows) +-- Verify that a transactional message emitted in a streamed subtransaction +-- is associated with the subtransaction's XID. +SET debug_logical_replication_streaming = immediate; +BEGIN; +SAVEPOINT sp; +SELECT 'msg_subxact' FROM pg_logical_emit_message(true, 'test', 'msg'); + ?column? +------------- + msg_subxact +(1 row) + +ROLLBACK TO SAVEPOINT sp; +COMMIT; +WITH changes AS ( + SELECT xid, data + FROM pg_logical_slot_get_changes('regression_slot', NULL, NULL, + 'include-xids', '0', 'skip-empty-xacts', '1', 'stream-changes', '1') +) +SELECT m.xid = a.xid AS subxid_matches +FROM changes m, changes a +WHERE m.data LIKE 'streaming message:%' + AND a.data = 'aborting streamed (sub)transaction'; + subxid_matches +---------------- + t +(1 row) + +RESET debug_logical_replication_streaming; DROP TABLE stream_test; SELECT pg_drop_replication_slot('regression_slot'); pg_drop_replication_slot diff --git a/contrib/test_decoding/sql/stream.sql b/contrib/test_decoding/sql/stream.sql index 5e45a8e9b64..ec68efed421 100644 --- a/contrib/test_decoding/sql/stream.sql +++ b/contrib/test_decoding/sql/stream.sql @@ -82,5 +82,25 @@ ROLLBACK; INSERT INTO stream_test VALUES ('after-abort'); SELECT data FROM pg_logical_slot_get_changes('regression_slot', NULL, NULL, 'include-xids', '0', 'skip-empty-xacts', '1'); +-- Verify that a transactional message emitted in a streamed subtransaction +-- is associated with the subtransaction's XID. +SET debug_logical_replication_streaming = immediate; +BEGIN; +SAVEPOINT sp; +SELECT 'msg_subxact' FROM pg_logical_emit_message(true, 'test', 'msg'); +ROLLBACK TO SAVEPOINT sp; +COMMIT; + +WITH changes AS ( + SELECT xid, data + FROM pg_logical_slot_get_changes('regression_slot', NULL, NULL, + 'include-xids', '0', 'skip-empty-xacts', '1', 'stream-changes', '1') +) +SELECT m.xid = a.xid AS subxid_matches +FROM changes m, changes a +WHERE m.data LIKE 'streaming message:%' + AND a.data = 'aborting streamed (sub)transaction'; +RESET debug_logical_replication_streaming; + DROP TABLE stream_test; SELECT pg_drop_replication_slot('regression_slot'); diff --git a/contrib/test_decoding/test_decoding.c b/contrib/test_decoding/test_decoding.c index 78185837aef..0f6e05b3bef 100644 --- a/contrib/test_decoding/test_decoding.c +++ b/contrib/test_decoding/test_decoding.c @@ -72,9 +72,9 @@ static void pg_decode_truncate(LogicalDecodingContext *ctx, static bool pg_decode_filter(LogicalDecodingContext *ctx, ReplOriginId origin_id); static void pg_decode_message(LogicalDecodingContext *ctx, - ReorderBufferTXN *txn, XLogRecPtr lsn, - bool transactional, const char *prefix, - Size sz, const char *message); + ReorderBufferTXN *txn, TransactionId xid, + XLogRecPtr lsn, bool transactional, + const char *prefix, Size sz, const char *message); static bool pg_decode_filter_prepare(LogicalDecodingContext *ctx, TransactionId xid, const char *gid); @@ -112,9 +112,10 @@ static void pg_decode_stream_change(LogicalDecodingContext *ctx, Relation relation, ReorderBufferChange *change); static void pg_decode_stream_message(LogicalDecodingContext *ctx, - ReorderBufferTXN *txn, XLogRecPtr lsn, - bool transactional, const char *prefix, - Size sz, const char *message); + ReorderBufferTXN *txn, TransactionId xid, + XLogRecPtr lsn, bool transactional, + const char *prefix, Size sz, + const char *message); static void pg_decode_stream_truncate(LogicalDecodingContext *ctx, ReorderBufferTXN *txn, int nrelations, Relation relations[], @@ -753,8 +754,9 @@ pg_decode_truncate(LogicalDecodingContext *ctx, ReorderBufferTXN *txn, static void pg_decode_message(LogicalDecodingContext *ctx, - ReorderBufferTXN *txn, XLogRecPtr lsn, bool transactional, - const char *prefix, Size sz, const char *message) + ReorderBufferTXN *txn, TransactionId xid, XLogRecPtr lsn, + bool transactional, const char *prefix, Size sz, + const char *message) { TestDecodingData *data = ctx->output_plugin_private; TestDecodingTxnData *txndata; @@ -954,8 +956,8 @@ pg_decode_stream_change(LogicalDecodingContext *ctx, */ static void pg_decode_stream_message(LogicalDecodingContext *ctx, - ReorderBufferTXN *txn, XLogRecPtr lsn, bool transactional, - const char *prefix, Size sz, const char *message) + ReorderBufferTXN *txn, TransactionId xid, XLogRecPtr lsn, + bool transactional, const char *prefix, Size sz, const char *message) { /* Output stream start if we haven't yet for transactional messages. */ if (transactional) diff --git a/doc/src/sgml/logicaldecoding.sgml b/doc/src/sgml/logicaldecoding.sgml index 6dc49108997..f21e1115a94 100644 --- a/doc/src/sgml/logicaldecoding.sgml +++ b/doc/src/sgml/logicaldecoding.sgml @@ -1128,6 +1128,7 @@ typedef bool (*LogicalDecodeFilterByOriginCB) (struct LogicalDecodingContext *ct <programlisting> typedef void (*LogicalDecodeMessageCB) (struct LogicalDecodingContext *ctx, ReorderBufferTXN *txn, + TransactionId xid, XLogRecPtr message_lsn, bool transactional, const char *prefix, @@ -1138,7 +1139,11 @@ typedef void (*LogicalDecodeMessageCB) (struct LogicalDecodingContext *ctx, the transaction, like the time stamp at which it has been committed and its XID. Note however that it can be NULL when the message is non-transactional and the XID was not assigned yet in the transaction - which logged the message. The <parameter>lsn</parameter> has WAL + which logged the message. The <parameter>xid</parameter> parameter is + the XID of the transaction that emitted the message; for a + transactional message emitted in a subtransaction, it is the + subtransaction's XID while <parameter>txn</parameter> is the top-level + transaction. The <parameter>lsn</parameter> has WAL location of the message. The <parameter>transactional</parameter> says if the message was sent as transactional or not. Similar to the change callback, in case of decoding a prepared (but yet uncommitted) @@ -1360,12 +1365,19 @@ typedef void (*LogicalDecodeStreamChangeCB) (struct LogicalDecodingContext *ctx, <programlisting> typedef void (*LogicalDecodeStreamMessageCB) (struct LogicalDecodingContext *ctx, ReorderBufferTXN *txn, + TransactionId xid, XLogRecPtr message_lsn, bool transactional, const char *prefix, Size message_size, const char *message); </programlisting> + The <parameter>txn</parameter> parameter is the top-level transaction + being streamed, and <parameter>xid</parameter> is the XID of the + transaction that emitted the message, which is a subtransaction's XID + if it was emitted in one. For the other stream callbacks, the XID of + the subtransaction is available as + <literal>change-&gt;txn-&gt;xid</literal>. </para> </sect3> diff --git a/src/backend/replication/logical/logical.c b/src/backend/replication/logical/logical.c index 8fe255fa960..44a27d0b653 100644 --- a/src/backend/replication/logical/logical.c +++ b/src/backend/replication/logical/logical.c @@ -81,7 +81,7 @@ static void change_cb_wrapper(ReorderBuffer *cache, ReorderBufferTXN *txn, static void truncate_cb_wrapper(ReorderBuffer *cache, ReorderBufferTXN *txn, int nrelations, Relation relations[], ReorderBufferChange *change); static void message_cb_wrapper(ReorderBuffer *cache, ReorderBufferTXN *txn, - XLogRecPtr message_lsn, bool transactional, + TransactionId xid, XLogRecPtr message_lsn, bool transactional, const char *prefix, Size message_size, const char *message); /* streaming callbacks */ @@ -98,7 +98,7 @@ static void stream_commit_cb_wrapper(ReorderBuffer *cache, ReorderBufferTXN *txn static void stream_change_cb_wrapper(ReorderBuffer *cache, ReorderBufferTXN *txn, Relation relation, ReorderBufferChange *change); static void stream_message_cb_wrapper(ReorderBuffer *cache, ReorderBufferTXN *txn, - XLogRecPtr message_lsn, bool transactional, + TransactionId xid, XLogRecPtr message_lsn, bool transactional, const char *prefix, Size message_size, const char *message); static void stream_truncate_cb_wrapper(ReorderBuffer *cache, ReorderBufferTXN *txn, int nrelations, Relation relations[], ReorderBufferChange *change); @@ -1336,7 +1336,7 @@ filter_by_origin_cb_wrapper(LogicalDecodingContext *ctx, ReplOriginId origin_id) static void message_cb_wrapper(ReorderBuffer *cache, ReorderBufferTXN *txn, - XLogRecPtr message_lsn, bool transactional, + TransactionId xid, XLogRecPtr message_lsn, bool transactional, const char *prefix, Size message_size, const char *message) { LogicalDecodingContext *ctx = cache->private_data; @@ -1364,7 +1364,7 @@ message_cb_wrapper(ReorderBuffer *cache, ReorderBufferTXN *txn, ctx->end_xact = false; /* do the actual work: call callback */ - ctx->callbacks.message_cb(ctx, txn, message_lsn, transactional, prefix, + ctx->callbacks.message_cb(ctx, txn, xid, message_lsn, transactional, prefix, message_size, message); /* Pop the error context stack */ @@ -1647,7 +1647,7 @@ stream_change_cb_wrapper(ReorderBuffer *cache, ReorderBufferTXN *txn, static void stream_message_cb_wrapper(ReorderBuffer *cache, ReorderBufferTXN *txn, - XLogRecPtr message_lsn, bool transactional, + TransactionId xid, XLogRecPtr message_lsn, bool transactional, const char *prefix, Size message_size, const char *message) { LogicalDecodingContext *ctx = cache->private_data; @@ -1674,12 +1674,12 @@ stream_message_cb_wrapper(ReorderBuffer *cache, ReorderBufferTXN *txn, /* set output state */ ctx->accept_writes = true; - ctx->write_xid = txn != NULL ? txn->xid : InvalidTransactionId; + ctx->write_xid = xid; ctx->write_location = message_lsn; ctx->end_xact = false; /* do the actual work: call callback */ - ctx->callbacks.stream_message_cb(ctx, txn, message_lsn, transactional, prefix, + ctx->callbacks.stream_message_cb(ctx, txn, xid, message_lsn, transactional, prefix, message_size, message); /* Pop the error context stack */ diff --git a/src/backend/replication/logical/reorderbuffer.c b/src/backend/replication/logical/reorderbuffer.c index 46843985317..2917d971f17 100644 --- a/src/backend/replication/logical/reorderbuffer.c +++ b/src/backend/replication/logical/reorderbuffer.c @@ -920,7 +920,8 @@ ReorderBufferQueueMessage(ReorderBuffer *rb, TransactionId xid, SetupHistoricSnapshot(snapshot_now, NULL); PG_TRY(); { - rb->message(rb, txn, lsn, false, prefix, message_size, message); + rb->message(rb, txn, xid, lsn, false, prefix, message_size, + message); TeardownHistoricSnapshot(false); } @@ -2104,13 +2105,13 @@ ReorderBufferApplyMessage(ReorderBuffer *rb, ReorderBufferTXN *txn, ReorderBufferChange *change, bool streaming) { if (streaming) - rb->stream_message(rb, txn, change->lsn, true, - change->data.msg.prefix, + rb->stream_message(rb, txn, change->txn->xid, change->lsn, + true, change->data.msg.prefix, change->data.msg.message_size, change->data.msg.message); else - rb->message(rb, txn, change->lsn, true, - change->data.msg.prefix, + rb->message(rb, txn, change->txn->xid, change->lsn, + true, change->data.msg.prefix, change->data.msg.message_size, change->data.msg.message); } diff --git a/src/backend/replication/pgoutput/pgoutput.c b/src/backend/replication/pgoutput/pgoutput.c index c8caa163c15..1c0a9cf09b9 100644 --- a/src/backend/replication/pgoutput/pgoutput.c +++ b/src/backend/replication/pgoutput/pgoutput.c @@ -55,9 +55,10 @@ static void pgoutput_truncate(LogicalDecodingContext *ctx, ReorderBufferTXN *txn, int nrelations, Relation relations[], ReorderBufferChange *change); static void pgoutput_message(LogicalDecodingContext *ctx, - ReorderBufferTXN *txn, XLogRecPtr message_lsn, - bool transactional, const char *prefix, - Size sz, const char *message); + ReorderBufferTXN *txn, TransactionId xid, + XLogRecPtr message_lsn, bool transactional, + const char *prefix, Size sz, + const char *message); static bool pgoutput_origin_filter(LogicalDecodingContext *ctx, ReplOriginId origin_id); static void pgoutput_begin_prepare_txn(LogicalDecodingContext *ctx, @@ -1722,11 +1723,11 @@ pgoutput_truncate(LogicalDecodingContext *ctx, ReorderBufferTXN *txn, static void pgoutput_message(LogicalDecodingContext *ctx, ReorderBufferTXN *txn, - XLogRecPtr message_lsn, bool transactional, const char *prefix, Size sz, - const char *message) + TransactionId xid, XLogRecPtr message_lsn, bool transactional, + const char *prefix, Size sz, const char *message) { PGOutputData *data = (PGOutputData *) ctx->output_plugin_private; - TransactionId xid = InvalidTransactionId; + TransactionId stream_xid = InvalidTransactionId; if (!data->messages) return; @@ -1736,7 +1737,7 @@ pgoutput_message(LogicalDecodingContext *ctx, ReorderBufferTXN *txn, * pgoutput_change. */ if (data->in_streaming) - xid = txn->xid; + stream_xid = xid; /* * Output BEGIN if we haven't yet. Avoid for non-transactional messages. @@ -1752,7 +1753,7 @@ pgoutput_message(LogicalDecodingContext *ctx, ReorderBufferTXN *txn, OutputPluginPrepareWrite(ctx, true); logicalrep_write_message(ctx->out, - xid, + stream_xid, message_lsn, transactional, prefix, diff --git a/src/include/replication/output_plugin.h b/src/include/replication/output_plugin.h index 842fcde67f9..2f12fe09ea2 100644 --- a/src/include/replication/output_plugin.h +++ b/src/include/replication/output_plugin.h @@ -80,10 +80,14 @@ typedef void (*LogicalDecodeCommitCB) (struct LogicalDecodingContext *ctx, XLogRecPtr commit_lsn); /* - * Called for the generic logical decoding messages. + * Called for the generic logical decoding messages. "xid" is the XID of the + * (sub)transaction that emitted the message; for a transactional message + * emitted in a subtransaction it differs from txn->xid, as txn is the + * top-level transaction. */ typedef void (*LogicalDecodeMessageCB) (struct LogicalDecodingContext *ctx, ReorderBufferTXN *txn, + TransactionId xid, XLogRecPtr message_lsn, bool transactional, const char *prefix, @@ -191,10 +195,12 @@ typedef void (*LogicalDecodeStreamChangeCB) (struct LogicalDecodingContext *ctx, /* * Callback for streaming generic logical decoding messages from in-progress - * transactions. + * transactions. "txn" is the top-level transaction and "xid" the XID of the + * (sub)transaction that emitted the message. */ typedef void (*LogicalDecodeStreamMessageCB) (struct LogicalDecodingContext *ctx, ReorderBufferTXN *txn, + TransactionId xid, XLogRecPtr message_lsn, bool transactional, const char *prefix, diff --git a/src/include/replication/reorderbuffer.h b/src/include/replication/reorderbuffer.h index cc47b472db3..612a7a81f40 100644 --- a/src/include/replication/reorderbuffer.h +++ b/src/include/replication/reorderbuffer.h @@ -500,6 +500,7 @@ typedef void (*ReorderBufferCommitCB) (ReorderBuffer *rb, /* message callback signature */ typedef void (*ReorderBufferMessageCB) (ReorderBuffer *rb, ReorderBufferTXN *txn, + TransactionId xid, XLogRecPtr message_lsn, bool transactional, const char *prefix, Size sz, @@ -559,6 +560,7 @@ typedef void (*ReorderBufferStreamChangeCB) (ReorderBuffer *rb, /* stream message callback signature */ typedef void (*ReorderBufferStreamMessageCB) (ReorderBuffer *rb, ReorderBufferTXN *txn, + TransactionId xid, XLogRecPtr message_lsn, bool transactional, const char *prefix, Size sz, -- 2.56.0
^ permalink raw reply [nested|flat] 44+ messages in thread

* Re: Add a hook for handling logical decoding messages on subscribers.
@ 2026-11-15 17:13  Zhijie Hou <houzhijie22@gmail.com>
  parent: Masahiko Sawada <sawada.mshk@gmail.com>
  0 siblings, 1 reply; 44+ messages in thread

From: Zhijie Hou @ 2026-11-15 17:13 UTC (permalink / raw)
  To: Masahiko Sawada <sawada.mshk@gmail.com>; Bertrand Drouvot <bertranddrouvot.pg@gmail.com>; +Cc: Amit Kapila <amit.kapila16@gmail.com>; Bharath Rupireddy <bharath.rupireddyforpostgres@gmail.com>; Chao Li <li.evan.chao@gmail.com>; Fujii Masao <masao.fujii@gmail.com>; pgsql-hackers

Hi,

On Sat, Nov14, 2026 at 07:35 AM Masahiko Sawada wrote:
>
> Agreed. It's better for callbacks to get both top-level and
> subtransaction's ids.
>
> I've updated the patches (the 0002 patch unchanged).

 > -	ctx->write_xid = txn != NULL ? txn->xid : InvalidTransactionId;
 > +	ctx->write_xid = xid;

I think it would be better to keep the original code for consistency, as 
all the other callbacks set ctx->write_xid to txn->xid (the top-level 
transaction's XID). IIUC, this was changed to make the test pass since 
test_decoding does not otherwise output the subxid, but we could instead 
use pgoutput to test that and keep passing the top-level XID in 
ctx->write_xid. Please see the attached top-up patch for this change.


BTW, since we are changing the callback signature anyway, another wild 
idea would be to pass ReorderBufferChange directly instead so that it is 
fully consistent with the other change callbacks. That said, it would 
require more changes in third-party output plugins, so I don't insist on 
it if you disagree — just sharing for reference.

-- 
Best Regards,
Zhijie Hou

Attachments:

nocfbot-0001-Top-up-for-v8-0001-Fix-streamed-block-XID-in-stream_.patch (text/plain, 5.2K) download From 952f55c52d9706be6f34f9c2554f4bd748f845f2 Mon Sep 17 00:00:00 2001 From: Zhijie Hou <houzhijie22@gmail.com> Date: Mon, 16 Nov 2026 00:41:24 +0800 Subject: [PATCH] Top-up for v8-0001: Fix streamed block XID in stream_message_cb_wrapper. Author: Zhijie Hou <houzhijie22@gmail.com> --- contrib/test_decoding/expected/stream.out | 31 ++++++++++++++++++----- contrib/test_decoding/sql/stream.sql | 21 ++++++++++----- src/backend/replication/logical/logical.c | 2 +- 3 files changed, 41 insertions(+), 13 deletions(-) diff --git a/contrib/test_decoding/expected/stream.out b/contrib/test_decoding/expected/stream.out index f34a07ba761..d589c7a2b3f 100644 --- a/contrib/test_decoding/expected/stream.out +++ b/contrib/test_decoding/expected/stream.out @@ -159,6 +159,13 @@ SELECT data FROM pg_logical_slot_get_changes('regression_slot', NULL, NULL, 'inc -- Verify that a transactional message emitted in a streamed subtransaction -- is associated with the subtransaction's XID. +CREATE PUBLICATION stream_pub; +SELECT 'init' FROM pg_create_logical_replication_slot('regression_pgoutput_slot', 'pgoutput'); + ?column? +---------- + init +(1 row) + SET debug_logical_replication_streaming = immediate; BEGIN; SAVEPOINT sp; @@ -170,21 +177,33 @@ SELECT 'msg_subxact' FROM pg_logical_emit_message(true, 'test', 'msg'); ROLLBACK TO SAVEPOINT sp; COMMIT; +-- Check that the streamed MESSAGE ('M') carries the aborted subtransaction's +-- XID rather than the top-level XID: in pgoutput's wire format, 'M' stores +-- its 4-byte XID at bytes 2..5, while STREAM ABORT ('A') stores the top-level +-- XID at bytes 2..5 and the aborted subtransaction's XID at bytes 6..9. WITH changes AS ( - SELECT xid, data - FROM pg_logical_slot_get_changes('regression_slot', NULL, NULL, - 'include-xids', '0', 'skip-empty-xacts', '1', 'stream-changes', '1') + SELECT data + FROM pg_logical_slot_get_binary_changes('regression_pgoutput_slot', NULL, NULL, + 'proto_version', '2', 'publication_names', 'stream_pub', + 'messages', 'true', 'streaming', 'on') ) -SELECT m.xid = a.xid AS subxid_matches +SELECT substr(m.data, 2, 4) = substr(a.data, 6, 4) AS subxid_matches FROM changes m, changes a -WHERE m.data LIKE 'streaming message:%' - AND a.data = 'aborting streamed (sub)transaction'; +WHERE get_byte(m.data, 0) = ascii('M') + AND get_byte(a.data, 0) = ascii('A'); subxid_matches ---------------- t (1 row) RESET debug_logical_replication_streaming; +SELECT pg_drop_replication_slot('regression_pgoutput_slot'); + pg_drop_replication_slot +-------------------------- + +(1 row) + +DROP PUBLICATION stream_pub; DROP TABLE stream_test; SELECT pg_drop_replication_slot('regression_slot'); pg_drop_replication_slot diff --git a/contrib/test_decoding/sql/stream.sql b/contrib/test_decoding/sql/stream.sql index ec68efed421..6cef3c83b09 100644 --- a/contrib/test_decoding/sql/stream.sql +++ b/contrib/test_decoding/sql/stream.sql @@ -84,6 +84,8 @@ SELECT data FROM pg_logical_slot_get_changes('regression_slot', NULL, NULL, 'inc -- Verify that a transactional message emitted in a streamed subtransaction -- is associated with the subtransaction's XID. +CREATE PUBLICATION stream_pub; +SELECT 'init' FROM pg_create_logical_replication_slot('regression_pgoutput_slot', 'pgoutput'); SET debug_logical_replication_streaming = immediate; BEGIN; SAVEPOINT sp; @@ -91,16 +93,23 @@ SELECT 'msg_subxact' FROM pg_logical_emit_message(true, 'test', 'msg'); ROLLBACK TO SAVEPOINT sp; COMMIT; +-- Check that the streamed MESSAGE ('M') carries the aborted subtransaction's +-- XID rather than the top-level XID: in pgoutput's wire format, 'M' stores +-- its 4-byte XID at bytes 2..5, while STREAM ABORT ('A') stores the top-level +-- XID at bytes 2..5 and the aborted subtransaction's XID at bytes 6..9. WITH changes AS ( - SELECT xid, data - FROM pg_logical_slot_get_changes('regression_slot', NULL, NULL, - 'include-xids', '0', 'skip-empty-xacts', '1', 'stream-changes', '1') + SELECT data + FROM pg_logical_slot_get_binary_changes('regression_pgoutput_slot', NULL, NULL, + 'proto_version', '2', 'publication_names', 'stream_pub', + 'messages', 'true', 'streaming', 'on') ) -SELECT m.xid = a.xid AS subxid_matches +SELECT substr(m.data, 2, 4) = substr(a.data, 6, 4) AS subxid_matches FROM changes m, changes a -WHERE m.data LIKE 'streaming message:%' - AND a.data = 'aborting streamed (sub)transaction'; +WHERE get_byte(m.data, 0) = ascii('M') + AND get_byte(a.data, 0) = ascii('A'); RESET debug_logical_replication_streaming; +SELECT pg_drop_replication_slot('regression_pgoutput_slot'); +DROP PUBLICATION stream_pub; DROP TABLE stream_test; SELECT pg_drop_replication_slot('regression_slot'); diff --git a/src/backend/replication/logical/logical.c b/src/backend/replication/logical/logical.c index 44a27d0b653..95760981b26 100644 --- a/src/backend/replication/logical/logical.c +++ b/src/backend/replication/logical/logical.c @@ -1674,7 +1674,7 @@ stream_message_cb_wrapper(ReorderBuffer *cache, ReorderBufferTXN *txn, /* set output state */ ctx->accept_writes = true; - ctx->write_xid = xid; + ctx->write_xid = txn != NULL ? txn->xid : InvalidTransactionId; ctx->write_location = message_lsn; ctx->end_xact = false; -- 2.54.0 (Apple Git-157)
^ permalink raw reply [nested|flat] 44+ messages in thread

* Re: Add a hook for handling logical decoding messages on subscribers.
@ 2026-11-16 19:05  Masahiko Sawada <sawada.mshk@gmail.com>
  parent: Zhijie Hou <houzhijie22@gmail.com>
  0 siblings, 1 reply; 44+ messages in thread

From: Masahiko Sawada @ 2026-11-16 19:05 UTC (permalink / raw)
  To: Zhijie Hou <houzhijie22@gmail.com>; +Cc: Bertrand Drouvot <bertranddrouvot.pg@gmail.com>; Amit Kapila <amit.kapila16@gmail.com>; Bharath Rupireddy <bharath.rupireddyforpostgres@gmail.com>; Chao Li <li.evan.chao@gmail.com>; Fujii Masao <masao.fujii@gmail.com>; pgsql-hackers

On Sun, Nov 15, 2026 at 9:13 AM Zhijie Hou <houzhijie22@gmail.com> wrote:
>
> Hi,
>
> On Sat, Nov14, 2026 at 07:35 AM Masahiko Sawada wrote:
> >
> > Agreed. It's better for callbacks to get both top-level and
> > subtransaction's ids.
> >
> > I've updated the patches (the 0002 patch unchanged).
>
>  > -    ctx->write_xid = txn != NULL ? txn->xid : InvalidTransactionId;
>  > +    ctx->write_xid = xid;
>
> I think it would be better to keep the original code for consistency, as
> all the other callbacks set ctx->write_xid to txn->xid (the top-level
> transaction's XID). IIUC, this was changed to make the test pass since
> test_decoding does not otherwise output the subxid, but we could instead
> use pgoutput to test that and keep passing the top-level XID in
> ctx->write_xid. Please see the attached top-up patch for this change.

Agreed. I've applied your patch.

> BTW, since we are changing the callback signature anyway, another wild
> idea would be to pass ReorderBufferChange directly instead so that it is
> fully consistent with the other change callbacks. That said, it would
> require more changes in third-party output plugins, so I don't insist on
> it if you disagree — just sharing for reference.

It seems overkill to me. For non-transactional messages, we don't
create a ReorderBufferChange for them, so we would have to allocate
one just for the callback and copy the prefix and the message into it.
We would still need a separate transactional argument, so I guess it
would not help us pass more information than the xid does.

I've attached the updated patches.

Regards,

-- 
Masahiko Sawada
Amazon Web Services: https://aws.amazon.com

Attachments: download all (2 files, .tar.gz)

v9-0001-Add-an-xid-argument-to-the-logical-decoding-messa.patch (text/x-patch, 20.3K) download From e2fafc8e94c73f0dba87d05ff3eebd4755399420 Mon Sep 17 00:00:00 2001 From: Zhijie Hou <houzhijie22@gmail.com> Date: Sat, 31 Oct 2026 19:55:25 +0800 Subject: [PATCH v9 1/2] Add an xid argument to the logical decoding message callbacks. Previously, when streaming an in-progress transaction, a transactional logical decoding message emitted in a subtransaction was passed to the stream_message callback with the top-level transaction only, and the callback had no way to know which subtransaction had emitted it. The change callbacks get that through ReorderBufferChange, but a message has none. pgoutput therefore sent the message with the top-level XID, and when the subtransaction was later rolled back and STREAM ABORT arrived for it, the subscriber could not discard the message with it. Oversight in commit 7259736a6e5. To fix this, we add the XID of the (sub)transaction that emitted the message as a new argument of both the message and stream_message callbacks, and have pgoutput send it. Output plugins can now see which subtransaction a message belongs to, as they already can for the other change types, while still getting the top-level transaction as before. Both callbacks get the argument so that a plugin can keep implementing them with one function, as pgoutput does. We considered fixing this in a backpatchable way, such as passing the subtransaction's ReorderBufferTXN to the callback instead of the top-level one, but that silently changes the meaning of an existing argument, and an output plugin that keeps its state on the top-level transaction would break. Adding an argument breaks the ABI, but it is the cleaner interface, so we chose it and don't backpatch. The bug exists in the back branches too, but there have been no reports, and the built-in logical replication doesn't use logical messages there. Author: Zhijie Hou <houzhijie22@gmail.com> Co-authored-by: Masahiko Sawada <sawada.mshk@gmail.com> Reviewed-by: Amit Kapila <amit.kapila16@gmail.com> Discussion: https://postgr.es/m/b9b5bd27-6416-48b4-8f73-c5a60e2c97e2@gmail.com --- contrib/test_decoding/expected/stream.out | 47 +++++++++++++++++++ contrib/test_decoding/sql/stream.sql | 29 ++++++++++++ contrib/test_decoding/test_decoding.c | 22 +++++---- doc/src/sgml/logicaldecoding.sgml | 14 +++++- src/backend/replication/logical/logical.c | 12 ++--- .../replication/logical/reorderbuffer.c | 11 +++-- src/backend/replication/pgoutput/pgoutput.c | 17 +++---- src/include/replication/output_plugin.h | 10 +++- src/include/replication/reorderbuffer.h | 2 + 9 files changed, 132 insertions(+), 32 deletions(-) diff --git a/contrib/test_decoding/expected/stream.out b/contrib/test_decoding/expected/stream.out index 0ec5c933610..d589c7a2b3f 100644 --- a/contrib/test_decoding/expected/stream.out +++ b/contrib/test_decoding/expected/stream.out @@ -157,6 +157,53 @@ SELECT data FROM pg_logical_slot_get_changes('regression_slot', NULL, NULL, 'inc COMMIT (3 rows) +-- Verify that a transactional message emitted in a streamed subtransaction +-- is associated with the subtransaction's XID. +CREATE PUBLICATION stream_pub; +SELECT 'init' FROM pg_create_logical_replication_slot('regression_pgoutput_slot', 'pgoutput'); + ?column? +---------- + init +(1 row) + +SET debug_logical_replication_streaming = immediate; +BEGIN; +SAVEPOINT sp; +SELECT 'msg_subxact' FROM pg_logical_emit_message(true, 'test', 'msg'); + ?column? +------------- + msg_subxact +(1 row) + +ROLLBACK TO SAVEPOINT sp; +COMMIT; +-- Check that the streamed MESSAGE ('M') carries the aborted subtransaction's +-- XID rather than the top-level XID: in pgoutput's wire format, 'M' stores +-- its 4-byte XID at bytes 2..5, while STREAM ABORT ('A') stores the top-level +-- XID at bytes 2..5 and the aborted subtransaction's XID at bytes 6..9. +WITH changes AS ( + SELECT data + FROM pg_logical_slot_get_binary_changes('regression_pgoutput_slot', NULL, NULL, + 'proto_version', '2', 'publication_names', 'stream_pub', + 'messages', 'true', 'streaming', 'on') +) +SELECT substr(m.data, 2, 4) = substr(a.data, 6, 4) AS subxid_matches +FROM changes m, changes a +WHERE get_byte(m.data, 0) = ascii('M') + AND get_byte(a.data, 0) = ascii('A'); + subxid_matches +---------------- + t +(1 row) + +RESET debug_logical_replication_streaming; +SELECT pg_drop_replication_slot('regression_pgoutput_slot'); + pg_drop_replication_slot +-------------------------- + +(1 row) + +DROP PUBLICATION stream_pub; DROP TABLE stream_test; SELECT pg_drop_replication_slot('regression_slot'); pg_drop_replication_slot diff --git a/contrib/test_decoding/sql/stream.sql b/contrib/test_decoding/sql/stream.sql index 5e45a8e9b64..6cef3c83b09 100644 --- a/contrib/test_decoding/sql/stream.sql +++ b/contrib/test_decoding/sql/stream.sql @@ -82,5 +82,34 @@ ROLLBACK; INSERT INTO stream_test VALUES ('after-abort'); SELECT data FROM pg_logical_slot_get_changes('regression_slot', NULL, NULL, 'include-xids', '0', 'skip-empty-xacts', '1'); +-- Verify that a transactional message emitted in a streamed subtransaction +-- is associated with the subtransaction's XID. +CREATE PUBLICATION stream_pub; +SELECT 'init' FROM pg_create_logical_replication_slot('regression_pgoutput_slot', 'pgoutput'); +SET debug_logical_replication_streaming = immediate; +BEGIN; +SAVEPOINT sp; +SELECT 'msg_subxact' FROM pg_logical_emit_message(true, 'test', 'msg'); +ROLLBACK TO SAVEPOINT sp; +COMMIT; + +-- Check that the streamed MESSAGE ('M') carries the aborted subtransaction's +-- XID rather than the top-level XID: in pgoutput's wire format, 'M' stores +-- its 4-byte XID at bytes 2..5, while STREAM ABORT ('A') stores the top-level +-- XID at bytes 2..5 and the aborted subtransaction's XID at bytes 6..9. +WITH changes AS ( + SELECT data + FROM pg_logical_slot_get_binary_changes('regression_pgoutput_slot', NULL, NULL, + 'proto_version', '2', 'publication_names', 'stream_pub', + 'messages', 'true', 'streaming', 'on') +) +SELECT substr(m.data, 2, 4) = substr(a.data, 6, 4) AS subxid_matches +FROM changes m, changes a +WHERE get_byte(m.data, 0) = ascii('M') + AND get_byte(a.data, 0) = ascii('A'); +RESET debug_logical_replication_streaming; +SELECT pg_drop_replication_slot('regression_pgoutput_slot'); +DROP PUBLICATION stream_pub; + DROP TABLE stream_test; SELECT pg_drop_replication_slot('regression_slot'); diff --git a/contrib/test_decoding/test_decoding.c b/contrib/test_decoding/test_decoding.c index 78185837aef..0f6e05b3bef 100644 --- a/contrib/test_decoding/test_decoding.c +++ b/contrib/test_decoding/test_decoding.c @@ -72,9 +72,9 @@ static void pg_decode_truncate(LogicalDecodingContext *ctx, static bool pg_decode_filter(LogicalDecodingContext *ctx, ReplOriginId origin_id); static void pg_decode_message(LogicalDecodingContext *ctx, - ReorderBufferTXN *txn, XLogRecPtr lsn, - bool transactional, const char *prefix, - Size sz, const char *message); + ReorderBufferTXN *txn, TransactionId xid, + XLogRecPtr lsn, bool transactional, + const char *prefix, Size sz, const char *message); static bool pg_decode_filter_prepare(LogicalDecodingContext *ctx, TransactionId xid, const char *gid); @@ -112,9 +112,10 @@ static void pg_decode_stream_change(LogicalDecodingContext *ctx, Relation relation, ReorderBufferChange *change); static void pg_decode_stream_message(LogicalDecodingContext *ctx, - ReorderBufferTXN *txn, XLogRecPtr lsn, - bool transactional, const char *prefix, - Size sz, const char *message); + ReorderBufferTXN *txn, TransactionId xid, + XLogRecPtr lsn, bool transactional, + const char *prefix, Size sz, + const char *message); static void pg_decode_stream_truncate(LogicalDecodingContext *ctx, ReorderBufferTXN *txn, int nrelations, Relation relations[], @@ -753,8 +754,9 @@ pg_decode_truncate(LogicalDecodingContext *ctx, ReorderBufferTXN *txn, static void pg_decode_message(LogicalDecodingContext *ctx, - ReorderBufferTXN *txn, XLogRecPtr lsn, bool transactional, - const char *prefix, Size sz, const char *message) + ReorderBufferTXN *txn, TransactionId xid, XLogRecPtr lsn, + bool transactional, const char *prefix, Size sz, + const char *message) { TestDecodingData *data = ctx->output_plugin_private; TestDecodingTxnData *txndata; @@ -954,8 +956,8 @@ pg_decode_stream_change(LogicalDecodingContext *ctx, */ static void pg_decode_stream_message(LogicalDecodingContext *ctx, - ReorderBufferTXN *txn, XLogRecPtr lsn, bool transactional, - const char *prefix, Size sz, const char *message) + ReorderBufferTXN *txn, TransactionId xid, XLogRecPtr lsn, + bool transactional, const char *prefix, Size sz, const char *message) { /* Output stream start if we haven't yet for transactional messages. */ if (transactional) diff --git a/doc/src/sgml/logicaldecoding.sgml b/doc/src/sgml/logicaldecoding.sgml index 6dc49108997..f21e1115a94 100644 --- a/doc/src/sgml/logicaldecoding.sgml +++ b/doc/src/sgml/logicaldecoding.sgml @@ -1128,6 +1128,7 @@ typedef bool (*LogicalDecodeFilterByOriginCB) (struct LogicalDecodingContext *ct <programlisting> typedef void (*LogicalDecodeMessageCB) (struct LogicalDecodingContext *ctx, ReorderBufferTXN *txn, + TransactionId xid, XLogRecPtr message_lsn, bool transactional, const char *prefix, @@ -1138,7 +1139,11 @@ typedef void (*LogicalDecodeMessageCB) (struct LogicalDecodingContext *ctx, the transaction, like the time stamp at which it has been committed and its XID. Note however that it can be NULL when the message is non-transactional and the XID was not assigned yet in the transaction - which logged the message. The <parameter>lsn</parameter> has WAL + which logged the message. The <parameter>xid</parameter> parameter is + the XID of the transaction that emitted the message; for a + transactional message emitted in a subtransaction, it is the + subtransaction's XID while <parameter>txn</parameter> is the top-level + transaction. The <parameter>lsn</parameter> has WAL location of the message. The <parameter>transactional</parameter> says if the message was sent as transactional or not. Similar to the change callback, in case of decoding a prepared (but yet uncommitted) @@ -1360,12 +1365,19 @@ typedef void (*LogicalDecodeStreamChangeCB) (struct LogicalDecodingContext *ctx, <programlisting> typedef void (*LogicalDecodeStreamMessageCB) (struct LogicalDecodingContext *ctx, ReorderBufferTXN *txn, + TransactionId xid, XLogRecPtr message_lsn, bool transactional, const char *prefix, Size message_size, const char *message); </programlisting> + The <parameter>txn</parameter> parameter is the top-level transaction + being streamed, and <parameter>xid</parameter> is the XID of the + transaction that emitted the message, which is a subtransaction's XID + if it was emitted in one. For the other stream callbacks, the XID of + the subtransaction is available as + <literal>change-&gt;txn-&gt;xid</literal>. </para> </sect3> diff --git a/src/backend/replication/logical/logical.c b/src/backend/replication/logical/logical.c index 8fe255fa960..95760981b26 100644 --- a/src/backend/replication/logical/logical.c +++ b/src/backend/replication/logical/logical.c @@ -81,7 +81,7 @@ static void change_cb_wrapper(ReorderBuffer *cache, ReorderBufferTXN *txn, static void truncate_cb_wrapper(ReorderBuffer *cache, ReorderBufferTXN *txn, int nrelations, Relation relations[], ReorderBufferChange *change); static void message_cb_wrapper(ReorderBuffer *cache, ReorderBufferTXN *txn, - XLogRecPtr message_lsn, bool transactional, + TransactionId xid, XLogRecPtr message_lsn, bool transactional, const char *prefix, Size message_size, const char *message); /* streaming callbacks */ @@ -98,7 +98,7 @@ static void stream_commit_cb_wrapper(ReorderBuffer *cache, ReorderBufferTXN *txn static void stream_change_cb_wrapper(ReorderBuffer *cache, ReorderBufferTXN *txn, Relation relation, ReorderBufferChange *change); static void stream_message_cb_wrapper(ReorderBuffer *cache, ReorderBufferTXN *txn, - XLogRecPtr message_lsn, bool transactional, + TransactionId xid, XLogRecPtr message_lsn, bool transactional, const char *prefix, Size message_size, const char *message); static void stream_truncate_cb_wrapper(ReorderBuffer *cache, ReorderBufferTXN *txn, int nrelations, Relation relations[], ReorderBufferChange *change); @@ -1336,7 +1336,7 @@ filter_by_origin_cb_wrapper(LogicalDecodingContext *ctx, ReplOriginId origin_id) static void message_cb_wrapper(ReorderBuffer *cache, ReorderBufferTXN *txn, - XLogRecPtr message_lsn, bool transactional, + TransactionId xid, XLogRecPtr message_lsn, bool transactional, const char *prefix, Size message_size, const char *message) { LogicalDecodingContext *ctx = cache->private_data; @@ -1364,7 +1364,7 @@ message_cb_wrapper(ReorderBuffer *cache, ReorderBufferTXN *txn, ctx->end_xact = false; /* do the actual work: call callback */ - ctx->callbacks.message_cb(ctx, txn, message_lsn, transactional, prefix, + ctx->callbacks.message_cb(ctx, txn, xid, message_lsn, transactional, prefix, message_size, message); /* Pop the error context stack */ @@ -1647,7 +1647,7 @@ stream_change_cb_wrapper(ReorderBuffer *cache, ReorderBufferTXN *txn, static void stream_message_cb_wrapper(ReorderBuffer *cache, ReorderBufferTXN *txn, - XLogRecPtr message_lsn, bool transactional, + TransactionId xid, XLogRecPtr message_lsn, bool transactional, const char *prefix, Size message_size, const char *message) { LogicalDecodingContext *ctx = cache->private_data; @@ -1679,7 +1679,7 @@ stream_message_cb_wrapper(ReorderBuffer *cache, ReorderBufferTXN *txn, ctx->end_xact = false; /* do the actual work: call callback */ - ctx->callbacks.stream_message_cb(ctx, txn, message_lsn, transactional, prefix, + ctx->callbacks.stream_message_cb(ctx, txn, xid, message_lsn, transactional, prefix, message_size, message); /* Pop the error context stack */ diff --git a/src/backend/replication/logical/reorderbuffer.c b/src/backend/replication/logical/reorderbuffer.c index 46843985317..2917d971f17 100644 --- a/src/backend/replication/logical/reorderbuffer.c +++ b/src/backend/replication/logical/reorderbuffer.c @@ -920,7 +920,8 @@ ReorderBufferQueueMessage(ReorderBuffer *rb, TransactionId xid, SetupHistoricSnapshot(snapshot_now, NULL); PG_TRY(); { - rb->message(rb, txn, lsn, false, prefix, message_size, message); + rb->message(rb, txn, xid, lsn, false, prefix, message_size, + message); TeardownHistoricSnapshot(false); } @@ -2104,13 +2105,13 @@ ReorderBufferApplyMessage(ReorderBuffer *rb, ReorderBufferTXN *txn, ReorderBufferChange *change, bool streaming) { if (streaming) - rb->stream_message(rb, txn, change->lsn, true, - change->data.msg.prefix, + rb->stream_message(rb, txn, change->txn->xid, change->lsn, + true, change->data.msg.prefix, change->data.msg.message_size, change->data.msg.message); else - rb->message(rb, txn, change->lsn, true, - change->data.msg.prefix, + rb->message(rb, txn, change->txn->xid, change->lsn, + true, change->data.msg.prefix, change->data.msg.message_size, change->data.msg.message); } diff --git a/src/backend/replication/pgoutput/pgoutput.c b/src/backend/replication/pgoutput/pgoutput.c index 57b3ccbed60..4cb36f6d075 100644 --- a/src/backend/replication/pgoutput/pgoutput.c +++ b/src/backend/replication/pgoutput/pgoutput.c @@ -55,9 +55,10 @@ static void pgoutput_truncate(LogicalDecodingContext *ctx, ReorderBufferTXN *txn, int nrelations, Relation relations[], ReorderBufferChange *change); static void pgoutput_message(LogicalDecodingContext *ctx, - ReorderBufferTXN *txn, XLogRecPtr message_lsn, - bool transactional, const char *prefix, - Size sz, const char *message); + ReorderBufferTXN *txn, TransactionId xid, + XLogRecPtr message_lsn, bool transactional, + const char *prefix, Size sz, + const char *message); static bool pgoutput_origin_filter(LogicalDecodingContext *ctx, ReplOriginId origin_id); static void pgoutput_begin_prepare_txn(LogicalDecodingContext *ctx, @@ -1722,11 +1723,11 @@ pgoutput_truncate(LogicalDecodingContext *ctx, ReorderBufferTXN *txn, static void pgoutput_message(LogicalDecodingContext *ctx, ReorderBufferTXN *txn, - XLogRecPtr message_lsn, bool transactional, const char *prefix, Size sz, - const char *message) + TransactionId xid, XLogRecPtr message_lsn, bool transactional, + const char *prefix, Size sz, const char *message) { PGOutputData *data = (PGOutputData *) ctx->output_plugin_private; - TransactionId xid = InvalidTransactionId; + TransactionId stream_xid = InvalidTransactionId; if (!data->messages) return; @@ -1736,7 +1737,7 @@ pgoutput_message(LogicalDecodingContext *ctx, ReorderBufferTXN *txn, * pgoutput_change. */ if (data->in_streaming) - xid = txn->xid; + stream_xid = xid; /* * Output BEGIN if we haven't yet. Avoid for non-transactional messages. @@ -1752,7 +1753,7 @@ pgoutput_message(LogicalDecodingContext *ctx, ReorderBufferTXN *txn, OutputPluginPrepareWrite(ctx, true); logicalrep_write_message(ctx->out, - xid, + stream_xid, message_lsn, transactional, prefix, diff --git a/src/include/replication/output_plugin.h b/src/include/replication/output_plugin.h index 842fcde67f9..2f12fe09ea2 100644 --- a/src/include/replication/output_plugin.h +++ b/src/include/replication/output_plugin.h @@ -80,10 +80,14 @@ typedef void (*LogicalDecodeCommitCB) (struct LogicalDecodingContext *ctx, XLogRecPtr commit_lsn); /* - * Called for the generic logical decoding messages. + * Called for the generic logical decoding messages. "xid" is the XID of the + * (sub)transaction that emitted the message; for a transactional message + * emitted in a subtransaction it differs from txn->xid, as txn is the + * top-level transaction. */ typedef void (*LogicalDecodeMessageCB) (struct LogicalDecodingContext *ctx, ReorderBufferTXN *txn, + TransactionId xid, XLogRecPtr message_lsn, bool transactional, const char *prefix, @@ -191,10 +195,12 @@ typedef void (*LogicalDecodeStreamChangeCB) (struct LogicalDecodingContext *ctx, /* * Callback for streaming generic logical decoding messages from in-progress - * transactions. + * transactions. "txn" is the top-level transaction and "xid" the XID of the + * (sub)transaction that emitted the message. */ typedef void (*LogicalDecodeStreamMessageCB) (struct LogicalDecodingContext *ctx, ReorderBufferTXN *txn, + TransactionId xid, XLogRecPtr message_lsn, bool transactional, const char *prefix, diff --git a/src/include/replication/reorderbuffer.h b/src/include/replication/reorderbuffer.h index cc47b472db3..612a7a81f40 100644 --- a/src/include/replication/reorderbuffer.h +++ b/src/include/replication/reorderbuffer.h @@ -500,6 +500,7 @@ typedef void (*ReorderBufferCommitCB) (ReorderBuffer *rb, /* message callback signature */ typedef void (*ReorderBufferMessageCB) (ReorderBuffer *rb, ReorderBufferTXN *txn, + TransactionId xid, XLogRecPtr message_lsn, bool transactional, const char *prefix, Size sz, @@ -559,6 +560,7 @@ typedef void (*ReorderBufferStreamChangeCB) (ReorderBuffer *rb, /* stream message callback signature */ typedef void (*ReorderBufferStreamMessageCB) (ReorderBuffer *rb, ReorderBufferTXN *txn, + TransactionId xid, XLogRecPtr message_lsn, bool transactional, const char *prefix, Size sz, -- 2.56.0
v9-0002-Add-a-hook-for-handling-logical-messages-on-subsc.patch (text/x-patch, 121.3K) download From caf37f6e50745422f0cd433979f393ec75aa4a56 Mon Sep 17 00:00:00 2001 From: Masahiko Sawada <sawada.mshk@gmail.com> Date: Mon, 8 Jun 2026 11:25:47 -0700 Subject: [PATCH v9 2/2] Add a hook for handling logical messages on subscribers. Previously, pgoutput could emit logical decoding messages, those written with pg_logical_emit_message(), but the built-in subscriber never requested them from the publisher, and apply_handle_message() was only a placeholder that discarded the MESSAGE message type. Acting on such messages therefore meant writing a separate logical decoding client, and giving up the apply worker's streaming, conflict handling and progress tracking. Logical messages are a general-purpose side channel carried in band with the data, useful for replicating deparsed DDL, for carrying change-data-capture context that the row data does not have, and for cache invalidation delivered in commit order relative to the rows it refers to. This commit adds a new "messages" subscription option, which asks the publisher for the pgoutput "messages" option, and dispatches each received message to a new LogicalRepMessageHandle_hook. Behavior is unchanged when the option is off or when no handler is installed. A transactional message is applied as a step of the remote transaction it belongs to, so the handler's work commits atomically with that transaction, and transaction streaming and ALTER SUBSCRIPTION ... SKIP apply to it as they do to any other change. A non-transactional message belongs to no remote transaction and is committed on its own. A message may still reach the handler more than once, for example after an apply worker restart, so handlers must be idempotent. Note also that messages are not scoped by publication: every message emitted in the publisher database is sent, subject only to origin filtering, and pg_logical_emit_message() requires no special privileges. A handler must therefore treat the payload as untrusted input, and as an arbitrary string of bytes rather than as text in the subscriber's encoding. The publisher must be running PostgreSQL 14 or later, where pgoutput gained the "messages" option; against an older publisher the option is accepted but no message is ever sent. Bump catalog version. Reviewed-by: Bharath Rupireddy <bharath.rupireddyforpostgres@gmail.com> Reviewed-by: Fujii Masao <masao.fujii@gmail.com> Reviewed-by: Amit Kapila <amit.kapila16@gmail.com> Reviewed-by: Chao Li <li.evan.chao@gmail.com> Discussion: https://postgr.es/m/CAD21AoCTNGiddikkUcDKj5QLnsg-51bpr-o6L-GTHWZL4ZFYtQ@mail.gmail.com --- doc/src/sgml/catalogs.sgml | 11 + doc/src/sgml/func/func-admin.sgml | 7 +- doc/src/sgml/logical-replication.sgml | 35 ++ doc/src/sgml/ref/alter_subscription.sgml | 5 +- doc/src/sgml/ref/create_subscription.sgml | 43 +++ src/backend/catalog/pg_subscription.c | 1 + src/backend/catalog/system_views.sql | 5 +- src/backend/commands/subscriptioncmds.c | 27 +- .../libpqwalreceiver/libpqwalreceiver.c | 4 + src/backend/replication/logical/proto.c | 37 ++ src/backend/replication/logical/worker.c | 100 +++++- src/bin/pg_dump/pg_dump.c | 16 +- src/bin/pg_dump/pg_dump.h | 1 + src/bin/pg_dump/t/002_pg_dump.pl | 4 +- src/bin/psql/describe.c | 7 +- src/bin/psql/tab-complete.in.c | 6 +- src/include/catalog/pg_subscription.h | 5 + src/include/replication/logicalproto.h | 13 + src/include/replication/logicalworker.h | 49 +++ src/include/replication/walreceiver.h | 1 + src/test/modules/Makefile | 1 + src/test/modules/meson.build | 1 + .../modules/test_logicalmsg_hooks/.gitignore | 4 + .../modules/test_logicalmsg_hooks/Makefile | 20 ++ .../modules/test_logicalmsg_hooks/meson.build | 28 ++ .../test_logicalmsg_hooks/t/001_basic.pl | 318 ++++++++++++++++++ .../test_logicalmsg_hooks.c | 140 ++++++++ src/test/regress/expected/subscription.out | 202 ++++++----- src/test/regress/sql/subscription.sql | 17 + src/tools/pgindent/typedefs.list | 2 + 30 files changed, 1000 insertions(+), 110 deletions(-) create mode 100644 src/test/modules/test_logicalmsg_hooks/.gitignore create mode 100644 src/test/modules/test_logicalmsg_hooks/Makefile create mode 100644 src/test/modules/test_logicalmsg_hooks/meson.build create mode 100644 src/test/modules/test_logicalmsg_hooks/t/001_basic.pl create mode 100644 src/test/modules/test_logicalmsg_hooks/test_logicalmsg_hooks.c diff --git a/doc/src/sgml/catalogs.sgml b/doc/src/sgml/catalogs.sgml index 07ba05dccd9..5e7f3032baa 100644 --- a/doc/src/sgml/catalogs.sgml +++ b/doc/src/sgml/catalogs.sgml @@ -8250,6 +8250,17 @@ SCRAM-SHA-256$<replaceable>&lt;iteration count&gt;</replaceable>:<replaceable>&l </para></entry> </row> + <row> + <entry role="catalog_table_entry"><para role="column_definition"> + <structfield>submessages</structfield> <type>bool</type> + </para> + <para> + If true, the subscription requests the publisher to send logical + decoding messages, which are passed to a handler registered by an + extension on the subscriber. + </para></entry> + </row> + <row> <entry role="catalog_table_entry"><para role="column_definition"> <structfield>subconninfo</structfield> <type>text</type> diff --git a/doc/src/sgml/func/func-admin.sgml b/doc/src/sgml/func/func-admin.sgml index ed872264f02..e63d72f20ff 100644 --- a/doc/src/sgml/func/func-admin.sgml +++ b/doc/src/sgml/func/func-admin.sgml @@ -1479,7 +1479,12 @@ postgres=# SELECT '0/0'::pg_lsn + pd.segment_number * ps.setting::int + :offset <literal>false</literal>) controls if the message is immediately flushed to WAL or not. <parameter>flush</parameter> has no effect with <parameter>transactional</parameter>, as the message's WAL - record is flushed along with its transaction. + record is flushed along with its transaction. In addition to logical + decoding clients, messages emitted by this function are delivered to + subscribers whose subscriptions have + <link linkend="sql-createsubscription-params-with-messages"><literal>messages</literal></link> + enabled, where they are passed to a handler registered by an extension + on the subscriber. </para></entry> </row> diff --git a/doc/src/sgml/logical-replication.sgml b/doc/src/sgml/logical-replication.sgml index 4701a3d9d18..849724efc43 100644 --- a/doc/src/sgml/logical-replication.sgml +++ b/doc/src/sgml/logical-replication.sgml @@ -2661,6 +2661,41 @@ CONTEXT: processing remote data for replication origin "pg_16395" during "INSER not need privileges to <literal>SET ROLE</literal> to the sequence owner. </para> + <para> + Logical decoding messages are an exception to the user switching described + above. If the subscription has <link linkend="sql-createsubscription-params-with-messages"><literal>messages</literal></link> + enabled, the handler registered for them runs with the privileges of the + subscription owner. Messages are not restricted to the tables or schemas + of the subscribed publications either, and by default any role may call + <link linkend="pg-logical-emit-message"><function>pg_logical_emit_message</function></link>. + Any role that can connect to the publisher database can therefore choose + what the handler is given, and a handler that acts on what it receives does + so with the subscription owner's privileges. Restrict who can emit messages + on the publisher. Revoke the default privilege and grant it to a role created + for this purpose: +<programlisting> +REVOKE EXECUTE ON FUNCTION pg_logical_emit_message(boolean, text, text, boolean) FROM PUBLIC; +REVOKE EXECUTE ON FUNCTION pg_logical_emit_message(boolean, text, bytea, boolean) FROM PUBLIC; +GRANT EXECUTE ON FUNCTION pg_logical_emit_message(boolean, text, text, boolean) TO message_emitter; +GRANT EXECUTE ON FUNCTION pg_logical_emit_message(boolean, text, bytea, boolean) TO message_emitter; +</programlisting> + Function privileges are per database, so this has to be done in each database + that such a subscription replicates from, and in <literal>template1</literal> + if databases created later are to inherit it. No role that is not trusted to + drive the handler should be a member of the new role, since a role that can + <literal>SET ROLE</literal> to it can emit whatever it likes. Where a message + has to be emitted on behalf of an ordinary user, for example from an event + trigger that captures that user's DDL, use a <literal>SECURITY DEFINER</literal> + function owned by that role rather than granting the privilege more widely. + </para> + + <para> + This is a control on the publisher, and the subscriber has no way to verify + that it is in place. If you do not administer both the publisher and the + subscriber, do not install a message handler that does anything you would + not let every role on the publisher do. + </para> + <para> On the publisher, privileges are only checked once at the start of a replication connection and are not re-checked as each change record is read. diff --git a/doc/src/sgml/ref/alter_subscription.sgml b/doc/src/sgml/ref/alter_subscription.sgml index 545264e8a0a..72b1b0fcd83 100644 --- a/doc/src/sgml/ref/alter_subscription.sgml +++ b/doc/src/sgml/ref/alter_subscription.sgml @@ -302,8 +302,9 @@ ALTER SUBSCRIPTION <replaceable class="parameter">name</replaceable> RENAME TO < <link linkend="sql-createsubscription-params-with-two-phase"><literal>two_phase</literal></link>, <link linkend="sql-createsubscription-params-with-retain-dead-tuples"><literal>retain_dead_tuples</literal></link>, <link linkend="sql-createsubscription-params-with-max-retention-duration"><literal>max_retention_duration</literal></link>, - <link linkend="sql-createsubscription-params-with-wal-receiver-timeout"><literal>wal_receiver_timeout</literal></link>, and - <link linkend="sql-createsubscription-params-with-conflict-log-destination"><literal>conflict_log_destination</literal></link>. + <link linkend="sql-createsubscription-params-with-wal-receiver-timeout"><literal>wal_receiver_timeout</literal></link>, + <link linkend="sql-createsubscription-params-with-conflict-log-destination"><literal>conflict_log_destination</literal></link>, and + <link linkend="sql-createsubscription-params-with-messages"><literal>messages</literal></link>. Only a superuser can set <literal>password_required = false</literal>. </para> diff --git a/doc/src/sgml/ref/create_subscription.sgml b/doc/src/sgml/ref/create_subscription.sgml index 6248c228a64..1c38daaf434 100644 --- a/doc/src/sgml/ref/create_subscription.sgml +++ b/doc/src/sgml/ref/create_subscription.sgml @@ -478,6 +478,49 @@ CREATE SUBSCRIPTION <replaceable class="parameter">subscription_name</replaceabl </listitem> </varlistentry> + <varlistentry id="sql-createsubscription-params-with-messages"> + <term><literal>messages</literal> (<type>boolean</type>)</term> + <listitem> + <para> + Specifies whether the subscription requests the publisher to send + logical decoding messages, that is, those emitted with + <link linkend="pg-logical-emit-message"><function>pg_logical_emit_message</function></link>. + The default is <literal>false</literal>. + </para> + <para> + Received messages are only acted upon if an extension installed on + the subscriber has registered a handler for them; otherwise they are + received and discarded. A message may be passed to the handler more + than once, for example after an apply worker restart, so a handler + must be prepared to process the same message repeatedly. + </para> + <para> + A transactional message is applied as part of the remote transaction + that emitted it, and is therefore delivered in commit order with + respect to other transactions, not in the order in which it was written. + Note that + <link linkend="sql-createsubscription-params-with-streaming"><literal>streaming</literal></link> + and <link linkend="sql-altersubscription-params-skip"><literal>SKIP</literal></link> + apply to transactional messages just as they do to other changes. A + non-transactional message is delivered as soon as it is decoded, even + if the transaction that emitted it later aborted. + </para> + <para> + Messages are not restricted to the tables or schemas of the + subscribed publications: every message emitted in the publisher + database is sent, subject only to + <link linkend="sql-createsubscription-params-with-origin"><literal>origin</literal></link> + filtering. Message contents are supplied by whoever calls + <function>pg_logical_emit_message</function> on the publisher, which + by default any role may do, so a handler should treat them as + untrusted input. A message is an arbitrary string of bytes, and is + transmitted without character set conversion; it may therefore + contain embedded nulls, and need not be valid text in the encoding + of either database. + </para> + </listitem> + </varlistentry> + <varlistentry id="sql-createsubscription-params-with-origin"> <term><literal>origin</literal> (<type>string</type>)</term> <listitem> diff --git a/src/backend/catalog/pg_subscription.c b/src/backend/catalog/pg_subscription.c index 60c15b22194..138cdecd436 100644 --- a/src/backend/catalog/pg_subscription.c +++ b/src/backend/catalog/pg_subscription.c @@ -133,6 +133,7 @@ GetSubscription(Oid subid, bool missing_ok) sub->maxretention = subform->submaxretention; sub->retentionactive = subform->subretentionactive; sub->conflictlogrelid = subform->subconflictlogrelid; + sub->messages = subform->submessages; /* Get slotname */ datum = SysCacheGetAttr(SUBSCRIPTIONOID, diff --git a/src/backend/catalog/system_views.sql b/src/backend/catalog/system_views.sql index 6dce9192ba0..19b96bba1fd 100644 --- a/src/backend/catalog/system_views.sql +++ b/src/backend/catalog/system_views.sql @@ -1545,8 +1545,9 @@ GRANT SELECT (oid, subdbid, subskiplsn, subname, subowner, subenabled, subbinary, substream, subtwophasestate, subdisableonerr, subpasswordrequired, subrunasowner, subfailover, subretaindeadtuples, submaxretention, subretentionactive, - subserver, subconflictlogrelid, subconflictlogdest, subslotname, - subsynccommit, subwalrcvtimeout, subpublications, suborigin) + subserver, subconflictlogrelid, subconflictlogdest, submessages, + subslotname, subsynccommit, subwalrcvtimeout, subpublications, + suborigin) ON pg_subscription TO public; CREATE VIEW pg_stat_subscription_stats AS diff --git a/src/backend/commands/subscriptioncmds.c b/src/backend/commands/subscriptioncmds.c index ce45f7e16d4..9dc8d608b8d 100644 --- a/src/backend/commands/subscriptioncmds.c +++ b/src/backend/commands/subscriptioncmds.c @@ -84,6 +84,7 @@ #define SUBOPT_LSN 0x00020000 #define SUBOPT_ORIGIN 0x00040000 #define SUBOPT_CONFLICT_LOG_DEST 0x00080000 +#define SUBOPT_MESSAGES 0x00100000 /* check if the 'val' has 'bits' set */ #define IsSet(val, bits) (((val) & (bits)) == (bits)) @@ -110,6 +111,7 @@ typedef struct SubOpts bool runasowner; bool failover; bool retaindeadtuples; + bool messages; int32 maxretention; char *origin; ConflictLogDest conflictlogdest; @@ -208,6 +210,8 @@ parse_subscription_options(ParseState *pstate, List *stmt_options, opts->origin = pstrdup(LOGICALREP_ORIGIN_ANY); if (IsSet(supported_opts, SUBOPT_CONFLICT_LOG_DEST)) opts->conflictlogdest = CONFLICT_LOG_DEST_LOG; + if (IsSet(supported_opts, SUBOPT_MESSAGES)) + opts->messages = false; /* Parse options */ foreach(lc, stmt_options) @@ -460,6 +464,15 @@ parse_subscription_options(ParseState *pstate, List *stmt_options, opts->conflictlogdest = GetConflictLogDest(val); opts->specified_opts |= SUBOPT_CONFLICT_LOG_DEST; } + else if (IsSet(supported_opts, SUBOPT_MESSAGES) && + strcmp(defel->defname, "messages") == 0) + { + if (IsSet(opts->specified_opts, SUBOPT_MESSAGES)) + errorConflictingDefElem(defel, pstate); + + opts->specified_opts |= SUBOPT_MESSAGES; + opts->messages = defGetBoolean(defel); + } else ereport(ERROR, (errcode(ERRCODE_SYNTAX_ERROR), @@ -700,7 +713,7 @@ CreateSubscription(ParseState *pstate, CreateSubscriptionStmt *stmt, SUBOPT_RETAIN_DEAD_TUPLES | SUBOPT_MAX_RETENTION_DURATION | SUBOPT_WAL_RECEIVER_TIMEOUT | SUBOPT_ORIGIN | - SUBOPT_CONFLICT_LOG_DEST); + SUBOPT_CONFLICT_LOG_DEST | SUBOPT_MESSAGES); parse_subscription_options(pstate, stmt->options, supported_opts, &opts); /* @@ -872,6 +885,8 @@ CreateSubscription(ParseState *pstate, CreateSubscriptionStmt *stmt, values[Anum_pg_subscription_subretentionactive - 1] = BoolGetDatum(opts.retaindeadtuples); values[Anum_pg_subscription_subserver - 1] = ObjectIdGetDatum(serverid); + values[Anum_pg_subscription_submessages - 1] = + BoolGetDatum(opts.messages); if (stmt->conninfo) { Assert(stmt->conninfo == conninfo && !OidIsValid(serverid)); @@ -1714,7 +1729,8 @@ AlterSubscription(ParseState *pstate, AlterSubscriptionStmt *stmt, SUBOPT_MAX_RETENTION_DURATION | SUBOPT_WAL_RECEIVER_TIMEOUT | SUBOPT_ORIGIN | - SUBOPT_CONFLICT_LOG_DEST); + SUBOPT_CONFLICT_LOG_DEST | + SUBOPT_MESSAGES); break; case ALTER_SUBSCRIPTION_ENABLED: @@ -2128,6 +2144,13 @@ AlterSubscription(ParseState *pstate, AlterSubscriptionStmt *stmt, } } + if (IsSet(opts.specified_opts, SUBOPT_MESSAGES)) + { + values[Anum_pg_subscription_submessages - 1] = + BoolGetDatum(opts.messages); + replaces[Anum_pg_subscription_submessages - 1] = true; + } + update_tuple = true; break; } diff --git a/src/backend/replication/libpqwalreceiver/libpqwalreceiver.c b/src/backend/replication/libpqwalreceiver/libpqwalreceiver.c index dcc738f4e38..ffd03909ab4 100644 --- a/src/backend/replication/libpqwalreceiver/libpqwalreceiver.c +++ b/src/backend/replication/libpqwalreceiver/libpqwalreceiver.c @@ -664,6 +664,10 @@ libpqrcv_startstreaming(WalReceiverConn *conn, appendQuotedLiteral(&cmd, options->proto.logical.origin); } + if (options->proto.logical.messages && + PQserverVersion(conn->streamConn) >= 140000) + appendStringInfo(&cmd, ", messages 'on'"); + pubnames = options->proto.logical.publication_names; pubnames_str = stringlist_to_identifierstr(pubnames); appendStringInfoString(&cmd, ", publication_names "); diff --git a/src/backend/replication/logical/proto.c b/src/backend/replication/logical/proto.c index 86ad97cd937..6461361d68d 100644 --- a/src/backend/replication/logical/proto.c +++ b/src/backend/replication/logical/proto.c @@ -660,6 +660,43 @@ logicalrep_write_message(StringInfo out, TransactionId xid, XLogRecPtr lsn, pq_sendbytes(out, message, sz); } +/* + * Read MESSAGE from stream. + */ +void +logicalrep_read_message(StringInfo in, LogicalRepMessageData *msg_data) +{ + Size len; + char *msg; + uint8 flags; + + /* read and decode flags */ + flags = pq_getmsgint(in, 1); + msg_data->transactional = (flags & MESSAGE_TRANSACTIONAL) != 0; + + msg_data->lsn = pq_getmsgint64(in); + msg_data->prefix = pstrdup(pq_getmsgstring(in)); + + /* read message length */ + len = pq_getmsgint(in, 4); + msg_data->message_size = len; + + /* and data */ + msg = palloc(len + 1); + pq_copymsgbytes(in, msg, len); + + /* + * The payload is an arbitrary string of bytes, so message_size, not + * strlen(), is what gives its length. NULL-terminate it anyway, as + * logicalrep_read_tuple() does for column values: that way it can be + * handed to code expecting a C string without reading past the end, which + * is convenient for logging and debugging. Such code will of course stop + * at the first embedded null, if any. + */ + msg[len] = '\0'; + msg_data->message = msg; +} + /* * Write relation description to the output stream. */ diff --git a/src/backend/replication/logical/worker.c b/src/backend/replication/logical/worker.c index 95044620e68..166fdc5b2fb 100644 --- a/src/backend/replication/logical/worker.c +++ b/src/backend/replication/logical/worker.c @@ -561,6 +561,9 @@ typedef struct ApplySubXactData static ApplySubXactData subxact_data = {0, 0, InvalidTransactionId, NULL}; +/* Hook for plugins to get control in handling logical decoding messages */ +LogicalRepMessageHandle_hook_type LogicalRepMessageHandle_hook = NULL; + static inline void subxact_filename(char *path, Oid subid, TransactionId xid); static inline void changes_filename(char *path, Oid subid, TransactionId xid); @@ -1709,6 +1712,84 @@ apply_handle_origin(StringInfo s) errmsg_internal("ORIGIN message sent out of order"))); } +/* + * Handle MESSAGE message. + * + * Invoke a hook function if set. + */ +static void +apply_handle_message(StringInfo s) +{ + LogicalRepMessageData msg; + + /* + * The publisher must not send a message that this worker did not ask for; + * see set_stream_options(). + */ + if (!MySubscription->messages || am_tablesync_worker()) + ereport(ERROR, + (errcode(ERRCODE_PROTOCOL_VIOLATION), + errmsg_internal("MESSAGE message received but messages are not enabled for this worker"))); + + if (!LogicalRepMessageHandle_hook) + return; + + if (is_skipping_changes() || + handle_streamed_transaction(LOGICAL_REP_MSG_MESSAGE, s)) + return; + + begin_replication_step(); + + logicalrep_read_message(s, &msg); + + /* + * The publisher sends a non-transactional message on its own, never + * inside a transaction or a streamed block; see + * ReorderBufferQueueMessage(). Reject one that arrives inside a + * transaction before calling the handler, as the commit below would + * otherwise split the remote transaction in two. + */ + if (!msg.transactional && + (in_remote_transaction || in_streamed_transaction)) + ereport(ERROR, + (errcode(ERRCODE_PROTOCOL_VIOLATION), + errmsg_internal("non-transactional MESSAGE message received inside a transaction"))); + + (*LogicalRepMessageHandle_hook) (&msg); + + end_replication_step(); + + /* + * A transactional message is applied as a step of the remote transaction + * that emitted it, and is committed together with it when applying the + * commit message. A non-transactional message belongs to no remote + * transaction, so commit it here. + */ + if (!msg.transactional) + { + /* + * The message doesn't belong to any remote transaction, so there is + * no remote commit LSN nor timestamp to record. Clear the state left + * over by the previously applied transaction so that this commit + * doesn't inherit it. + */ + replorigin_xact_clear(false); + + CommitTransactionCommand(); + + pgstat_report_stat(false); + + /* + * Report the flush position only after the local commit is durable. + * Otherwise send_feedback() would report everything received so far + * as flushed, the publisher would advance confirmed_flush_lsn past + * this message, and a crash before the commit reaches disk would lose + * the handler's work without the message ever being sent again. + */ + store_flush_position(msg.lsn, XactLastCommitEnd); + } +} + /* * Initialize fileset (if not already done). * @@ -3894,12 +3975,7 @@ apply_dispatch(StringInfo s) break; case LOGICAL_REP_MSG_MESSAGE: - - /* - * Logical replication does not use generic logical messages yet. - * Although, it could be used by other applications that use this - * output plugin. - */ + apply_handle_message(s); break; case LOGICAL_REP_MSG_STREAM_START: @@ -5185,7 +5261,8 @@ maybe_reread_subscription(void) newsub->passwordrequired != MySubscription->passwordrequired || strcmp(newsub->origin, MySubscription->origin) != 0 || newsub->owner != MySubscription->owner || - !equal(newsub->publications, MySubscription->publications)) + !equal(newsub->publications, MySubscription->publications) || + newsub->messages != MySubscription->messages) { if (am_parallel_apply_worker()) ereport(LOG, @@ -5676,6 +5753,15 @@ set_stream_options(WalRcvStreamOptions *options, options->proto.logical.twophase = false; options->proto.logical.origin = pstrdup(MySubscription->origin); + + /* + * Logical messages are relation-agnostic, so they don't map cleanly onto + * the tablesync worker of a particular relation, and there would be no + * well-defined ordering between a message and the initial copy. Leave + * them to the (parallel) apply workers. + */ + options->proto.logical.messages = (MySubscription->messages && + !am_tablesync_worker()); } /* diff --git a/src/bin/pg_dump/pg_dump.c b/src/bin/pg_dump/pg_dump.c index a3f38f3b886..fe1bf38a748 100644 --- a/src/bin/pg_dump/pg_dump.c +++ b/src/bin/pg_dump/pg_dump.c @@ -5142,6 +5142,7 @@ getSubscriptions(Archive *fout) int i_subfailover; int i_subretaindeadtuples; int i_submaxretention; + int i_submessages; int i, ntups; @@ -5240,9 +5241,14 @@ getSubscriptions(Archive *fout) " '-1' AS subwalrcvtimeout,\n"); if (fout->remoteVersion >= 190000) - appendPQExpBufferStr(query, " fs.srvname AS subservername\n"); + appendPQExpBufferStr(query, " fs.srvname AS subservername,\n"); else - appendPQExpBufferStr(query, " NULL AS subservername\n"); + appendPQExpBufferStr(query, " NULL AS subservername,\n"); + + if (fout->remoteVersion >= 200000) + appendPQExpBufferStr(query, " s.submessages\n"); + else + appendPQExpBufferStr(query, " false AS submessages\n"); appendPQExpBufferStr(query, "FROM pg_subscription s\n"); @@ -5291,6 +5297,7 @@ getSubscriptions(Archive *fout) i_subpublications = PQfnumber(res, "subpublications"); i_suborigin = PQfnumber(res, "suborigin"); i_suboriginremotelsn = PQfnumber(res, "suboriginremotelsn"); + i_submessages = PQfnumber(res, "submessages"); subinfo = pg_malloc_array(SubscriptionInfo, ntups); @@ -5348,6 +5355,8 @@ getSubscriptions(Archive *fout) else subinfo[i].suboriginremotelsn = pg_strdup(PQgetvalue(res, i, i_suboriginremotelsn)); + subinfo[i].submessages = + (strcmp(PQgetvalue(res, i, i_submessages), "t") == 0); /* Decide whether we want to dump it */ selectDumpableObject(&(subinfo[i].dobj), fout); @@ -5600,6 +5609,9 @@ dumpSubscription(Archive *fout, const SubscriptionInfo *subinfo) if (subinfo->subretaindeadtuples) appendPQExpBufferStr(query, ", retain_dead_tuples = true"); + if (subinfo->submessages) + appendPQExpBufferStr(query, ", messages = true"); + if (subinfo->submaxretention) appendPQExpBuffer(query, ", max_retention_duration = %d", subinfo->submaxretention); diff --git a/src/bin/pg_dump/pg_dump.h b/src/bin/pg_dump/pg_dump.h index 41d932f003b..1df8a5252a4 100644 --- a/src/bin/pg_dump/pg_dump.h +++ b/src/bin/pg_dump/pg_dump.h @@ -721,6 +721,7 @@ typedef struct _SubscriptionInfo bool subrunasowner; bool subfailover; bool subretaindeadtuples; + bool submessages; int submaxretention; char *subservername; char *subconninfo; diff --git a/src/bin/pg_dump/t/002_pg_dump.pl b/src/bin/pg_dump/t/002_pg_dump.pl index ec73c4d5be9..a32710d085a 100644 --- a/src/bin/pg_dump/t/002_pg_dump.pl +++ b/src/bin/pg_dump/t/002_pg_dump.pl @@ -3424,9 +3424,9 @@ my %tests = ( create_order => 50, create_sql => 'CREATE SUBSCRIPTION sub3 CONNECTION \'dbname=doesnotexist\' PUBLICATION pub1 - WITH (connect = false, origin = any, streaming = on);', + WITH (connect = false, origin = any, streaming = on, messages = on);', regexp => qr/^ - \QCREATE SUBSCRIPTION sub3 CONNECTION 'dbname=doesnotexist' PUBLICATION pub1 WITH (connect = false, slot_name = 'sub3', streaming = on);\E + \QCREATE SUBSCRIPTION sub3 CONNECTION 'dbname=doesnotexist' PUBLICATION pub1 WITH (connect = false, slot_name = 'sub3', streaming = on, messages = true);\E /xm, like => { %full_runs, section_post_data => 1, }, unlike => { diff --git a/src/bin/psql/describe.c b/src/bin/psql/describe.c index e4a4ba47280..e6e48a9c66c 100644 --- a/src/bin/psql/describe.c +++ b/src/bin/psql/describe.c @@ -6774,7 +6774,7 @@ describeSubscriptions(const char *pattern, bool verbose) printQueryOpt myopt = pset.popt; static const bool translate_columns[] = {false, false, false, false, false, false, false, false, false, false, false, false, false, false, - false, false, false, false, false, false, false}; + false, false, false, false, false, false, false, false}; initPQExpBuffer(&buf); @@ -6852,6 +6852,11 @@ describeSubscriptions(const char *pattern, bool verbose) gettext_noop("Retention active")); } + if (pset.sversion >= 200000) + appendPQExpBuffer(&buf, + ", submessages AS \"%s\"\n", + gettext_noop("Messages")); + appendPQExpBuffer(&buf, ", subsynccommit AS \"%s\"\n" ", subconninfo AS \"%s\"\n", diff --git a/src/bin/psql/tab-complete.in.c b/src/bin/psql/tab-complete.in.c index 96de78cd087..2346c8b969f 100644 --- a/src/bin/psql/tab-complete.in.c +++ b/src/bin/psql/tab-complete.in.c @@ -2360,8 +2360,8 @@ match_previous_words(int pattern_id, /* ALTER SUBSCRIPTION <name> SET ( */ else if (Matches("ALTER", "SUBSCRIPTION", MatchAny, MatchAnyN, "SET", "(")) COMPLETE_WITH("binary", "conflict_log_destination", "disable_on_error", - "failover", "max_retention_duration", "origin", - "password_required", "retain_dead_tuples", + "failover", "max_retention_duration", "messages", + "origin", "password_required", "retain_dead_tuples", "run_as_owner", "slot_name", "streaming", "synchronous_commit", "two_phase", "wal_receiver_timeout"); @@ -3897,7 +3897,7 @@ match_previous_words(int pattern_id, else if (Matches("CREATE", "SUBSCRIPTION", MatchAnyN, "WITH", "(")) COMPLETE_WITH("binary", "conflict_log_destination", "connect", "copy_data", "create_slot", "disable_on_error", "enabled", "failover", - "max_retention_duration", "origin", + "max_retention_duration", "messages", "origin", "password_required", "retain_dead_tuples", "run_as_owner", "slot_name", "streaming", "synchronous_commit", "two_phase", diff --git a/src/include/catalog/pg_subscription.h b/src/include/catalog/pg_subscription.h index b860f02d166..07b28f86e0f 100644 --- a/src/include/catalog/pg_subscription.h +++ b/src/include/catalog/pg_subscription.h @@ -96,6 +96,9 @@ CATALOG(pg_subscription,6100,SubscriptionRelationId) BKI_SHARED_RELATION BKI_ROW * server */ Oid subconflictlogrelid; /* Relid of the conflict log table. */ + + bool submessages; /* True if the subscription wants to receive + * logical messages. */ #ifdef CATALOG_VARLEN /* variable-length fields start here */ /* @@ -173,6 +176,8 @@ typedef struct Subscription * exceeded max_retention_duration, when * defined */ Oid conflictlogrelid; /* conflict log table Oid */ + bool messages; /* True if the subscription wants to receive + * logical messages */ char *slotname; /* Name of the replication slot */ char *synccommit; /* Synchronous commit setting for worker */ char *walrcvtimeout; /* wal_receiver_timeout setting for worker */ diff --git a/src/include/replication/logicalproto.h b/src/include/replication/logicalproto.h index 058a955e20c..83741a2080d 100644 --- a/src/include/replication/logicalproto.h +++ b/src/include/replication/logicalproto.h @@ -191,6 +191,18 @@ typedef struct LogicalRepStreamAbortData TimestampTz abort_time; } LogicalRepStreamAbortData; +/* + * Logical decoding message information + */ +typedef struct LogicalRepMessageData +{ + XLogRecPtr lsn; + bool transactional; + const char *prefix; + Size message_size; /* length of message in bytes */ + const char *message; /* payload; may contain embedded nulls */ +} LogicalRepMessageData; + extern void logicalrep_write_begin(StringInfo out, ReorderBufferTXN *txn); extern void logicalrep_read_begin(StringInfo in, LogicalRepBeginData *begin_data); @@ -248,6 +260,7 @@ extern List *logicalrep_read_truncate(StringInfo in, bool *cascade, bool *restart_seqs); extern void logicalrep_write_message(StringInfo out, TransactionId xid, XLogRecPtr lsn, bool transactional, const char *prefix, Size sz, const char *message); +extern void logicalrep_read_message(StringInfo in, LogicalRepMessageData *msg_data); extern void logicalrep_write_rel(StringInfo out, TransactionId xid, Relation rel, Bitmapset *columns, PublishGencolsType include_gencols_type); diff --git a/src/include/replication/logicalworker.h b/src/include/replication/logicalworker.h index 7d748a28da8..3f745201c1a 100644 --- a/src/include/replication/logicalworker.h +++ b/src/include/replication/logicalworker.h @@ -16,6 +16,55 @@ extern PGDLLIMPORT volatile sig_atomic_t ParallelApplyMessagePending; +/* + * Forward-declared so that this header, which is included by several files + * unrelated to logical replication, does not have to include logicalproto.h. + * A handler that dereferences the struct must include that header itself. + */ +struct LogicalRepMessageData; + +/* + * Hook for extensions to handle logical decoding messages (see + * pg_logical_emit_message()) received from the publisher. Called only when + * the subscription has the "messages" option enabled. + * + * The handler runs in the apply worker, inside a transaction, with an active + * snapshot pushed. For a transactional message the handler's work is part of + * the remote transaction and commits with it; a non-transactional message is + * committed as an independent unit. The same message may be delivered more + * than once, so the handler must be idempotent. + * + * The payload is an arbitrary string of bytes. The WAL record makes no + * distinction between the text and the bytea form of + * pg_logical_emit_message(), and the bytes are transmitted without character + * set conversion, so the payload may contain embedded nulls and bytes that + * are not valid in the subscriber's encoding. Its length is given by + * msg->message_size, not by strlen(); the buffer is null-terminated only for + * convenience, and that null is not part of the payload. + * + * The handler always runs with the privileges of the subscription owner. + * The payload is chosen by whoever called pg_logical_emit_message() on the + * publisher, which by default any role may do, so a handler that acts on the + * payload gives every role on the publisher the subscription owner's + * privileges. A handler that needs less than that can switch to a less + * privileged role with SwitchToUntrustedUser(). It must take that role + * from its own configuration and never from the payload, which is written + * by the very user the switch defends against, and it should check the + * role with member_can_set_role() first: SwitchToUntrustedUser() throws an + * error if the subscription owner cannot SET ROLE to it, and that error + * loops as described below. MySubscription, declared in + * worker_internal.h, is available where the choice differs per + * subscription. + * + * Note that the apply worker runs with an empty search_path. An ERROR thrown + * by the handler restarts the apply worker, or disables the subscription if + * disable_on_error is set. Because the message is delivered again after the + * restart, a handler cannot reject a message by throwing an error; that only + * loops. A message the handler does not want must simply be ignored. + */ +typedef void (*LogicalRepMessageHandle_hook_type) (const struct LogicalRepMessageData *msg); +extern PGDLLIMPORT LogicalRepMessageHandle_hook_type LogicalRepMessageHandle_hook; + extern void ApplyWorkerMain(Datum main_arg); extern void ParallelApplyWorkerMain(Datum main_arg); extern void TableSyncWorkerMain(Datum main_arg); diff --git a/src/include/replication/walreceiver.h b/src/include/replication/walreceiver.h index 5cfe90f9989..42a1dab24b2 100644 --- a/src/include/replication/walreceiver.h +++ b/src/include/replication/walreceiver.h @@ -188,6 +188,7 @@ typedef struct * prepare time */ char *origin; /* Only publish data originating from the * specified origin */ + bool messages; /* Ask publisher to send logical messages */ } logical; } proto; } WalRcvStreamOptions; diff --git a/src/test/modules/Makefile b/src/test/modules/Makefile index 6c1504ac932..a8a31f2b23e 100644 --- a/src/test/modules/Makefile +++ b/src/test/modules/Makefile @@ -37,6 +37,7 @@ SUBDIRS = \ test_integerset \ test_json_parser \ test_lfind \ + test_logicalmsg_hooks \ test_lwlock_tranches \ test_misc \ test_oat_hooks \ diff --git a/src/test/modules/meson.build b/src/test/modules/meson.build index ef3cdb8faf9..f1aaeae71ab 100644 --- a/src/test/modules/meson.build +++ b/src/test/modules/meson.build @@ -39,6 +39,7 @@ subdir('test_int128') subdir('test_integerset') subdir('test_json_parser') subdir('test_lfind') +subdir('test_logicalmsg_hooks') subdir('test_lwlock_tranches') subdir('test_misc') subdir('test_oat_hooks') diff --git a/src/test/modules/test_logicalmsg_hooks/.gitignore b/src/test/modules/test_logicalmsg_hooks/.gitignore new file mode 100644 index 00000000000..5dcb3ff9723 --- /dev/null +++ b/src/test/modules/test_logicalmsg_hooks/.gitignore @@ -0,0 +1,4 @@ +# Generated subdirectories +/log/ +/results/ +/tmp_check/ diff --git a/src/test/modules/test_logicalmsg_hooks/Makefile b/src/test/modules/test_logicalmsg_hooks/Makefile new file mode 100644 index 00000000000..af129009b8a --- /dev/null +++ b/src/test/modules/test_logicalmsg_hooks/Makefile @@ -0,0 +1,20 @@ +# src/test/modules/test_logicalmsg_hooks/Makefile + +MODULE_big = test_logicalmsg_hooks +OBJS = \ + $(WIN32RES) \ + test_logicalmsg_hooks.o +PGFILEDESC = "test_logicalmsg_hooks - test logical replication message hooks" + +TAP_TESTS = 1 + +ifdef USE_PGXS +PG_CONFIG = pg_config +PGXS := $(shell $(PG_CONFIG) --pgxs) +include $(PGXS) +else +subdir = src/test/modules/test_logicalmsg_hooks +top_builddir = ../../../.. +include $(top_builddir)/src/Makefile.global +include $(top_srcdir)/contrib/contrib-global.mk +endif diff --git a/src/test/modules/test_logicalmsg_hooks/meson.build b/src/test/modules/test_logicalmsg_hooks/meson.build new file mode 100644 index 00000000000..8084d82e47b --- /dev/null +++ b/src/test/modules/test_logicalmsg_hooks/meson.build @@ -0,0 +1,28 @@ +# Copyright (c) 2022-2026, PostgreSQL Global Development Group + +test_logicalmsg_hooks_sources = files( + 'test_logicalmsg_hooks.c', +) + +if host_system == 'windows' + test_logicalmsg_hooks_sources += rc_lib_gen.process(win32ver_rc, extra_args: [ + '--NAME', 'test_logicalmsg_hooks', + '--FILEDESC', 'test_logicalmsg_hooks - test logical message hooks',]) +endif + +test_logicalmsg_hooks = shared_module('test_logicalmsg_hooks', + test_logicalmsg_hooks_sources, + kwargs: pg_test_mod_args, +) +test_install_libs += test_logicalmsg_hooks + +tests += { + 'name': 'test_logicalmsg_hooks', + 'sd': meson.current_source_dir(), + 'bd': meson.current_build_dir(), + 'tap': { + 'tests': [ + 't/001_basic.pl', + ], + }, +} diff --git a/src/test/modules/test_logicalmsg_hooks/t/001_basic.pl b/src/test/modules/test_logicalmsg_hooks/t/001_basic.pl new file mode 100644 index 00000000000..a86de1273ab --- /dev/null +++ b/src/test/modules/test_logicalmsg_hooks/t/001_basic.pl @@ -0,0 +1,318 @@ +# Copyright (c) 2026, PostgreSQL Global Development Group + +# Test LogicalRepMessageHandle_hook. The test module records every message it +# receives into public.test_logicalmsg_log on the subscriber, so most checks +# here are plain queries against that table. + +use strict; +use warnings FATAL => 'all'; + +use PostgreSQL::Test::Cluster; +use PostgreSQL::Test::Utils; +use Test::More; + +my $pub = PostgreSQL::Test::Cluster->new('publisher'); +$pub->init(allows_streaming => 'logical'); + +# Keep the memory limit low so that the transactions below are streamed. +$pub->append_conf( + 'postgresql.conf', q{ +logical_decoding_work_mem = 64kB +max_prepared_transactions = 10 +}); +$pub->start; + +my $sub = PostgreSQL::Test::Cluster->new('subscriber'); +$sub->init(allows_streaming => 'logical'); +$sub->append_conf( + 'postgresql.conf', q{ +shared_preload_libraries = 'test_logicalmsg_hooks' +max_parallel_apply_workers_per_subscription = 2 +max_prepared_transactions = 10 +}); +$sub->start; + +$pub->safe_psql( + 'postgres', qq{ +CREATE TABLE test (a int); +INSERT INTO test VALUES (1); +CREATE PUBLICATION pub FOR ALL TABLES; +}); + +my $pub_connstr = $pub->connstr . ' dbname=postgres'; + +$sub->safe_psql( + 'postgres', qq{ +CREATE TABLE test (a int primary key); +CREATE SUBSCRIPTION sub CONNECTION '$pub_connstr' PUBLICATION pub WITH (messages = true, disable_on_error = true) +}); + +$sub->wait_for_subscription_sync($pub, 'sub'); + +# Number of messages recorded under the given prefix. +sub logged_count +{ + my ($prefix) = @_; + + return $sub->safe_psql('postgres', + "SELECT count(*) FROM test_logicalmsg_log WHERE prefix = '$prefix'"); +} + +# Verify that a transactional message is passed to the hook, and that the +# message data given to the hook is correct. +$pub->safe_psql('postgres', + q{SELECT pg_logical_emit_message(true, 'basic', 'transactional message')} +); +$pub->wait_for_catchup('sub'); + +is( $sub->safe_psql( + 'postgres', + q{SELECT transactional, message_size, message FROM test_logicalmsg_log + WHERE prefix = 'basic'}), + 't|21|transactional message', + "transactional message is passed to the hook"); + +# Verify that a non-transactional message is applied even if the transaction +# that emitted it rolls back. +$pub->safe_psql( + 'postgres', q{ +BEGIN; +SELECT pg_logical_emit_message(false, 'nontransactional', 'emitted then rolled back'); +ROLLBACK; +}); + +# Commit another message to flush WAL, as ROLLBACK doesn't flush. +$pub->safe_psql('postgres', + q{SELECT pg_logical_emit_message(true, 'flush', 'flush')}); +$pub->wait_for_catchup('sub'); + +is( $sub->safe_psql( + 'postgres', + q{SELECT transactional, message FROM test_logicalmsg_log + WHERE prefix = 'nontransactional'}), + 'f|emitted then rolled back', + "non-transactional message of an aborted transaction is applied"); + +# Verify that a transactional message emitted in a subtransaction that is +# rolled back is not passed to the hook. +$pub->safe_psql( + 'postgres', q{ +BEGIN; +SAVEPOINT sp; +SELECT pg_logical_emit_message(true, 'subxact_aborted', 'not delivered'); +ROLLBACK TO SAVEPOINT sp; +SELECT pg_logical_emit_message(true, 'subxact_committed', 'delivered'); +COMMIT; +}); +$pub->wait_for_catchup('sub'); + +is(logged_count('subxact_aborted'), + 0, "message of an aborted subtransaction is not applied"); +is(logged_count('subxact_committed'), + 1, "message of a committed subtransaction is applied"); + +# Verify that transactional messages are applied in streaming transactions, +# both when the leader apply worker serializes the transaction to a file and +# when it hands the transaction to a parallel apply worker. +foreach my $mode ('on', 'parallel') +{ + $sub->safe_psql('postgres', + "ALTER SUBSCRIPTION sub SET (streaming = $mode)"); + + # Reconnect so that the new streaming mode takes effect immediately. + $sub->safe_psql('postgres', 'ALTER SUBSCRIPTION sub DISABLE'); + $sub->safe_psql('postgres', 'ALTER SUBSCRIPTION sub ENABLE'); + + $pub->safe_psql('postgres', + q{SELECT pg_stat_reset_replication_slot('sub')}); + + $pub->safe_psql( + 'postgres', qq{ +BEGIN; +SAVEPOINT sp; +SELECT pg_logical_emit_message(true, 'streaming_subxact_aborted_$mode', 'not delivered'); +INSERT INTO test SELECT generate_series(1000, 6000); +ROLLBACK TO SAVEPOINT sp; +SELECT pg_logical_emit_message(true, 'streaming_$mode', 'streamed message'); +COMMIT; +}); + $pub->wait_for_catchup('sub'); + + # Make sure the transaction really was streamed, so that this test cannot + # silently degrade into a non-streaming one. + $pub->poll_query_until('postgres', + q{SELECT stream_txns > 0 FROM pg_stat_replication_slots WHERE slot_name = 'sub'} + ) or die "transaction was not streamed"; + + is(logged_count("streaming_subxact_aborted_$mode"), 0, + "message of an aborted streamed subtransaction is not applied (streaming = $mode)" + ); + is(logged_count("streaming_$mode"), + 1, + "message in a streamed transaction is applied (streaming = $mode)"); + + $pub->safe_psql('postgres', 'TRUNCATE test'); + $pub->wait_for_catchup('sub'); +} + +# Verify that a transaction whose only change is a message is applied when it +# is streamed. There is no volume to trigger streaming, so force it. The +# setting is picked up by the walsender started when the subscription is +# re-enabled below. +$pub->safe_psql('postgres', + 'ALTER SYSTEM SET debug_logical_replication_streaming = immediate'); +$pub->reload; +$pub->wait_for_log( + qr/parameter "debug_logical_replication_streaming" changed to "immediate"/ +); + +foreach my $mode ('on', 'parallel') +{ + $sub->safe_psql('postgres', + "ALTER SUBSCRIPTION sub SET (streaming = $mode)"); + $sub->safe_psql('postgres', 'ALTER SUBSCRIPTION sub DISABLE'); + $sub->safe_psql('postgres', 'ALTER SUBSCRIPTION sub ENABLE'); + + $pub->safe_psql('postgres', + q{SELECT pg_stat_reset_replication_slot('sub')}); + + $pub->safe_psql( + 'postgres', qq{ +BEGIN; +SELECT pg_logical_emit_message(true, 'streaming_only_$mode', 'the only change'); +COMMIT; +}); + $pub->wait_for_catchup('sub'); + + $pub->poll_query_until('postgres', + q{SELECT stream_txns > 0 FROM pg_stat_replication_slots WHERE slot_name = 'sub'} + ) or die "transaction was not streamed"; + + is(logged_count("streaming_only_$mode"), 1, + "message that is the only change of a streamed transaction is applied (streaming = $mode)" + ); +} + +$pub->safe_psql('postgres', + 'ALTER SYSTEM RESET debug_logical_replication_streaming'); +$pub->reload; +$pub->wait_for_log( + qr/parameter "debug_logical_replication_streaming" removed from configuration file/ +); + +$sub->safe_psql('postgres', 'ALTER SUBSCRIPTION sub SET (streaming = off)'); + +# Verify that ALTER SUBSCRIPTION ... SKIP skips a transactional message along +# with the rest of its transaction. The message is emitted before the +# conflicting change, so the hook runs and records the message on the first +# attempt; that work must be rolled back with the failed transaction, and must +# not be redone once the transaction is skipped. +$pub->safe_psql('postgres', 'INSERT INTO test VALUES (1)'); +$pub->wait_for_catchup('sub'); + +my $log_location = -s $sub->logfile; + +$pub->safe_psql( + 'postgres', q{ +BEGIN; +SELECT pg_logical_emit_message(true, 'skipped', 'rolled back, then skipped'); +INSERT INTO test VALUES (1); +COMMIT; +}); + +# Wait until the conflict disables the subscription. +$sub->poll_query_until('postgres', + q{SELECT subenabled = FALSE FROM pg_subscription WHERE subname = 'sub'}); + +# The hook did run before the transaction failed. +ok( $sub->log_contains( + qr/LOG[^\n]+received message: [^\n]+prefix: skipped/, + $log_location), + "hook is called for a message in a transaction that later fails"); + +# ... but its work was rolled back with the transaction. +is(logged_count('skipped'), 0, + "work done by the hook is rolled back with the remote transaction"); + +# Get the finish LSN of the failed transaction. +my $contents = slurp_file($sub->logfile, $log_location); +$contents =~ + qr/conflict detected on relation "public.test".*\n.*DETAIL:.*Could not apply remote change.*\n.*Key already exists in unique index "test_pkey", modified in transaction \d+: key .*, local row .*\n.*CONTEXT:.* for replication target relation "public.test" in transaction \d+, finished at ([[:xdigit:]]+\/[[:xdigit:]]+)/m + or die "could not get error-LSN"; +my $lsn = $1; + +$log_location = -s $sub->logfile; + +# Set skip LSN and re-enable the subscription. +$sub->safe_psql('postgres', qq{ALTER SUBSCRIPTION sub SKIP (lsn = '$lsn')}); +$sub->safe_psql('postgres', 'ALTER SUBSCRIPTION sub ENABLE'); + +# Wait for the failed transaction to be skipped. +$sub->poll_query_until('postgres', + q{SELECT subskiplsn = '0/0' FROM pg_subscription WHERE subname = 'sub'}); + +ok( !$sub->log_contains( + qr/LOG[^\n]+received message: [^\n]+prefix: skipped/, + $log_location), + "hook is not called for a message in a skipped transaction"); +is(logged_count('skipped'), 0, + "message in a skipped transaction is not applied"); + +# Verify that a message in a prepared transaction is applied when the +# transaction is committed, and discarded when it is rolled back. +$sub->safe_psql('postgres', 'ALTER SUBSCRIPTION sub DISABLE'); +$sub->safe_psql('postgres', 'ALTER SUBSCRIPTION sub SET (two_phase = on)'); +$sub->safe_psql('postgres', 'ALTER SUBSCRIPTION sub ENABLE'); +$sub->poll_query_until('postgres', + q{SELECT subtwophasestate = 'e' FROM pg_subscription WHERE subname = 'sub'} +); + +$pub->safe_psql( + 'postgres', q{ +BEGIN; +SELECT pg_logical_emit_message(true, 'prepared', 'committed later'); +PREPARE TRANSACTION 'msg_commit'; +}); +$pub->wait_for_catchup('sub'); + +is(logged_count('prepared'), + 0, "message in a prepared transaction is not applied before commit"); + +$pub->safe_psql('postgres', q{COMMIT PREPARED 'msg_commit'}); +$pub->wait_for_catchup('sub'); + +is(logged_count('prepared'), + 1, "message in a prepared transaction is applied at commit"); + +$pub->safe_psql( + 'postgres', q{ +BEGIN; +SELECT pg_logical_emit_message(true, 'prepared_rollback', 'rolled back later'); +PREPARE TRANSACTION 'msg_rollback'; +}); +$pub->wait_for_catchup('sub'); +$pub->safe_psql('postgres', q{ROLLBACK PREPARED 'msg_rollback'}); +$pub->wait_for_catchup('sub'); + +is(logged_count('prepared_rollback'), + 0, "message in a prepared transaction is discarded at rollback"); + +# Verify that no message reaches the hook once the option is turned off, which +# is also its default. +$sub->safe_psql('postgres', 'ALTER SUBSCRIPTION sub SET (messages = false)'); + +# Reconnect so that the change takes effect immediately. +$sub->safe_psql('postgres', 'ALTER SUBSCRIPTION sub DISABLE'); +$sub->safe_psql('postgres', 'ALTER SUBSCRIPTION sub ENABLE'); + +$pub->safe_psql('postgres', + q{SELECT pg_logical_emit_message(true, 'not_requested', 'messages is off')} +); +$pub->wait_for_catchup('sub'); + +is(logged_count('not_requested'), + 0, "no message is passed to the hook when messages is off"); + +$sub->stop; +$pub->stop; +done_testing(); diff --git a/src/test/modules/test_logicalmsg_hooks/test_logicalmsg_hooks.c b/src/test/modules/test_logicalmsg_hooks/test_logicalmsg_hooks.c new file mode 100644 index 00000000000..d8d6fc7c619 --- /dev/null +++ b/src/test/modules/test_logicalmsg_hooks/test_logicalmsg_hooks.c @@ -0,0 +1,140 @@ +/*-------------------------------------------------------------------------- + * + * test_logicalmsg_hooks.c + * Code for testing LogicalRepMessageHandle_hook + * + * The handler records every logical decoding message it receives into the + * table public.test_logicalmsg_log, so that tests can assert on the contents + * of a table rather than on the server log. Recording the messages this way + * also exercises the transactional behavior of the hook: a transactional + * message is recorded as part of the remote transaction that emitted it, and + * therefore disappears together with it if that transaction is rolled back or + * skipped, while a non-transactional message is recorded independently. + * + * The handler creates that table itself on the first message, so a test does + * not have to set it up. For a transactional message the creation is part of + * the remote transaction too, and is rolled back with it, so the table does + * not exist until some message has been applied successfully; a test must not + * query it before then. + * + * The messages are also logged, which lets a test tell "the handler ran but + * its work was rolled back" apart from "the handler was never called". + * + * Portions Copyright (c) 1996-2026, PostgreSQL Global Development Group + * Portions Copyright (c) 1994, Regents of the University of California + * + * IDENTIFICATION + * src/test/modules/test_logicalmsg_hooks/test_logicalmsg_hooks.c + * + * ------------------------------------------------------------------------- + */ + +#include "postgres.h" + +#include "catalog/namespace.h" +#include "catalog/pg_type.h" +#include "executor/spi.h" +#include "replication/logicalproto.h" +#include "replication/logicalworker.h" +#include "utils/builtins.h" +#include "utils/lsyscache.h" +#include "utils/pg_lsn.h" + +PG_MODULE_MAGIC; + +#define LOG_SCHEMA "public" +#define LOG_TABLE "test_logicalmsg_log" + +static LogicalRepMessageHandle_hook_type prev_logical_message_handler = NULL; +static void test_logical_message_handler(const LogicalRepMessageData *msg); + +/* + * Module load callback + */ +void +_PG_init(void) +{ + prev_logical_message_handler = LogicalRepMessageHandle_hook; + LogicalRepMessageHandle_hook = &test_logical_message_handler; +} + +/* + * Create the log table if not exists. + */ +static void +ensure_log_table_exists(void) +{ + Oid relid; + int ret; + + relid = get_relname_relid(LOG_TABLE, get_namespace_oid(LOG_SCHEMA, true)); + if (OidIsValid(relid)) + return; + + SPI_connect(); + + ret = SPI_execute("CREATE TABLE " LOG_SCHEMA "." LOG_TABLE + "(lsn pg_lsn, transactional boolean, prefix text, message_size int, message text)", + false, 0); + if (ret != SPI_OK_UTILITY) + elog(ERROR, "could not create " LOG_SCHEMA "." LOG_TABLE); + + SPI_finish(); +} + +static void +test_logical_message_handler(const LogicalRepMessageData *msg) +{ + Oid argtypes[5] = {LSNOID, BOOLOID, TEXTOID, INT4OID, TEXTOID}; + Datum values[5]; + int ret; + + /* + * Give a handler installed before ours its turn, so that several + * extensions can act on the same message. + */ + if (prev_logical_message_handler) + (*prev_logical_message_handler) (msg); + + ereport(LOG, + (errmsg("received message: LSN %X/%08X, prefix: %s, size: %zu, transactional: %d", + LSN_FORMAT_ARGS(msg->lsn), msg->prefix, + msg->message_size, msg->transactional))); + + ensure_log_table_exists(); + + /* + * Pass the message as query parameters rather than interpolating it into + * the query text. Message contents come from the publisher and are not to + * be trusted, and the payload may contain characters that would need + * quoting. Note also that message_size, not strlen(), is authoritative + * for the payload length. + * + * The payload is recorded as text because the accompanying test only ever + * emits text messages. A handler must not assume that in general: a + * payload is an arbitrary string of bytes, which may contain embedded + * nulls and bytes that are not valid in the subscriber's encoding. See + * the comments on LogicalRepMessageHandle_hook_type. + */ + values[0] = LSNGetDatum(msg->lsn); + values[1] = BoolGetDatum(msg->transactional); + values[2] = CStringGetTextDatum(msg->prefix); + values[3] = Int32GetDatum((int32) msg->message_size); + values[4] = PointerGetDatum(cstring_to_text_with_len(msg->message, + msg->message_size)); + + SPI_connect(); + + /* + * The apply worker runs with an empty search_path, so the table name must + * be schema-qualified. + */ + ret = SPI_execute_with_args("INSERT INTO " LOG_SCHEMA "." LOG_TABLE + " (lsn, transactional, prefix, message_size, message)" + " VALUES ($1, $2, $3, $4, $5)", + 5, argtypes, values, NULL, false, 0); + if (ret != SPI_OK_INSERT) + elog(ERROR, "could not record logical message: SPI returned %d", ret); + + SPI_finish(); +} diff --git a/src/test/regress/expected/subscription.out b/src/test/regress/expected/subscription.out index 6b2825a6a14..118e1ee3ce8 100644 --- a/src/test/regress/expected/subscription.out +++ b/src/test/regress/expected/subscription.out @@ -139,18 +139,18 @@ CREATE SUBSCRIPTION regress_testsub4 CONNECTION 'dbname=regress_doesnotexist' PU WARNING: subscription was created, but is not connected HINT: To initiate replication, you must manually create the replication slot, enable the subscription, and alter the subscription to refresh publications. \dRs+ regress_testsub4 - List of subscriptions - Name | Owner | Enabled | Publication | Binary | Streaming | Two-phase commit | Disable on error | Origin | Password required | Run as owner? | Failover | Server | Retain dead tuples | Max retention duration | Retention active | Synchronous commit | Conninfo | Receiver timeout | Skip LSN | Description -------------------+---------------------------+---------+-------------+--------+-----------+------------------+------------------+--------+-------------------+---------------+----------+--------+--------------------+------------------------+------------------+--------------------+-----------------------------+------------------+------------+------------- - regress_testsub4 | regress_subscription_user | f | {testpub} | f | parallel | d | f | none | t | f | f | | f | 0 | f | off | dbname=regress_doesnotexist | -1 | 0/00000000 | + List of subscriptions + Name | Owner | Enabled | Publication | Binary | Streaming | Two-phase commit | Disable on error | Origin | Password required | Run as owner? | Failover | Server | Retain dead tuples | Max retention duration | Retention active | Messages | Synchronous commit | Conninfo | Receiver timeout | Skip LSN | Description +------------------+---------------------------+---------+-------------+--------+-----------+------------------+------------------+--------+-------------------+---------------+----------+--------+--------------------+------------------------+------------------+----------+--------------------+-----------------------------+------------------+------------+------------- + regress_testsub4 | regress_subscription_user | f | {testpub} | f | parallel | d | f | none | t | f | f | | f | 0 | f | f | off | dbname=regress_doesnotexist | -1 | 0/00000000 | (1 row) ALTER SUBSCRIPTION regress_testsub4 SET (origin = any); \dRs+ regress_testsub4 - List of subscriptions - Name | Owner | Enabled | Publication | Binary | Streaming | Two-phase commit | Disable on error | Origin | Password required | Run as owner? | Failover | Server | Retain dead tuples | Max retention duration | Retention active | Synchronous commit | Conninfo | Receiver timeout | Skip LSN | Description -------------------+---------------------------+---------+-------------+--------+-----------+------------------+------------------+--------+-------------------+---------------+----------+--------+--------------------+------------------------+------------------+--------------------+-----------------------------+------------------+------------+------------- - regress_testsub4 | regress_subscription_user | f | {testpub} | f | parallel | d | f | any | t | f | f | | f | 0 | f | off | dbname=regress_doesnotexist | -1 | 0/00000000 | + List of subscriptions + Name | Owner | Enabled | Publication | Binary | Streaming | Two-phase commit | Disable on error | Origin | Password required | Run as owner? | Failover | Server | Retain dead tuples | Max retention duration | Retention active | Messages | Synchronous commit | Conninfo | Receiver timeout | Skip LSN | Description +------------------+---------------------------+---------+-------------+--------+-----------+------------------+------------------+--------+-------------------+---------------+----------+--------+--------------------+------------------------+------------------+----------+--------------------+-----------------------------+------------------+------------+------------- + regress_testsub4 | regress_subscription_user | f | {testpub} | f | parallel | d | f | any | t | f | f | | f | 0 | f | f | off | dbname=regress_doesnotexist | -1 | 0/00000000 | (1 row) DROP SUBSCRIPTION regress_testsub3; @@ -277,10 +277,10 @@ ALTER SUBSCRIPTION regress_testsub CONNECTION 'foobar'; ERROR: invalid connection string syntax: missing "=" after "foobar" in connection info string \dRs+ - List of subscriptions - Name | Owner | Enabled | Publication | Binary | Streaming | Two-phase commit | Disable on error | Origin | Password required | Run as owner? | Failover | Server | Retain dead tuples | Max retention duration | Retention active | Synchronous commit | Conninfo | Receiver timeout | Skip LSN | Description ------------------+---------------------------+---------+-------------+--------+-----------+------------------+------------------+--------+-------------------+---------------+----------+--------+--------------------+------------------------+------------------+--------------------+-----------------------------+------------------+------------+------------------- - regress_testsub | regress_subscription_user | f | {testpub} | f | parallel | d | f | any | t | f | f | | f | 0 | f | off | dbname=regress_doesnotexist | -1 | 0/00000000 | test subscription + List of subscriptions + Name | Owner | Enabled | Publication | Binary | Streaming | Two-phase commit | Disable on error | Origin | Password required | Run as owner? | Failover | Server | Retain dead tuples | Max retention duration | Retention active | Messages | Synchronous commit | Conninfo | Receiver timeout | Skip LSN | Description +-----------------+---------------------------+---------+-------------+--------+-----------+------------------+------------------+--------+-------------------+---------------+----------+--------+--------------------+------------------------+------------------+----------+--------------------+-----------------------------+------------------+------------+------------------- + regress_testsub | regress_subscription_user | f | {testpub} | f | parallel | d | f | any | t | f | f | | f | 0 | f | f | off | dbname=regress_doesnotexist | -1 | 0/00000000 | test subscription (1 row) ALTER SUBSCRIPTION regress_testsub SET PUBLICATION testpub2, testpub3 WITH (refresh = false); @@ -289,10 +289,10 @@ ALTER SUBSCRIPTION regress_testsub SET (slot_name = 'newname'); ALTER SUBSCRIPTION regress_testsub SET (password_required = false); ALTER SUBSCRIPTION regress_testsub SET (run_as_owner = true); \dRs+ - List of subscriptions - Name | Owner | Enabled | Publication | Binary | Streaming | Two-phase commit | Disable on error | Origin | Password required | Run as owner? | Failover | Server | Retain dead tuples | Max retention duration | Retention active | Synchronous commit | Conninfo | Receiver timeout | Skip LSN | Description ------------------+---------------------------+---------+---------------------+--------+-----------+------------------+------------------+--------+-------------------+---------------+----------+--------+--------------------+------------------------+------------------+--------------------+------------------------------+------------------+------------+------------------- - regress_testsub | regress_subscription_user | f | {testpub2,testpub3} | f | parallel | d | f | any | f | t | f | | f | 0 | f | off | dbname=regress_doesnotexist2 | -1 | 0/00000000 | test subscription + List of subscriptions + Name | Owner | Enabled | Publication | Binary | Streaming | Two-phase commit | Disable on error | Origin | Password required | Run as owner? | Failover | Server | Retain dead tuples | Max retention duration | Retention active | Messages | Synchronous commit | Conninfo | Receiver timeout | Skip LSN | Description +-----------------+---------------------------+---------+---------------------+--------+-----------+------------------+------------------+--------+-------------------+---------------+----------+--------+--------------------+------------------------+------------------+----------+--------------------+------------------------------+------------------+------------+------------------- + regress_testsub | regress_subscription_user | f | {testpub2,testpub3} | f | parallel | d | f | any | f | t | f | | f | 0 | f | f | off | dbname=regress_doesnotexist2 | -1 | 0/00000000 | test subscription (1 row) ALTER SUBSCRIPTION regress_testsub SET (password_required = true); @@ -308,10 +308,10 @@ ERROR: unrecognized subscription parameter: "create_slot" -- ok ALTER SUBSCRIPTION regress_testsub SKIP (lsn = '0/12345'); \dRs+ - List of subscriptions - Name | Owner | Enabled | Publication | Binary | Streaming | Two-phase commit | Disable on error | Origin | Password required | Run as owner? | Failover | Server | Retain dead tuples | Max retention duration | Retention active | Synchronous commit | Conninfo | Receiver timeout | Skip LSN | Description ------------------+---------------------------+---------+---------------------+--------+-----------+------------------+------------------+--------+-------------------+---------------+----------+--------+--------------------+------------------------+------------------+--------------------+------------------------------+------------------+------------+------------------- - regress_testsub | regress_subscription_user | f | {testpub2,testpub3} | f | parallel | d | f | any | t | f | f | | f | 0 | f | off | dbname=regress_doesnotexist2 | -1 | 0/00012345 | test subscription + List of subscriptions + Name | Owner | Enabled | Publication | Binary | Streaming | Two-phase commit | Disable on error | Origin | Password required | Run as owner? | Failover | Server | Retain dead tuples | Max retention duration | Retention active | Messages | Synchronous commit | Conninfo | Receiver timeout | Skip LSN | Description +-----------------+---------------------------+---------+---------------------+--------+-----------+------------------+------------------+--------+-------------------+---------------+----------+--------+--------------------+------------------------+------------------+----------+--------------------+------------------------------+------------------+------------+------------------- + regress_testsub | regress_subscription_user | f | {testpub2,testpub3} | f | parallel | d | f | any | t | f | f | | f | 0 | f | f | off | dbname=regress_doesnotexist2 | -1 | 0/00012345 | test subscription (1 row) -- ok - with lsn = NONE @@ -320,10 +320,10 @@ ALTER SUBSCRIPTION regress_testsub SKIP (lsn = NONE); ALTER SUBSCRIPTION regress_testsub SKIP (lsn = '0/0'); ERROR: invalid WAL location (LSN): 0/0 \dRs+ - List of subscriptions - Name | Owner | Enabled | Publication | Binary | Streaming | Two-phase commit | Disable on error | Origin | Password required | Run as owner? | Failover | Server | Retain dead tuples | Max retention duration | Retention active | Synchronous commit | Conninfo | Receiver timeout | Skip LSN | Description ------------------+---------------------------+---------+---------------------+--------+-----------+------------------+------------------+--------+-------------------+---------------+----------+--------+--------------------+------------------------+------------------+--------------------+------------------------------+------------------+------------+------------------- - regress_testsub | regress_subscription_user | f | {testpub2,testpub3} | f | parallel | d | f | any | t | f | f | | f | 0 | f | off | dbname=regress_doesnotexist2 | -1 | 0/00000000 | test subscription + List of subscriptions + Name | Owner | Enabled | Publication | Binary | Streaming | Two-phase commit | Disable on error | Origin | Password required | Run as owner? | Failover | Server | Retain dead tuples | Max retention duration | Retention active | Messages | Synchronous commit | Conninfo | Receiver timeout | Skip LSN | Description +-----------------+---------------------------+---------+---------------------+--------+-----------+------------------+------------------+--------+-------------------+---------------+----------+--------+--------------------+------------------------+------------------+----------+--------------------+------------------------------+------------------+------------+------------------- + regress_testsub | regress_subscription_user | f | {testpub2,testpub3} | f | parallel | d | f | any | t | f | f | | f | 0 | f | f | off | dbname=regress_doesnotexist2 | -1 | 0/00000000 | test subscription (1 row) BEGIN; @@ -359,10 +359,10 @@ ALTER SUBSCRIPTION regress_testsub_foo SET (wal_receiver_timeout = '80s'); ALTER SUBSCRIPTION regress_testsub_foo SET (wal_receiver_timeout = 'foobar'); ERROR: invalid value for parameter "wal_receiver_timeout": "foobar" \dRs+ - List of subscriptions - Name | Owner | Enabled | Publication | Binary | Streaming | Two-phase commit | Disable on error | Origin | Password required | Run as owner? | Failover | Server | Retain dead tuples | Max retention duration | Retention active | Synchronous commit | Conninfo | Receiver timeout | Skip LSN | Description ----------------------+---------------------------+---------+---------------------+--------+-----------+------------------+------------------+--------+-------------------+---------------+----------+--------+--------------------+------------------------+------------------+--------------------+------------------------------+------------------+------------+------------------- - regress_testsub_foo | regress_subscription_user | f | {testpub2,testpub3} | f | parallel | d | f | any | t | f | f | | f | 0 | f | local | dbname=regress_doesnotexist2 | 80s | 0/00000000 | test subscription + List of subscriptions + Name | Owner | Enabled | Publication | Binary | Streaming | Two-phase commit | Disable on error | Origin | Password required | Run as owner? | Failover | Server | Retain dead tuples | Max retention duration | Retention active | Messages | Synchronous commit | Conninfo | Receiver timeout | Skip LSN | Description +---------------------+---------------------------+---------+---------------------+--------+-----------+------------------+------------------+--------+-------------------+---------------+----------+--------+--------------------+------------------------+------------------+----------+--------------------+------------------------------+------------------+------------+------------------- + regress_testsub_foo | regress_subscription_user | f | {testpub2,testpub3} | f | parallel | d | f | any | t | f | f | | f | 0 | f | f | local | dbname=regress_doesnotexist2 | 80s | 0/00000000 | test subscription (1 row) -- rename back to keep the rest simple @@ -391,19 +391,19 @@ CREATE SUBSCRIPTION regress_testsub CONNECTION 'dbname=regress_doesnotexist' PUB WARNING: subscription was created, but is not connected HINT: To initiate replication, you must manually create the replication slot, enable the subscription, and alter the subscription to refresh publications. \dRs+ - List of subscriptions - Name | Owner | Enabled | Publication | Binary | Streaming | Two-phase commit | Disable on error | Origin | Password required | Run as owner? | Failover | Server | Retain dead tuples | Max retention duration | Retention active | Synchronous commit | Conninfo | Receiver timeout | Skip LSN | Description ------------------+---------------------------+---------+-------------+--------+-----------+------------------+------------------+--------+-------------------+---------------+----------+--------+--------------------+------------------------+------------------+--------------------+-----------------------------+------------------+------------+------------- - regress_testsub | regress_subscription_user | f | {testpub} | t | parallel | d | f | any | t | f | f | | f | 0 | f | off | dbname=regress_doesnotexist | -1 | 0/00000000 | + List of subscriptions + Name | Owner | Enabled | Publication | Binary | Streaming | Two-phase commit | Disable on error | Origin | Password required | Run as owner? | Failover | Server | Retain dead tuples | Max retention duration | Retention active | Messages | Synchronous commit | Conninfo | Receiver timeout | Skip LSN | Description +-----------------+---------------------------+---------+-------------+--------+-----------+------------------+------------------+--------+-------------------+---------------+----------+--------+--------------------+------------------------+------------------+----------+--------------------+-----------------------------+------------------+------------+------------- + regress_testsub | regress_subscription_user | f | {testpub} | t | parallel | d | f | any | t | f | f | | f | 0 | f | f | off | dbname=regress_doesnotexist | -1 | 0/00000000 | (1 row) ALTER SUBSCRIPTION regress_testsub SET (binary = false); ALTER SUBSCRIPTION regress_testsub SET (slot_name = NONE); \dRs+ - List of subscriptions - Name | Owner | Enabled | Publication | Binary | Streaming | Two-phase commit | Disable on error | Origin | Password required | Run as owner? | Failover | Server | Retain dead tuples | Max retention duration | Retention active | Synchronous commit | Conninfo | Receiver timeout | Skip LSN | Description ------------------+---------------------------+---------+-------------+--------+-----------+------------------+------------------+--------+-------------------+---------------+----------+--------+--------------------+------------------------+------------------+--------------------+-----------------------------+------------------+------------+------------- - regress_testsub | regress_subscription_user | f | {testpub} | f | parallel | d | f | any | t | f | f | | f | 0 | f | off | dbname=regress_doesnotexist | -1 | 0/00000000 | + List of subscriptions + Name | Owner | Enabled | Publication | Binary | Streaming | Two-phase commit | Disable on error | Origin | Password required | Run as owner? | Failover | Server | Retain dead tuples | Max retention duration | Retention active | Messages | Synchronous commit | Conninfo | Receiver timeout | Skip LSN | Description +-----------------+---------------------------+---------+-------------+--------+-----------+------------------+------------------+--------+-------------------+---------------+----------+--------+--------------------+------------------------+------------------+----------+--------------------+-----------------------------+------------------+------------+------------- + regress_testsub | regress_subscription_user | f | {testpub} | f | parallel | d | f | any | t | f | f | | f | 0 | f | f | off | dbname=regress_doesnotexist | -1 | 0/00000000 | (1 row) DROP SUBSCRIPTION regress_testsub; @@ -415,27 +415,27 @@ CREATE SUBSCRIPTION regress_testsub CONNECTION 'dbname=regress_doesnotexist' PUB WARNING: subscription was created, but is not connected HINT: To initiate replication, you must manually create the replication slot, enable the subscription, and alter the subscription to refresh publications. \dRs+ - List of subscriptions - Name | Owner | Enabled | Publication | Binary | Streaming | Two-phase commit | Disable on error | Origin | Password required | Run as owner? | Failover | Server | Retain dead tuples | Max retention duration | Retention active | Synchronous commit | Conninfo | Receiver timeout | Skip LSN | Description ------------------+---------------------------+---------+-------------+--------+-----------+------------------+------------------+--------+-------------------+---------------+----------+--------+--------------------+------------------------+------------------+--------------------+-----------------------------+------------------+------------+------------- - regress_testsub | regress_subscription_user | f | {testpub} | f | on | d | f | any | t | f | f | | f | 0 | f | off | dbname=regress_doesnotexist | -1 | 0/00000000 | + List of subscriptions + Name | Owner | Enabled | Publication | Binary | Streaming | Two-phase commit | Disable on error | Origin | Password required | Run as owner? | Failover | Server | Retain dead tuples | Max retention duration | Retention active | Messages | Synchronous commit | Conninfo | Receiver timeout | Skip LSN | Description +-----------------+---------------------------+---------+-------------+--------+-----------+------------------+------------------+--------+-------------------+---------------+----------+--------+--------------------+------------------------+------------------+----------+--------------------+-----------------------------+------------------+------------+------------- + regress_testsub | regress_subscription_user | f | {testpub} | f | on | d | f | any | t | f | f | | f | 0 | f | f | off | dbname=regress_doesnotexist | -1 | 0/00000000 | (1 row) ALTER SUBSCRIPTION regress_testsub SET (streaming = parallel); \dRs+ - List of subscriptions - Name | Owner | Enabled | Publication | Binary | Streaming | Two-phase commit | Disable on error | Origin | Password required | Run as owner? | Failover | Server | Retain dead tuples | Max retention duration | Retention active | Synchronous commit | Conninfo | Receiver timeout | Skip LSN | Description ------------------+---------------------------+---------+-------------+--------+-----------+------------------+------------------+--------+-------------------+---------------+----------+--------+--------------------+------------------------+------------------+--------------------+-----------------------------+------------------+------------+------------- - regress_testsub | regress_subscription_user | f | {testpub} | f | parallel | d | f | any | t | f | f | | f | 0 | f | off | dbname=regress_doesnotexist | -1 | 0/00000000 | + List of subscriptions + Name | Owner | Enabled | Publication | Binary | Streaming | Two-phase commit | Disable on error | Origin | Password required | Run as owner? | Failover | Server | Retain dead tuples | Max retention duration | Retention active | Messages | Synchronous commit | Conninfo | Receiver timeout | Skip LSN | Description +-----------------+---------------------------+---------+-------------+--------+-----------+------------------+------------------+--------+-------------------+---------------+----------+--------+--------------------+------------------------+------------------+----------+--------------------+-----------------------------+------------------+------------+------------- + regress_testsub | regress_subscription_user | f | {testpub} | f | parallel | d | f | any | t | f | f | | f | 0 | f | f | off | dbname=regress_doesnotexist | -1 | 0/00000000 | (1 row) ALTER SUBSCRIPTION regress_testsub SET (streaming = false); ALTER SUBSCRIPTION regress_testsub SET (slot_name = NONE); \dRs+ - List of subscriptions - Name | Owner | Enabled | Publication | Binary | Streaming | Two-phase commit | Disable on error | Origin | Password required | Run as owner? | Failover | Server | Retain dead tuples | Max retention duration | Retention active | Synchronous commit | Conninfo | Receiver timeout | Skip LSN | Description ------------------+---------------------------+---------+-------------+--------+-----------+------------------+------------------+--------+-------------------+---------------+----------+--------+--------------------+------------------------+------------------+--------------------+-----------------------------+------------------+------------+------------- - regress_testsub | regress_subscription_user | f | {testpub} | f | off | d | f | any | t | f | f | | f | 0 | f | off | dbname=regress_doesnotexist | -1 | 0/00000000 | + List of subscriptions + Name | Owner | Enabled | Publication | Binary | Streaming | Two-phase commit | Disable on error | Origin | Password required | Run as owner? | Failover | Server | Retain dead tuples | Max retention duration | Retention active | Messages | Synchronous commit | Conninfo | Receiver timeout | Skip LSN | Description +-----------------+---------------------------+---------+-------------+--------+-----------+------------------+------------------+--------+-------------------+---------------+----------+--------+--------------------+------------------------+------------------+----------+--------------------+-----------------------------+------------------+------------+------------- + regress_testsub | regress_subscription_user | f | {testpub} | f | off | d | f | any | t | f | f | | f | 0 | f | f | off | dbname=regress_doesnotexist | -1 | 0/00000000 | (1 row) -- fail - publication already exists @@ -450,10 +450,10 @@ ALTER SUBSCRIPTION regress_testsub ADD PUBLICATION testpub1, testpub2 WITH (refr ALTER SUBSCRIPTION regress_testsub ADD PUBLICATION testpub1, testpub2 WITH (refresh = false); ERROR: publication "testpub1" is already in subscription "regress_testsub" \dRs+ - List of subscriptions - Name | Owner | Enabled | Publication | Binary | Streaming | Two-phase commit | Disable on error | Origin | Password required | Run as owner? | Failover | Server | Retain dead tuples | Max retention duration | Retention active | Synchronous commit | Conninfo | Receiver timeout | Skip LSN | Description ------------------+---------------------------+---------+-----------------------------+--------+-----------+------------------+------------------+--------+-------------------+---------------+----------+--------+--------------------+------------------------+------------------+--------------------+-----------------------------+------------------+------------+------------- - regress_testsub | regress_subscription_user | f | {testpub,testpub1,testpub2} | f | off | d | f | any | t | f | f | | f | 0 | f | off | dbname=regress_doesnotexist | -1 | 0/00000000 | + List of subscriptions + Name | Owner | Enabled | Publication | Binary | Streaming | Two-phase commit | Disable on error | Origin | Password required | Run as owner? | Failover | Server | Retain dead tuples | Max retention duration | Retention active | Messages | Synchronous commit | Conninfo | Receiver timeout | Skip LSN | Description +-----------------+---------------------------+---------+-----------------------------+--------+-----------+------------------+------------------+--------+-------------------+---------------+----------+--------+--------------------+------------------------+------------------+----------+--------------------+-----------------------------+------------------+------------+------------- + regress_testsub | regress_subscription_user | f | {testpub,testpub1,testpub2} | f | off | d | f | any | t | f | f | | f | 0 | f | f | off | dbname=regress_doesnotexist | -1 | 0/00000000 | (1 row) -- fail - publication used more than once @@ -468,10 +468,10 @@ ERROR: publication "testpub3" is not in subscription "regress_testsub" -- ok - delete publications ALTER SUBSCRIPTION regress_testsub DROP PUBLICATION testpub1, testpub2 WITH (refresh = false); \dRs+ - List of subscriptions - Name | Owner | Enabled | Publication | Binary | Streaming | Two-phase commit | Disable on error | Origin | Password required | Run as owner? | Failover | Server | Retain dead tuples | Max retention duration | Retention active | Synchronous commit | Conninfo | Receiver timeout | Skip LSN | Description ------------------+---------------------------+---------+-------------+--------+-----------+------------------+------------------+--------+-------------------+---------------+----------+--------+--------------------+------------------------+------------------+--------------------+-----------------------------+------------------+------------+------------- - regress_testsub | regress_subscription_user | f | {testpub} | f | off | d | f | any | t | f | f | | f | 0 | f | off | dbname=regress_doesnotexist | -1 | 0/00000000 | + List of subscriptions + Name | Owner | Enabled | Publication | Binary | Streaming | Two-phase commit | Disable on error | Origin | Password required | Run as owner? | Failover | Server | Retain dead tuples | Max retention duration | Retention active | Messages | Synchronous commit | Conninfo | Receiver timeout | Skip LSN | Description +-----------------+---------------------------+---------+-------------+--------+-----------+------------------+------------------+--------+-------------------+---------------+----------+--------+--------------------+------------------------+------------------+----------+--------------------+-----------------------------+------------------+------------+------------- + regress_testsub | regress_subscription_user | f | {testpub} | f | off | d | f | any | t | f | f | | f | 0 | f | f | off | dbname=regress_doesnotexist | -1 | 0/00000000 | (1 row) DROP SUBSCRIPTION regress_testsub; @@ -507,19 +507,19 @@ CREATE SUBSCRIPTION regress_testsub CONNECTION 'dbname=regress_doesnotexist' PUB WARNING: subscription was created, but is not connected HINT: To initiate replication, you must manually create the replication slot, enable the subscription, and alter the subscription to refresh publications. \dRs+ - List of subscriptions - Name | Owner | Enabled | Publication | Binary | Streaming | Two-phase commit | Disable on error | Origin | Password required | Run as owner? | Failover | Server | Retain dead tuples | Max retention duration | Retention active | Synchronous commit | Conninfo | Receiver timeout | Skip LSN | Description ------------------+---------------------------+---------+-------------+--------+-----------+------------------+------------------+--------+-------------------+---------------+----------+--------+--------------------+------------------------+------------------+--------------------+-----------------------------+------------------+------------+------------- - regress_testsub | regress_subscription_user | f | {testpub} | f | parallel | p | f | any | t | f | f | | f | 0 | f | off | dbname=regress_doesnotexist | -1 | 0/00000000 | + List of subscriptions + Name | Owner | Enabled | Publication | Binary | Streaming | Two-phase commit | Disable on error | Origin | Password required | Run as owner? | Failover | Server | Retain dead tuples | Max retention duration | Retention active | Messages | Synchronous commit | Conninfo | Receiver timeout | Skip LSN | Description +-----------------+---------------------------+---------+-------------+--------+-----------+------------------+------------------+--------+-------------------+---------------+----------+--------+--------------------+------------------------+------------------+----------+--------------------+-----------------------------+------------------+------------+------------- + regress_testsub | regress_subscription_user | f | {testpub} | f | parallel | p | f | any | t | f | f | | f | 0 | f | f | off | dbname=regress_doesnotexist | -1 | 0/00000000 | (1 row) -- we can alter streaming when two_phase enabled ALTER SUBSCRIPTION regress_testsub SET (streaming = true); \dRs+ - List of subscriptions - Name | Owner | Enabled | Publication | Binary | Streaming | Two-phase commit | Disable on error | Origin | Password required | Run as owner? | Failover | Server | Retain dead tuples | Max retention duration | Retention active | Synchronous commit | Conninfo | Receiver timeout | Skip LSN | Description ------------------+---------------------------+---------+-------------+--------+-----------+------------------+------------------+--------+-------------------+---------------+----------+--------+--------------------+------------------------+------------------+--------------------+-----------------------------+------------------+------------+------------- - regress_testsub | regress_subscription_user | f | {testpub} | f | on | p | f | any | t | f | f | | f | 0 | f | off | dbname=regress_doesnotexist | -1 | 0/00000000 | + List of subscriptions + Name | Owner | Enabled | Publication | Binary | Streaming | Two-phase commit | Disable on error | Origin | Password required | Run as owner? | Failover | Server | Retain dead tuples | Max retention duration | Retention active | Messages | Synchronous commit | Conninfo | Receiver timeout | Skip LSN | Description +-----------------+---------------------------+---------+-------------+--------+-----------+------------------+------------------+--------+-------------------+---------------+----------+--------+--------------------+------------------------+------------------+----------+--------------------+-----------------------------+------------------+------------+------------- + regress_testsub | regress_subscription_user | f | {testpub} | f | on | p | f | any | t | f | f | | f | 0 | f | f | off | dbname=regress_doesnotexist | -1 | 0/00000000 | (1 row) ALTER SUBSCRIPTION regress_testsub SET (slot_name = NONE); @@ -529,10 +529,10 @@ CREATE SUBSCRIPTION regress_testsub CONNECTION 'dbname=regress_doesnotexist' PUB WARNING: subscription was created, but is not connected HINT: To initiate replication, you must manually create the replication slot, enable the subscription, and alter the subscription to refresh publications. \dRs+ - List of subscriptions - Name | Owner | Enabled | Publication | Binary | Streaming | Two-phase commit | Disable on error | Origin | Password required | Run as owner? | Failover | Server | Retain dead tuples | Max retention duration | Retention active | Synchronous commit | Conninfo | Receiver timeout | Skip LSN | Description ------------------+---------------------------+---------+-------------+--------+-----------+------------------+------------------+--------+-------------------+---------------+----------+--------+--------------------+------------------------+------------------+--------------------+-----------------------------+------------------+------------+------------- - regress_testsub | regress_subscription_user | f | {testpub} | f | on | p | f | any | t | f | f | | f | 0 | f | off | dbname=regress_doesnotexist | -1 | 0/00000000 | + List of subscriptions + Name | Owner | Enabled | Publication | Binary | Streaming | Two-phase commit | Disable on error | Origin | Password required | Run as owner? | Failover | Server | Retain dead tuples | Max retention duration | Retention active | Messages | Synchronous commit | Conninfo | Receiver timeout | Skip LSN | Description +-----------------+---------------------------+---------+-------------+--------+-----------+------------------+------------------+--------+-------------------+---------------+----------+--------+--------------------+------------------------+------------------+----------+--------------------+-----------------------------+------------------+------------+------------- + regress_testsub | regress_subscription_user | f | {testpub} | f | on | p | f | any | t | f | f | | f | 0 | f | f | off | dbname=regress_doesnotexist | -1 | 0/00000000 | (1 row) ALTER SUBSCRIPTION regress_testsub SET (slot_name = NONE); @@ -545,18 +545,18 @@ CREATE SUBSCRIPTION regress_testsub CONNECTION 'dbname=regress_doesnotexist' PUB WARNING: subscription was created, but is not connected HINT: To initiate replication, you must manually create the replication slot, enable the subscription, and alter the subscription to refresh publications. \dRs+ - List of subscriptions - Name | Owner | Enabled | Publication | Binary | Streaming | Two-phase commit | Disable on error | Origin | Password required | Run as owner? | Failover | Server | Retain dead tuples | Max retention duration | Retention active | Synchronous commit | Conninfo | Receiver timeout | Skip LSN | Description ------------------+---------------------------+---------+-------------+--------+-----------+------------------+------------------+--------+-------------------+---------------+----------+--------+--------------------+------------------------+------------------+--------------------+-----------------------------+------------------+------------+------------- - regress_testsub | regress_subscription_user | f | {testpub} | f | parallel | d | f | any | t | f | f | | f | 0 | f | off | dbname=regress_doesnotexist | -1 | 0/00000000 | + List of subscriptions + Name | Owner | Enabled | Publication | Binary | Streaming | Two-phase commit | Disable on error | Origin | Password required | Run as owner? | Failover | Server | Retain dead tuples | Max retention duration | Retention active | Messages | Synchronous commit | Conninfo | Receiver timeout | Skip LSN | Description +-----------------+---------------------------+---------+-------------+--------+-----------+------------------+------------------+--------+-------------------+---------------+----------+--------+--------------------+------------------------+------------------+----------+--------------------+-----------------------------+------------------+------------+------------- + regress_testsub | regress_subscription_user | f | {testpub} | f | parallel | d | f | any | t | f | f | | f | 0 | f | f | off | dbname=regress_doesnotexist | -1 | 0/00000000 | (1 row) ALTER SUBSCRIPTION regress_testsub SET (disable_on_error = true); \dRs+ - List of subscriptions - Name | Owner | Enabled | Publication | Binary | Streaming | Two-phase commit | Disable on error | Origin | Password required | Run as owner? | Failover | Server | Retain dead tuples | Max retention duration | Retention active | Synchronous commit | Conninfo | Receiver timeout | Skip LSN | Description ------------------+---------------------------+---------+-------------+--------+-----------+------------------+------------------+--------+-------------------+---------------+----------+--------+--------------------+------------------------+------------------+--------------------+-----------------------------+------------------+------------+------------- - regress_testsub | regress_subscription_user | f | {testpub} | f | parallel | d | t | any | t | f | f | | f | 0 | f | off | dbname=regress_doesnotexist | -1 | 0/00000000 | + List of subscriptions + Name | Owner | Enabled | Publication | Binary | Streaming | Two-phase commit | Disable on error | Origin | Password required | Run as owner? | Failover | Server | Retain dead tuples | Max retention duration | Retention active | Messages | Synchronous commit | Conninfo | Receiver timeout | Skip LSN | Description +-----------------+---------------------------+---------+-------------+--------+-----------+------------------+------------------+--------+-------------------+---------------+----------+--------+--------------------+------------------------+------------------+----------+--------------------+-----------------------------+------------------+------------+------------- + regress_testsub | regress_subscription_user | f | {testpub} | f | parallel | d | t | any | t | f | f | | f | 0 | f | f | off | dbname=regress_doesnotexist | -1 | 0/00000000 | (1 row) ALTER SUBSCRIPTION regress_testsub SET (slot_name = NONE); @@ -569,10 +569,10 @@ CREATE SUBSCRIPTION regress_testsub CONNECTION 'dbname=regress_doesnotexist' PUB WARNING: subscription was created, but is not connected HINT: To initiate replication, you must manually create the replication slot, enable the subscription, and alter the subscription to refresh publications. \dRs+ - List of subscriptions - Name | Owner | Enabled | Publication | Binary | Streaming | Two-phase commit | Disable on error | Origin | Password required | Run as owner? | Failover | Server | Retain dead tuples | Max retention duration | Retention active | Synchronous commit | Conninfo | Receiver timeout | Skip LSN | Description ------------------+---------------------------+---------+-------------+--------+-----------+------------------+------------------+--------+-------------------+---------------+----------+--------+--------------------+------------------------+------------------+--------------------+-----------------------------+------------------+------------+------------- - regress_testsub | regress_subscription_user | f | {testpub} | f | parallel | d | f | any | t | f | f | | f | 0 | f | off | dbname=regress_doesnotexist | -1 | 0/00000000 | + List of subscriptions + Name | Owner | Enabled | Publication | Binary | Streaming | Two-phase commit | Disable on error | Origin | Password required | Run as owner? | Failover | Server | Retain dead tuples | Max retention duration | Retention active | Messages | Synchronous commit | Conninfo | Receiver timeout | Skip LSN | Description +-----------------+---------------------------+---------+-------------+--------+-----------+------------------+------------------+--------+-------------------+---------------+----------+--------+--------------------+------------------------+------------------+----------+--------------------+-----------------------------+------------------+------------+------------- + regress_testsub | regress_subscription_user | f | {testpub} | f | parallel | d | f | any | t | f | f | | f | 0 | f | f | off | dbname=regress_doesnotexist | -1 | 0/00000000 | (1 row) ALTER SUBSCRIPTION regress_testsub SET (slot_name = NONE); @@ -589,10 +589,10 @@ NOTICE: max_retention_duration is ineffective when retain_dead_tuples is disabl WARNING: subscription was created, but is not connected HINT: To initiate replication, you must manually create the replication slot, enable the subscription, and alter the subscription to refresh publications. \dRs+ - List of subscriptions - Name | Owner | Enabled | Publication | Binary | Streaming | Two-phase commit | Disable on error | Origin | Password required | Run as owner? | Failover | Server | Retain dead tuples | Max retention duration | Retention active | Synchronous commit | Conninfo | Receiver timeout | Skip LSN | Description ------------------+---------------------------+---------+-------------+--------+-----------+------------------+------------------+--------+-------------------+---------------+----------+--------+--------------------+------------------------+------------------+--------------------+-----------------------------+------------------+------------+------------- - regress_testsub | regress_subscription_user | f | {testpub} | f | parallel | d | f | any | t | f | f | | f | 1000 | f | off | dbname=regress_doesnotexist | -1 | 0/00000000 | + List of subscriptions + Name | Owner | Enabled | Publication | Binary | Streaming | Two-phase commit | Disable on error | Origin | Password required | Run as owner? | Failover | Server | Retain dead tuples | Max retention duration | Retention active | Messages | Synchronous commit | Conninfo | Receiver timeout | Skip LSN | Description +-----------------+---------------------------+---------+-------------+--------+-----------+------------------+------------------+--------+-------------------+---------------+----------+--------+--------------------+------------------------+------------------+----------+--------------------+-----------------------------+------------------+------------+------------- + regress_testsub | regress_subscription_user | f | {testpub} | f | parallel | d | f | any | t | f | f | | f | 1000 | f | f | off | dbname=regress_doesnotexist | -1 | 0/00000000 | (1 row) -- fail - max_retention_duration must be non-negative @@ -601,12 +601,38 @@ ERROR: option "max_retention_duration" cannot be negative -- ok ALTER SUBSCRIPTION regress_testsub SET (max_retention_duration = 0); \dRs+ - List of subscriptions - Name | Owner | Enabled | Publication | Binary | Streaming | Two-phase commit | Disable on error | Origin | Password required | Run as owner? | Failover | Server | Retain dead tuples | Max retention duration | Retention active | Synchronous commit | Conninfo | Receiver timeout | Skip LSN | Description ------------------+---------------------------+---------+-------------+--------+-----------+------------------+------------------+--------+-------------------+---------------+----------+--------+--------------------+------------------------+------------------+--------------------+-----------------------------+------------------+------------+------------- - regress_testsub | regress_subscription_user | f | {testpub} | f | parallel | d | f | any | t | f | f | | f | 0 | f | off | dbname=regress_doesnotexist | -1 | 0/00000000 | + List of subscriptions + Name | Owner | Enabled | Publication | Binary | Streaming | Two-phase commit | Disable on error | Origin | Password required | Run as owner? | Failover | Server | Retain dead tuples | Max retention duration | Retention active | Messages | Synchronous commit | Conninfo | Receiver timeout | Skip LSN | Description +-----------------+---------------------------+---------+-------------+--------+-----------+------------------+------------------+--------+-------------------+---------------+----------+--------+--------------------+------------------------+------------------+----------+--------------------+-----------------------------+------------------+------------+------------- + regress_testsub | regress_subscription_user | f | {testpub} | f | parallel | d | f | any | t | f | f | | f | 0 | f | f | off | dbname=regress_doesnotexist | -1 | 0/00000000 | +(1 row) + +ALTER SUBSCRIPTION regress_testsub SET (slot_name = NONE); +DROP SUBSCRIPTION regress_testsub; +-- fail - messages must be boolean +CREATE SUBSCRIPTION regress_testsub CONNECTION 'dbname=regress_doesnotexist' PUBLICATION testpub WITH (connect = false, messages = foo); +ERROR: messages requires a Boolean value +-- ok +CREATE SUBSCRIPTION regress_testsub CONNECTION 'dbname=regress_doesnotexist' PUBLICATION testpub WITH (connect = false, messages = true); +WARNING: subscription was created, but is not connected +HINT: To initiate replication, you must manually create the replication slot, enable the subscription, and alter the subscription to refresh publications. +\dRs+ + List of subscriptions + Name | Owner | Enabled | Publication | Binary | Streaming | Two-phase commit | Disable on error | Origin | Password required | Run as owner? | Failover | Server | Retain dead tuples | Max retention duration | Retention active | Messages | Synchronous commit | Conninfo | Receiver timeout | Skip LSN | Description +-----------------+---------------------------+---------+-------------+--------+-----------+------------------+------------------+--------+-------------------+---------------+----------+--------+--------------------+------------------------+------------------+----------+--------------------+-----------------------------+------------------+------------+------------- + regress_testsub | regress_subscription_user | f | {testpub} | f | parallel | d | f | any | t | f | f | | f | 0 | f | t | off | dbname=regress_doesnotexist | -1 | 0/00000000 | +(1 row) + +-- ok +ALTER SUBSCRIPTION regress_testsub SET (messages = false); +\dRs+ + List of subscriptions + Name | Owner | Enabled | Publication | Binary | Streaming | Two-phase commit | Disable on error | Origin | Password required | Run as owner? | Failover | Server | Retain dead tuples | Max retention duration | Retention active | Messages | Synchronous commit | Conninfo | Receiver timeout | Skip LSN | Description +-----------------+---------------------------+---------+-------------+--------+-----------+------------------+------------------+--------+-------------------+---------------+----------+--------+--------------------+------------------------+------------------+----------+--------------------+-----------------------------+------------------+------------+------------- + regress_testsub | regress_subscription_user | f | {testpub} | f | parallel | d | f | any | t | f | f | | f | 0 | f | f | off | dbname=regress_doesnotexist | -1 | 0/00000000 | (1 row) +-- cleanup ALTER SUBSCRIPTION regress_testsub SET (slot_name = NONE); DROP SUBSCRIPTION regress_testsub; -- let's do some tests with pg_create_subscription rather than superuser diff --git a/src/test/regress/sql/subscription.sql b/src/test/regress/sql/subscription.sql index 1f34889a9a5..823b9318ead 100644 --- a/src/test/regress/sql/subscription.sql +++ b/src/test/regress/sql/subscription.sql @@ -457,6 +457,23 @@ ALTER SUBSCRIPTION regress_testsub SET (max_retention_duration = 0); ALTER SUBSCRIPTION regress_testsub SET (slot_name = NONE); DROP SUBSCRIPTION regress_testsub; +-- fail - messages must be boolean +CREATE SUBSCRIPTION regress_testsub CONNECTION 'dbname=regress_doesnotexist' PUBLICATION testpub WITH (connect = false, messages = foo); + +-- ok +CREATE SUBSCRIPTION regress_testsub CONNECTION 'dbname=regress_doesnotexist' PUBLICATION testpub WITH (connect = false, messages = true); + +\dRs+ + +-- ok +ALTER SUBSCRIPTION regress_testsub SET (messages = false); + +\dRs+ + +-- cleanup +ALTER SUBSCRIPTION regress_testsub SET (slot_name = NONE); +DROP SUBSCRIPTION regress_testsub; + -- let's do some tests with pg_create_subscription rather than superuser SET SESSION AUTHORIZATION regress_subscription_user3; diff --git a/src/tools/pgindent/typedefs.list b/src/tools/pgindent/typedefs.list index f10eb469266..f459fe0dfc4 100644 --- a/src/tools/pgindent/typedefs.list +++ b/src/tools/pgindent/typedefs.list @@ -1685,6 +1685,8 @@ LogicalRepBeginData LogicalRepCommitData LogicalRepCommitPreparedTxnData LogicalRepCtxStruct +LogicalRepMessageData +LogicalRepMessageHandle_hook_type LogicalRepMsgType LogicalRepPartMapEntry LogicalRepPreparedTxnData -- 2.56.0
^ permalink raw reply [nested|flat] 44+ messages in thread

* Re: Add a hook for handling logical decoding messages on subscribers.
@ 2026-11-17 08:14  Bertrand Drouvot <bertranddrouvot.pg@gmail.com>
  parent: Masahiko Sawada <sawada.mshk@gmail.com>
  0 siblings, 1 reply; 44+ messages in thread

From: Bertrand Drouvot @ 2026-11-17 08:14 UTC (permalink / raw)
  To: Masahiko Sawada <sawada.mshk@gmail.com>; +Cc: Zhijie Hou <houzhijie22@gmail.com>; Amit Kapila <amit.kapila16@gmail.com>; Bharath Rupireddy <bharath.rupireddyforpostgres@gmail.com>; Chao Li <li.evan.chao@gmail.com>; Fujii Masao <masao.fujii@gmail.com>; pgsql-hackers

Hi,

On Mon, Nov 16, 2026 at 11:05:00AM -0800, Masahiko Sawada wrote:
> 
> I've attached the updated patches.

Thanks!

I did not look in details but I have a question. I'm wondering what would happen
with an unpatched publisher when both messages and streaming are enabled. Suppose
a message is emitted in a subtransaction that is later aborted. The publisher
sends the top level XID with the message, whereas STREAM ABORT refers to the
subtransaction XID. IIUC, the patched subscriber would therefore not discard that
message.

Should 0002 handle this case somehow?

Regards,

-- 
Bertrand Drouvot
PostgreSQL Contributors Team
RDS Open Source Databases
Amazon Web Services: https://aws.amazon.com






^ permalink  raw  reply  [nested|flat] 44+ messages in thread

* Re: Add a hook for handling logical decoding messages on subscribers.
@ 2026-11-17 22:14  Masahiko Sawada <sawada.mshk@gmail.com>
  parent: Bertrand Drouvot <bertranddrouvot.pg@gmail.com>
  0 siblings, 1 reply; 44+ messages in thread

From: Masahiko Sawada @ 2026-11-17 22:14 UTC (permalink / raw)
  To: Bertrand Drouvot <bertranddrouvot.pg@gmail.com>; +Cc: Zhijie Hou <houzhijie22@gmail.com>; Amit Kapila <amit.kapila16@gmail.com>; Bharath Rupireddy <bharath.rupireddyforpostgres@gmail.com>; Chao Li <li.evan.chao@gmail.com>; Fujii Masao <masao.fujii@gmail.com>; pgsql-hackers

On Tue, Nov 17, 2026 at 12:14 AM Bertrand Drouvot
<bertranddrouvot.pg@gmail.com> wrote:
>
> Hi,
>
> On Mon, Nov 16, 2026 at 11:05:00AM -0800, Masahiko Sawada wrote:
> >
> > I've attached the updated patches.
>
> Thanks!
>
> I did not look in details but I have a question. I'm wondering what would happen
> with an unpatched publisher when both messages and streaming are enabled. Suppose
> a message is emitted in a subtransaction that is later aborted. The publisher
> sends the top level XID with the message, whereas STREAM ABORT refers to the
> subtransaction XID. IIUC, the patched subscriber would therefore not discard that
> message.
>
> Should 0002 handle this case somehow?

Good point. In that case, the subscriber would apply the message when
the top-level transaction commits, even though the subtransaction that
emitted it was rolled back.

Given that the 0001 patch is not backpatched, I think the subscriber
should request the messages option only from v20 or later publishers,
like it doesn't send the origin option to publishers older than v16.
The patch already checks the publisher version there, so we just need
to raise it from v14 to v20. What do you think?

Regards,

-- 
Masahiko Sawada
Amazon Web Services: https://aws.amazon.com






^ permalink  raw  reply  [nested|flat] 44+ messages in thread

* Re: Add a hook for handling logical decoding messages on subscribers.
@ 2026-11-18 06:43  Bertrand Drouvot <bertranddrouvot.pg@gmail.com>
  parent: Masahiko Sawada <sawada.mshk@gmail.com>
  0 siblings, 0 replies; 44+ messages in thread

From: Bertrand Drouvot @ 2026-11-18 06:43 UTC (permalink / raw)
  To: Masahiko Sawada <sawada.mshk@gmail.com>; +Cc: Zhijie Hou <houzhijie22@gmail.com>; Amit Kapila <amit.kapila16@gmail.com>; Bharath Rupireddy <bharath.rupireddyforpostgres@gmail.com>; Chao Li <li.evan.chao@gmail.com>; Fujii Masao <masao.fujii@gmail.com>; pgsql-hackers

Hi,

On Tue, Nov 17, 2026 at 02:14:15PM -0800, Masahiko Sawada wrote:
> On Tue, Nov 17, 2026 at 12:14 AM Bertrand Drouvot
> <bertranddrouvot.pg@gmail.com> wrote:
> >
> > Hi,
> >
> > On Mon, Nov 16, 2026 at 11:05:00AM -0800, Masahiko Sawada wrote:
> > >
> > > I've attached the updated patches.
> >
> > Thanks!
> >
> > I did not look in details but I have a question. I'm wondering what would happen
> > with an unpatched publisher when both messages and streaming are enabled. Suppose
> > a message is emitted in a subtransaction that is later aborted. The publisher
> > sends the top level XID with the message, whereas STREAM ABORT refers to the
> > subtransaction XID. IIUC, the patched subscriber would therefore not discard that
> > message.
> >
> > Should 0002 handle this case somehow?
> 
> Good point. In that case, the subscriber would apply the message when
> the top-level transaction commits, even though the subtransaction that
> emitted it was rolled back.

Yeah, thanks for confirming.

> Given that the 0001 patch is not backpatched, I think the subscriber
> should request the messages option only from v20 or later publishers,
> like it doesn't send the origin option to publishers older than v16.
> The patch already checks the publisher version there, so we just need
> to raise it from v14 to v20. What do you think?

One difference with origin is that older publishers do not support that option
at all, whereas v14 through v19 publishers support messages and the issue occurs
only when streaming is enabled.

I wonder if it would make sense to request messages from those publishers when
streaming is disabled, and require v20 or later when streaming is enabled?

Regards,

-- 
Bertrand Drouvot
PostgreSQL Contributors Team
RDS Open Source Databases
Amazon Web Services: https://aws.amazon.com






^ permalink  raw  reply  [nested|flat] 44+ messages in thread


end of thread, other threads:[~2026-11-18 06:43 UTC | newest]

Thread overview: 44+ messages (download: mbox mbox.gz follow: Atom feed)
-- links below jump to the message on this page --
2026-06-19 22:33 Add a hook for handling logical decoding messages on subscribers. Masahiko Sawada <sawada.mshk@gmail.com>
2026-06-22 22:39 ` Bharath Rupireddy <bharath.rupireddyforpostgres@gmail.com>
2026-06-23 08:52   ` Amit Kapila <amit.kapila16@gmail.com>
2026-06-23 18:01     ` Masahiko Sawada <sawada.mshk@gmail.com>
2026-06-24 00:21       ` Masahiko Sawada <sawada.mshk@gmail.com>
2026-07-09 12:14       ` Fujii Masao <masao.fujii@gmail.com>
2026-08-04 00:41         ` Masahiko Sawada <sawada.mshk@gmail.com>
2026-08-04 03:03           ` Fujii Masao <masao.fujii@gmail.com>
2026-08-04 19:06             ` Masahiko Sawada <sawada.mshk@gmail.com>
2026-08-04 03:04           ` Bharath Rupireddy <bharath.rupireddyforpostgres@gmail.com>
2026-08-04 20:25             ` Masahiko Sawada <sawada.mshk@gmail.com>
2026-08-04 22:25               ` Bharath Rupireddy <bharath.rupireddyforpostgres@gmail.com>
2026-08-04 08:05           ` Chao Li <li.evan.chao@gmail.com>
2026-08-05 16:32             ` Masahiko Sawada <sawada.mshk@gmail.com>
2026-08-05 18:55               ` Masahiko Sawada <sawada.mshk@gmail.com>
2026-08-06 04:44                 ` Chao Li <li.evan.chao@gmail.com>
2026-08-06 07:03                 ` Bharath Rupireddy <bharath.rupireddyforpostgres@gmail.com>
2026-08-06 16:58                   ` Masahiko Sawada <sawada.mshk@gmail.com>
2026-08-06 17:41                     ` Bharath Rupireddy <bharath.rupireddyforpostgres@gmail.com>
2026-09-18 00:42                       ` Masahiko Sawada <sawada.mshk@gmail.com>
2026-09-20 03:52                         ` Bharath Rupireddy <bharath.rupireddyforpostgres@gmail.com>
2026-09-21 06:23                         ` Amit Kapila <amit.kapila16@gmail.com>
2026-09-21 06:54                           ` Amit Kapila <amit.kapila16@gmail.com>
2026-09-21 16:20                           ` Bharath Rupireddy <bharath.rupireddyforpostgres@gmail.com>
2026-10-16 21:44                             ` Masahiko Sawada <sawada.mshk@gmail.com>
2026-10-16 22:16                               ` Bharath Rupireddy <bharath.rupireddyforpostgres@gmail.com>
2026-10-30 22:34                                 ` Masahiko Sawada <sawada.mshk@gmail.com>
2026-10-31 12:35                                   ` Zhijie Hou <houzhijie22@gmail.com>
2026-11-02 12:47                                     ` Amit Kapila <amit.kapila16@gmail.com>
2026-11-09 19:02                                       ` Masahiko Sawada <sawada.mshk@gmail.com>
2026-11-10 01:21                                         ` Masahiko Sawada <sawada.mshk@gmail.com>
2026-11-11 22:40                                           ` Masahiko Sawada <sawada.mshk@gmail.com>
2026-11-12 09:32                                             ` Amit Kapila <amit.kapila16@gmail.com>
2026-11-12 10:06                                               ` Bertrand Drouvot <bertranddrouvot.pg@gmail.com>
2026-11-13 23:35                                                 ` Masahiko Sawada <sawada.mshk@gmail.com>
2026-11-15 17:13                                                   ` Zhijie Hou <houzhijie22@gmail.com>
2026-11-16 19:05                                                     ` Masahiko Sawada <sawada.mshk@gmail.com>
2026-11-17 08:14                                                       ` Bertrand Drouvot <bertranddrouvot.pg@gmail.com>
2026-11-17 22:14                                                         ` Masahiko Sawada <sawada.mshk@gmail.com>
2026-11-18 06:43                                                           ` Bertrand Drouvot <bertranddrouvot.pg@gmail.com>
2026-11-02 15:06                                   ` torikoshia <torikoshia@oss.nttdata.com>
2026-11-09 23:59                                     ` Masahiko Sawada <sawada.mshk@gmail.com>
2026-10-16 20:49                           ` Masahiko Sawada <sawada.mshk@gmail.com>
2026-06-23 18:48   ` Masahiko Sawada <sawada.mshk@gmail.com>

This inbox is served by DDX for PostgreSQL; see mirroring instructions
for how to clone and mirror all data and code used for this inbox