Subject: row filtering for logical replication

From: Euler Taveira <[email-masked]>
Date: Wed, 28 Feb 2018 20:03:02 -0300

Hi,

The attached patches add support for filtering rows in the publisher.
The output plugin will do the work if a filter was defined in CREATE
PUBLICATION command. An optional WHERE clause can be added after the
table name in the CREATE PUBLICATION such as:

CREATE PUBLICATION foo FOR TABLE departments WHERE (id > 2000 AND id <= 3000);

Row that doesn't match the WHERE clause will not be sent to the subscribers.

Patches 0001 and 0002 are only refactors and can be applied
independently. 0003 doesn't include row filtering on initial
synchronization.

Comments?

-- 
   Euler Taveira                                   Timbira -
http://www.timbira.com.br/
   PostgreSQL: Consultoria, Desenvolvimento, Suporte 24x7 e Treinamento
From ae95eb51dd72c2e8ba278da950b478f6c6741fc0 Mon Sep 17 00:00:00 2001
From: Euler Taveira <[email-masked]>
Date: Tue, 27 Feb 2018 02:21:03 +0000
Subject: [PATCH 1/3] Refactor function create_estate_for_relation

Relation localrel is the only LogicalRepRelMapEntry structure member
that is useful for create_estate_for_relation.
---
 src/backend/replication/logical/worker.c | 14 +++++++-------
 1 file changed, 7 insertions(+), 7 deletions(-)

diff --git a/src/backend/replication/logical/worker.c b/src/backend/replication/logical/worker.c
index 04985c9..6820c1a 100644
--- a/src/backend/replication/logical/worker.c
+++ b/src/backend/replication/logical/worker.c
@@ -183,7 +183,7 @@ ensure_transaction(void)
  * This is based on similar code in copy.c
  */
 static EState *
-create_estate_for_relation(LogicalRepRelMapEntry *rel)
+create_estate_for_relation(Relation rel)
 {
 	EState	   *estate;
 	ResultRelInfo *resultRelInfo;
@@ -193,12 +193,12 @@ create_estate_for_relation(LogicalRepRelMapEntry *rel)
 
 	rte = makeNode(RangeTblEntry);
 	rte->rtekind = RTE_RELATION;
-	rte->relid = RelationGetRelid(rel->localrel);
-	rte->relkind = rel->localrel->rd_rel->relkind;
+	rte->relid = RelationGetRelid(rel);
+	rte->relkind = rel->rd_rel->relkind;
 	estate->es_range_table = list_make1(rte);
 
 	resultRelInfo = makeNode(ResultRelInfo);
-	InitResultRelInfo(resultRelInfo, rel->localrel, 1, NULL, 0);
+	InitResultRelInfo(resultRelInfo, rel, 1, NULL, 0);
 
 	estate->es_result_relations = resultRelInfo;
 	estate->es_num_result_relations = 1;
@@ -584,7 +584,7 @@ apply_handle_insert(StringInfo s)
 	}
 
 	/* Initialize the executor state. */
-	estate = create_estate_for_relation(rel);
+	estate = create_estate_for_relation(rel->localrel);
 	remoteslot = ExecInitExtraTupleSlot(estate,
 										RelationGetDescr(rel->localrel));
 
@@ -688,7 +688,7 @@ apply_handle_update(StringInfo s)
 	check_relation_updatable(rel);
 
 	/* Initialize the executor state. */
-	estate = create_estate_for_relation(rel);
+	estate = create_estate_for_relation(rel->localrel);
 	remoteslot = ExecInitExtraTupleSlot(estate,
 										RelationGetDescr(rel->localrel));
 	localslot = ExecInitExtraTupleSlot(estate,
@@ -806,7 +806,7 @@ apply_handle_delete(StringInfo s)
 	check_relation_updatable(rel);
 
 	/* Initialize the executor state. */
-	estate = create_estate_for_relation(rel);
+	estate = create_estate_for_relation(rel->localrel);
 	remoteslot = ExecInitExtraTupleSlot(estate,
 										RelationGetDescr(rel->localrel));
 	localslot = ExecInitExtraTupleSlot(estate,
-- 
2.7.4

From 8421809e38d3e1d43b00feaac4b8bccaa6738079 Mon Sep 17 00:00:00 2001
From: Euler Taveira <[email-masked]>
Date: Wed, 24 Jan 2018 17:01:31 -0200
Subject: [PATCH 2/3] Rename a WHERE node

A WHERE clause will be used for row filtering in logical replication. We
already have a similar node: 'WHERE (condition here)'. Let's rename the
node to a generic name and use it for row filtering too.
---
 src/backend/parser/gram.y | 6 +++---
 1 file changed, 3 insertions(+), 3 deletions(-)

diff --git a/src/backend/parser/gram.y b/src/backend/parser/gram.y
index d99f2be..bf32362 100644
--- a/src/backend/parser/gram.y
+++ b/src/backend/parser/gram.y
@@ -468,7 +468,7 @@ static Node *makeRecursiveViewSelect(char *relname, List *aliases, Node *query);
 %type <node>	def_arg columnElem where_clause where_or_current_clause
 				a_expr b_expr c_expr AexprConst indirection_el opt_slice_bound
 				columnref in_expr having_clause func_table xmltable array_expr
-				ExclusionWhereClause operator_def_arg
+				OptWhereClause operator_def_arg
 %type <list>	rowsfrom_item rowsfrom_list opt_col_def_list
 %type <boolean> opt_ordinality
 %type <list>	ExclusionConstraintList ExclusionConstraintElem
@@ -3734,7 +3734,7 @@ ConstraintElem:
 					$$ = (Node *)n;
 				}
 			| EXCLUDE access_method_clause '(' ExclusionConstraintList ')'
-				opt_definition OptConsTableSpace ExclusionWhereClause
+				opt_definition OptConsTableSpace OptWhereClause
 				ConstraintAttributeSpec
 				{
 					Constraint *n = makeNode(Constraint);
@@ -3831,7 +3831,7 @@ ExclusionConstraintElem: index_elem WITH any_operator
 			}
 		;
 
-ExclusionWhereClause:
+OptWhereClause:
 			WHERE '(' a_expr ')'					{ $$ = $3; }
 			| /*EMPTY*/								{ $$ = NULL; }
 		;
-- 
2.7.4

From 6567e49f95823532bc2ceccf87ed570ca4ce398d Mon Sep 17 00:00:00 2001
From: Euler Taveira <[email-masked]>
Date: Tue, 27 Feb 2018 04:03:13 +0000
Subject: [PATCH 3/3] Row filtering for logical replication

When you define or modify a publication you optionally can filter rows
to be published using a WHERE condition. This condition is any
expression that evaluates to boolean. Only those rows that
satisfy the WHERE condition will be sent to subscribers.
---
 doc/src/sgml/ref/alter_publication.sgml     |  9 ++-
 doc/src/sgml/ref/create_publication.sgml    | 14 ++++-
 src/backend/catalog/pg_publication.c        | 45 +++++++++++--
 src/backend/commands/publicationcmds.c      | 69 ++++++++++++++------
 src/backend/parser/gram.y                   | 26 ++++++--
 src/backend/parser/parse_agg.c              | 10 +++
 src/backend/parser/parse_expr.c             |  5 ++
 src/backend/parser/parse_func.c             |  3 +
 src/backend/replication/logical/worker.c    |  2 +-
 src/backend/replication/pgoutput/pgoutput.c | 98 ++++++++++++++++++++++++++++-
 src/include/catalog/pg_publication.h        |  8 ++-
 src/include/catalog/pg_publication_rel.h    | 11 +++-
 src/include/nodes/nodes.h                   |  1 +
 src/include/nodes/parsenodes.h              | 11 +++-
 src/include/parser/parse_node.h             |  3 +-
 src/include/replication/logicalrelation.h   |  2 +
 src/test/subscription/t/001_rep_changes.pl  | 29 ++++++++-
 17 files changed, 299 insertions(+), 47 deletions(-)

diff --git a/doc/src/sgml/ref/alter_publication.sgml b/doc/src/sgml/ref/alter_publication.sgml
index 534e598..dc579b2 100644
--- a/doc/src/sgml/ref/alter_publication.sgml
+++ b/doc/src/sgml/ref/alter_publication.sgml
@@ -21,8 +21,8 @@ PostgreSQL documentation
 
  <refsynopsisdiv>
 <synopsis>
-ALTER PUBLICATION <replaceable class="parameter">name</replaceable> ADD TABLE [ ONLY ] <replaceable class="parameter">table_name</replaceable> [ * ] [, ...]
-ALTER PUBLICATION <replaceable class="parameter">name</replaceable> SET TABLE [ ONLY ] <replaceable class="parameter">table_name</replaceable> [ * ] [, ...]
+ALTER PUBLICATION <replaceable class="parameter">name</replaceable> ADD TABLE [ ONLY ] <replaceable class="parameter">table_name</replaceable> [ * ] [ WHERE <replaceable class="parameter">condition</replaceable> ] [, ...]
+ALTER PUBLICATION <replaceable class="parameter">name</replaceable> SET TABLE [ ONLY ] <replaceable class="parameter">table_name</replaceable> [ * ] [ WHERE <replaceable class="parameter">condition</replaceable> ] [, ...]
 ALTER PUBLICATION <replaceable class="parameter">name</replaceable> DROP TABLE [ ONLY ] <replaceable class="parameter">table_name</replaceable> [ * ] [, ...]
 ALTER PUBLICATION <replaceable class="parameter">name</replaceable> SET ( <replaceable class="parameter">publication_parameter</replaceable> [= <replaceable class="parameter">value</replaceable>] [, ... ] )
 ALTER PUBLICATION <replaceable class="parameter">name</replaceable> OWNER TO { <replaceable>new_owner</replaceable> | CURRENT_USER | SESSION_USER }
@@ -91,7 +91,10 @@ ALTER PUBLICATION <replaceable class="parameter">name</replaceable> RENAME TO <r
       table name, only that table is affected.  If <literal>ONLY</literal> is not
       specified, the table and all its descendant tables (if any) are
       affected.  Optionally, <literal>*</literal> can be specified after the table
-      name to explicitly indicate that descendant tables are included.
+      name to explicitly indicate that descendant tables are included. If the
+      optional <literal>WHERE</literal> clause is specified, rows that do not
+      satisfy the <replaceable class="parameter">condition</replaceable> will
+      not be published.
      </para>
     </listitem>
    </varlistentry>
diff --git a/doc/src/sgml/ref/create_publication.sgml b/doc/src/sgml/ref/create_publication.sgml
index bfe12d5..e42f3d4 100644
--- a/doc/src/sgml/ref/create_publication.sgml
+++ b/doc/src/sgml/ref/create_publication.sgml
@@ -22,7 +22,7 @@ PostgreSQL documentation
  <refsynopsisdiv>
 <synopsis>
 CREATE PUBLICATION <replaceable class="parameter">name</replaceable>
-    [ FOR TABLE [ ONLY ] <replaceable class="parameter">table_name</replaceable> [ * ] [, ...]
+    [ FOR TABLE [ ONLY ] <replaceable class="parameter">table_name</replaceable> [ * ] [ WHERE <replaceable class="parameter">condition</replaceable> ] [, ...]
       | FOR ALL TABLES ]
     [ WITH ( <replaceable class="parameter">publication_parameter</replaceable> [= <replaceable class="parameter">value</replaceable>] [, ... ] ) ]
 
@@ -68,7 +68,10 @@ CREATE PUBLICATION <replaceable class="parameter">name</replaceable>
       that table is added to the publication.  If <literal>ONLY</literal> is not
       specified, the table and all its descendant tables (if any) are added.
       Optionally, <literal>*</literal> can be specified after the table name to
-      explicitly indicate that descendant tables are included.
+      explicitly indicate that descendant tables are included. If the optional
+      <literal>WHERE</literal> clause is specified, rows that do not satisfy
+      the <replaceable class="parameter">condition</replaceable> will not be
+      published.
      </para>
 
      <para>
@@ -184,6 +187,13 @@ CREATE PUBLICATION mypublication FOR TABLE users, departments;
   </para>
 
   <para>
+   Create a publication that publishes all changes from active departments:
+<programlisting>
+CREATE PUBLICATION active_departments FOR TABLE departments WHERE (active IS TRUE);
+</programlisting>
+  </para>
+
+  <para>
    Create a publication that publishes all changes in all tables:
 <programlisting>
 CREATE PUBLICATION alltables FOR ALL TABLES;
diff --git a/src/backend/catalog/pg_publication.c b/src/backend/catalog/pg_publication.c
index ba18258..43d754d 100644
--- a/src/backend/catalog/pg_publication.c
+++ b/src/backend/catalog/pg_publication.c
@@ -34,6 +34,10 @@
 #include "catalog/pg_publication.h"
 #include "catalog/pg_publication_rel.h"
 
+#include "parser/parse_clause.h"
+#include "parser/parse_collate.h"
+#include "parser/parse_relation.h"
+
 #include "utils/array.h"
 #include "utils/builtins.h"
 #include "utils/catcache.h"
@@ -142,18 +146,21 @@ pg_relation_is_publishable(PG_FUNCTION_ARGS)
  * Insert new publication / relation mapping.
  */
 ObjectAddress
-publication_add_relation(Oid pubid, Relation targetrel,
+publication_add_relation(Oid pubid, PublicationRelationQual *targetrel,
 						 bool if_not_exists)
 {
 	Relation	rel;
 	HeapTuple	tup;
 	Datum		values[Natts_pg_publication_rel];
 	bool		nulls[Natts_pg_publication_rel];
-	Oid			relid = RelationGetRelid(targetrel);
+	Oid			relid = RelationGetRelid(targetrel->relation);
 	Oid			prrelid;
 	Publication *pub = GetPublication(pubid);
 	ObjectAddress myself,
 				referenced;
+	ParseState		*pstate;
+	RangeTblEntry	*rte;
+	Node			*whereclause;
 
 	rel = heap_open(PublicationRelRelationId, RowExclusiveLock);
 
@@ -173,10 +180,26 @@ publication_add_relation(Oid pubid, Relation targetrel,
 		ereport(ERROR,
 				(errcode(ERRCODE_DUPLICATE_OBJECT),
 				 errmsg("relation \"%s\" is already member of publication \"%s\"",
-						RelationGetRelationName(targetrel), pub->name)));
+						RelationGetRelationName(targetrel->relation), pub->name)));
 	}
 
-	check_publication_add_relation(targetrel);
+	check_publication_add_relation(targetrel->relation);
+
+	/* Set up a pstate to parse with */
+	pstate = make_parsestate(NULL);
+	pstate->p_sourcetext = nodeToString(targetrel->whereClause);
+
+	rte = addRangeTableEntryForRelation(pstate, targetrel->relation,
+										NULL, false, false);
+	addRTEtoQuery(pstate, rte, false, true, true);
+
+	whereclause = transformWhereClause(pstate,
+								copyObject(targetrel->whereClause),
+								EXPR_KIND_PUBLICATION_WHERE,
+								"PUBLICATION");
+
+	/* Fix up collation information */
+	assign_expr_collations(pstate, whereclause);
 
 	/* Form a tuple. */
 	memset(values, 0, sizeof(values));
@@ -187,6 +210,12 @@ publication_add_relation(Oid pubid, Relation targetrel,
 	values[Anum_pg_publication_rel_prrelid - 1] =
 		ObjectIdGetDatum(relid);
 
+	/* Add row filter, if available */
+	if (whereclause)
+		values[Anum_pg_publication_rel_prrowfilter - 1] = CStringGetTextDatum(nodeToString(whereclause));
+	else
+		nulls[Anum_pg_publication_rel_prrowfilter - 1] = true;
+
 	tup = heap_form_tuple(RelationGetDescr(rel), values, nulls);
 
 	/* Insert tuple into catalog. */
@@ -203,11 +232,17 @@ publication_add_relation(Oid pubid, Relation targetrel,
 	ObjectAddressSet(referenced, RelationRelationId, relid);
 	recordDependencyOn(&myself, &referenced, DEPENDENCY_AUTO);
 
+	/* Add dependency on the objects mentioned in the row filter expression */
+	if (whereclause)
+		recordDependencyOnExpr(&myself, whereclause, pstate->p_rtable, DEPENDENCY_NORMAL);
+
+	free_parsestate(pstate);
+
 	/* Close the table. */
 	heap_close(rel, RowExclusiveLock);
 
 	/* Invalidate relcache so that publication info is rebuilt. */
-	CacheInvalidateRelcache(targetrel);
+	CacheInvalidateRelcache(targetrel->relation);
 
 	return myself;
 }
diff --git a/src/backend/commands/publicationcmds.c b/src/backend/commands/publicationcmds.c
index 9c5aa9e..96347bb 100644
--- a/src/backend/commands/publicationcmds.c
+++ b/src/backend/commands/publicationcmds.c
@@ -324,6 +324,27 @@ AlterPublicationTables(AlterPublicationStmt *stmt, Relation rel,
 
 	Assert(list_length(stmt->tables) > 0);
 
+	/*
+	 * ALTER PUBLICATION ... DROP TABLE cannot contain a WHERE clause.  Use
+	 * publication_table_list node (that accepts a WHERE clause) but forbid the
+	 * WHERE clause in it.  The use of relation_expr_list node just for the
+	 * DROP TABLE part does not worth the trouble.
+	 */
+	if (stmt->tableAction == DEFELEM_DROP)
+	{
+		ListCell	*lc;
+
+		foreach(lc, stmt->tables)
+		{
+			PublicationTable *t = lfirst(lc);
+			if (t->whereClause)
+				ereport(ERROR,
+						(errcode(ERRCODE_OBJECT_NOT_IN_PREREQUISITE_STATE),
+						 errmsg("cannot use a WHERE clause for removing table from publication \"%s\"",
+								NameStr(pubform->pubname))));
+		}
+	}
+
 	rels = OpenTableList(stmt->tables);
 
 	if (stmt->tableAction == DEFELEM_ADD)
@@ -345,9 +366,9 @@ AlterPublicationTables(AlterPublicationStmt *stmt, Relation rel,
 
 			foreach(newlc, rels)
 			{
-				Relation	newrel = (Relation) lfirst(newlc);
+				PublicationRelationQual	*newrel = (PublicationRelationQual *) lfirst(newlc);
 
-				if (RelationGetRelid(newrel) == oldrelid)
+				if (RelationGetRelid(newrel->relation) == oldrelid)
 				{
 					found = true;
 					break;
@@ -356,7 +377,8 @@ AlterPublicationTables(AlterPublicationStmt *stmt, Relation rel,
 
 			if (!found)
 			{
-				Relation	oldrel = heap_open(oldrelid,
+				PublicationRelationQual *oldrel = palloc(sizeof(PublicationRelationQual));
+				oldrel->relation = heap_open(oldrelid,
 											   ShareUpdateExclusiveLock);
 
 				delrels = lappend(delrels, oldrel);
@@ -479,16 +501,18 @@ OpenTableList(List *tables)
 	List	   *relids = NIL;
 	List	   *rels = NIL;
 	ListCell   *lc;
+	PublicationRelationQual	*relqual;
 
 	/*
 	 * Open, share-lock, and check all the explicitly-specified relations
 	 */
 	foreach(lc, tables)
 	{
-		RangeVar   *rv = lfirst(lc);
-		Relation	rel;
-		bool		recurse = rv->inh;
-		Oid			myrelid;
+		PublicationTable	*t = lfirst(lc);
+		RangeVar  			*rv = t->relation;
+		Relation			rel;
+		bool				recurse = rv->inh;
+		Oid					myrelid;
 
 		CHECK_FOR_INTERRUPTS();
 
@@ -507,7 +531,10 @@ OpenTableList(List *tables)
 			heap_close(rel, ShareUpdateExclusiveLock);
 			continue;
 		}
-		rels = lappend(rels, rel);
+		relqual = palloc(sizeof(PublicationRelationQual));
+		relqual->relation = rel;
+		relqual->whereClause = t->whereClause;
+		rels = lappend(rels, relqual);
 		relids = lappend_oid(relids, myrelid);
 
 		if (recurse)
@@ -537,7 +564,11 @@ OpenTableList(List *tables)
 
 				/* find_all_inheritors already got lock */
 				rel = heap_open(childrelid, NoLock);
-				rels = lappend(rels, rel);
+				relqual = palloc(sizeof(PublicationRelationQual));
+				relqual->relation = rel;
+				/* child inherits WHERE clause from parent */
+				relqual->whereClause = t->whereClause;
+				rels = lappend(rels, relqual);
 				relids = lappend_oid(relids, childrelid);
 			}
 		}
@@ -558,10 +589,12 @@ CloseTableList(List *rels)
 
 	foreach(lc, rels)
 	{
-		Relation	rel = (Relation) lfirst(lc);
+		PublicationRelationQual	*rel = (PublicationRelationQual *) lfirst(lc);
 
-		heap_close(rel, NoLock);
+		heap_close(rel->relation, NoLock);
 	}
+
+	list_free_deep(rels);
 }
 
 /*
@@ -577,13 +610,13 @@ PublicationAddTables(Oid pubid, List *rels, bool if_not_exists,
 
 	foreach(lc, rels)
 	{
-		Relation	rel = (Relation) lfirst(lc);
+		PublicationRelationQual	*rel = (PublicationRelationQual *) lfirst(lc);
 		ObjectAddress obj;
 
 		/* Must be owner of the table or superuser. */
-		if (!pg_class_ownercheck(RelationGetRelid(rel), GetUserId()))
-			aclcheck_error(ACLCHECK_NOT_OWNER, get_relkind_objtype(rel->rd_rel->relkind),
-						   RelationGetRelationName(rel));
+		if (!pg_class_ownercheck(RelationGetRelid(rel->relation), GetUserId()))
+			aclcheck_error(ACLCHECK_NOT_OWNER, get_relkind_objtype(rel->relation->rd_rel->relkind),
+						   RelationGetRelationName(rel->relation));
 
 		obj = publication_add_relation(pubid, rel, if_not_exists);
 		if (stmt)
@@ -609,8 +642,8 @@ PublicationDropTables(Oid pubid, List *rels, bool missing_ok)
 
 	foreach(lc, rels)
 	{
-		Relation	rel = (Relation) lfirst(lc);
-		Oid			relid = RelationGetRelid(rel);
+		PublicationRelationQual *rel = (PublicationRelationQual *) lfirst(lc);
+		Oid			relid = RelationGetRelid(rel->relation);
 
 		prid = GetSysCacheOid2(PUBLICATIONRELMAP, ObjectIdGetDatum(relid),
 							   ObjectIdGetDatum(pubid));
@@ -622,7 +655,7 @@ PublicationDropTables(Oid pubid, List *rels, bool missing_ok)
 			ereport(ERROR,
 					(errcode(ERRCODE_UNDEFINED_OBJECT),
 					 errmsg("relation \"%s\" is not part of the publication",
-							RelationGetRelationName(rel))));
+							RelationGetRelationName(rel->relation))));
 		}
 
 		ObjectAddressSet(obj, PublicationRelRelationId, prid);
diff --git a/src/backend/parser/gram.y b/src/backend/parser/gram.y
index bf32362..94cdd7d 100644
--- a/src/backend/parser/gram.y
+++ b/src/backend/parser/gram.y
@@ -396,13 +396,13 @@ static Node *makeRecursiveViewSelect(char *relname, List *aliases, Node *query);
 				relation_expr_list dostmt_opt_list
 				transform_element_list transform_type_list
 				TriggerTransitions TriggerReferencing
-				publication_name_list
+				publication_name_list publication_table_list
 				vacuum_relation_list opt_vacuum_relation_list
 
 %type <list>	group_by_list
 %type <node>	group_by_item empty_grouping_set rollup_clause cube_clause
 %type <node>	grouping_sets_clause
-%type <node>	opt_publication_for_tables publication_for_tables
+%type <node>	opt_publication_for_tables publication_for_tables publication_table_elem
 %type <value>	publication_name_item
 
 %type <list>	opt_fdw_options fdw_options
@@ -9520,7 +9520,7 @@ opt_publication_for_tables:
 		;
 
 publication_for_tables:
-			FOR TABLE relation_expr_list
+			FOR TABLE publication_table_list
 				{
 					$$ = (Node *) $3;
 				}
@@ -9551,7 +9551,7 @@ AlterPublicationStmt:
 					n->options = $5;
 					$$ = (Node *)n;
 				}
-			| ALTER PUBLICATION name ADD_P TABLE relation_expr_list
+			| ALTER PUBLICATION name ADD_P TABLE publication_table_list
 				{
 					AlterPublicationStmt *n = makeNode(AlterPublicationStmt);
 					n->pubname = $3;
@@ -9559,7 +9559,7 @@ AlterPublicationStmt:
 					n->tableAction = DEFELEM_ADD;
 					$$ = (Node *)n;
 				}
-			| ALTER PUBLICATION name SET TABLE relation_expr_list
+			| ALTER PUBLICATION name SET TABLE publication_table_list
 				{
 					AlterPublicationStmt *n = makeNode(AlterPublicationStmt);
 					n->pubname = $3;
@@ -9567,7 +9567,7 @@ AlterPublicationStmt:
 					n->tableAction = DEFELEM_SET;
 					$$ = (Node *)n;
 				}
-			| ALTER PUBLICATION name DROP TABLE relation_expr_list
+			| ALTER PUBLICATION name DROP TABLE publication_table_list
 				{
 					AlterPublicationStmt *n = makeNode(AlterPublicationStmt);
 					n->pubname = $3;
@@ -9577,6 +9577,20 @@ AlterPublicationStmt:
 				}
 		;
 
+publication_table_list:
+			publication_table_elem									{ $$ = list_make1($1); }
+			| publication_table_list ',' publication_table_elem		{ $$ = lappend($1, $3); }
+		;
+
+publication_table_elem: relation_expr OptWhereClause
+				{
+					PublicationTable *n = makeNode(PublicationTable);
+					n->relation = $1;
+					n->whereClause = $2;
+					$$ = (Node *) n;
+				}
+		;
+
 /*****************************************************************************
  *
  * CREATE SUBSCRIPTION name ...
diff --git a/src/backend/parser/parse_agg.c b/src/backend/parser/parse_agg.c
index 377a7ed..7e1c3d8 100644
--- a/src/backend/parser/parse_agg.c
+++ b/src/backend/parser/parse_agg.c
@@ -522,6 +522,13 @@ check_agglevels_and_constraints(ParseState *pstate, Node *expr)
 				err = _("grouping operations are not allowed in CALL arguments");
 
 			break;
+		case EXPR_KIND_PUBLICATION_WHERE:
+			if (isAgg)
+				err = _("aggregate functions are not allowed in publication WHERE conditions");
+			else
+				err = _("grouping operations are not allowed in publication WHERE conditions");
+
+			break;
 
 			/*
 			 * There is intentionally no default: case here, so that the
@@ -902,6 +909,9 @@ transformWindowFuncCall(ParseState *pstate, WindowFunc *wfunc,
 		case EXPR_KIND_CALL_ARGUMENT:
 			err = _("window functions are not allowed in CALL arguments");
 			break;
+		case EXPR_KIND_PUBLICATION_WHERE:
+			err = _("window functions are not allowed in publication WHERE conditions");
+			break;
 
 			/*
 			 * There is intentionally no default: case here, so that the
diff --git a/src/backend/parser/parse_expr.c b/src/backend/parser/parse_expr.c
index 385e54a..7bd1695 100644
--- a/src/backend/parser/parse_expr.c
+++ b/src/backend/parser/parse_expr.c
@@ -1849,6 +1849,9 @@ transformSubLink(ParseState *pstate, SubLink *sublink)
 		case EXPR_KIND_CALL_ARGUMENT:
 			err = _("cannot use subquery in CALL argument");
 			break;
+		case EXPR_KIND_PUBLICATION_WHERE:
+			err = _("cannot use subquery in publication WHERE condition");
+			break;
 
 			/*
 			 * There is intentionally no default: case here, so that the
@@ -3475,6 +3478,8 @@ ParseExprKindName(ParseExprKind exprKind)
 			return "PARTITION BY";
 		case EXPR_KIND_CALL_ARGUMENT:
 			return "CALL";
+		case EXPR_KIND_PUBLICATION_WHERE:
+			return "publication WHERE";
 
 			/*
 			 * There is intentionally no default: case here, so that the
diff --git a/src/backend/parser/parse_func.c b/src/backend/parser/parse_func.c
index 2a4ac09..8e9cc58 100644
--- a/src/backend/parser/parse_func.c
+++ b/src/backend/parser/parse_func.c
@@ -2293,6 +2293,9 @@ check_srf_call_placement(ParseState *pstate, Node *last_srf, int location)
 		case EXPR_KIND_CALL_ARGUMENT:
 			err = _("set-returning functions are not allowed in CALL arguments");
 			break;
+		case EXPR_KIND_PUBLICATION_WHERE:
+			err = _("set-returning functions are not allowed in publication WHERE conditions");
+			break;
 
 			/*
 			 * There is intentionally no default: case here, so that the
diff --git a/src/backend/replication/logical/worker.c b/src/backend/replication/logical/worker.c
index 6820c1a..fe0a6ca 100644
--- a/src/backend/replication/logical/worker.c
+++ b/src/backend/replication/logical/worker.c
@@ -182,7 +182,7 @@ ensure_transaction(void)
  *
  * This is based on similar code in copy.c
  */
-static EState *
+EState *
 create_estate_for_relation(Relation rel)
 {
 	EState	   *estate;
diff --git a/src/backend/replication/pgoutput/pgoutput.c b/src/backend/replication/pgoutput/pgoutput.c
index d538f25..30bdefa 100644
--- a/src/backend/replication/pgoutput/pgoutput.c
+++ b/src/backend/replication/pgoutput/pgoutput.c
@@ -12,13 +12,23 @@
  */
 #include "postgres.h"
 
+#include "catalog/pg_type.h"
 #include "catalog/pg_publication.h"
+#include "catalog/pg_publication_rel.h"
+
+#include "executor/executor.h"
+#include "nodes/execnodes.h"
+#include "nodes/nodeFuncs.h"
+#include "optimizer/planner.h"
+#include "parser/parse_coerce.h"
 
 #include "replication/logical.h"
 #include "replication/logicalproto.h"
+#include "replication/logicalrelation.h"
 #include "replication/origin.h"
 #include "replication/pgoutput.h"
 
+#include "utils/builtins.h"
 #include "utils/inval.h"
 #include "utils/int8.h"
 #include "utils/lsyscache.h"
@@ -56,6 +66,7 @@ typedef struct RelationSyncEntry
 	bool		schema_sent;	/* did we send the schema? */
 	bool		replicate_valid;
 	PublicationActions pubactions;
+	List		*row_filter;
 } RelationSyncEntry;
 
 /* Map used to remember which relation schemas we sent. */
@@ -286,6 +297,62 @@ pgoutput_change(LogicalDecodingContext *ctx, ReorderBufferTXN *txn,
 			Assert(false);
 	}
 
+	/* ... then check row filter */
+	if (list_length(relentry->row_filter) > 0)
+	{
+		HeapTuple		old_tuple;
+		HeapTuple		new_tuple;
+		TupleDesc		tupdesc;
+		EState			*estate;
+		ExprContext		*ecxt;
+		MemoryContext	oldcxt;
+		ListCell		*lc;
+
+		old_tuple = change->data.tp.oldtuple ? &change->data.tp.oldtuple->tuple : NULL;
+		new_tuple = change->data.tp.newtuple ? &change->data.tp.newtuple->tuple : NULL;
+		tupdesc = RelationGetDescr(relation);
+		estate = create_estate_for_relation(relation);
+
+		/* prepare context per tuple */
+		ecxt = GetPerTupleExprContext(estate);
+		oldcxt = MemoryContextSwitchTo(estate->es_query_cxt);
+		ecxt->ecxt_scantuple = ExecInitExtraTupleSlot(estate, tupdesc);
+		MemoryContextSwitchTo(oldcxt);
+
+		ExecStoreTuple(new_tuple ? new_tuple : old_tuple, ecxt->ecxt_scantuple, InvalidBuffer, false);
+
+		foreach (lc, relentry->row_filter)
+		{
+			Node		*row_filter;
+			ExprState	*expr_state;
+			Expr		*expr;
+			Oid			expr_type;
+			Datum		res;
+			bool		isnull;
+			char		*s = NULL;
+
+			row_filter = (Node *) lfirst(lc);
+
+			/* evaluates row filter */
+			expr_type = exprType(row_filter);
+			expr = (Expr *) coerce_to_target_type(NULL, row_filter, expr_type, BOOLOID, -1, COERCION_ASSIGNMENT, COERCE_IMPLICIT_CAST, -1);
+			expr = expression_planner(expr);
+			expr_state = ExecInitExpr(expr, NULL);
+			res = ExecEvalExpr(expr_state, ecxt, &isnull);
+
+			/* if tuple does not match row filter, bail out */
+			if (!DatumGetBool(res) || isnull)
+				return;
+
+			s = nodeToString(row_filter);
+			elog(DEBUG2, "filter \"%s\" was matched", s);
+			pfree(s);
+		}
+
+		ExecDropSingleTupleTableSlot(ecxt->ecxt_scantuple);
+		FreeExecutorState(estate);
+	}
+
 	/* Avoid leaking memory by using and resetting our own context */
 	old = MemoryContextSwitchTo(data->context);
 
@@ -506,10 +573,14 @@ get_rel_sync_entry(PGOutputData *data, Oid relid)
 		 */
 		entry->pubactions.pubinsert = entry->pubactions.pubupdate =
 			entry->pubactions.pubdelete = false;
+		entry->row_filter = NIL;
 
 		foreach(lc, data->publications)
 		{
 			Publication *pub = lfirst(lc);
+			HeapTuple	rf_tuple;
+			Datum		rf_datum;
+			bool		rf_isnull;
 
 			/*
 			 * Skip tables that look like they are from a heap rewrite (see
@@ -543,9 +614,25 @@ get_rel_sync_entry(PGOutputData *data, Oid relid)
 				entry->pubactions.pubdelete |= pub->pubactions.pubdelete;
 			}
 
-			if (entry->pubactions.pubinsert && entry->pubactions.pubupdate &&
-				entry->pubactions.pubdelete)
-				break;
+			/* Cache row filters, if available */
+			rf_tuple = SearchSysCache2(PUBLICATIONRELMAP, ObjectIdGetDatum(relid), ObjectIdGetDatum(pub->oid));
+			if (HeapTupleIsValid(rf_tuple))
+			{
+				rf_datum = SysCacheGetAttr(PUBLICATIONRELMAP, rf_tuple, Anum_pg_publication_rel_prrowfilter, &rf_isnull);
+
+				if (!rf_isnull)
+				{
+					MemoryContext oldctx = MemoryContextSwitchTo(CacheMemoryContext);
+					char	*s = TextDatumGetCString(rf_datum);
+					Node	*rf_node = stringToNode(s);
+					entry->row_filter = lappend(entry->row_filter, rf_node);
+					MemoryContextSwitchTo(oldctx);
+
+					elog(DEBUG2, "row filter \"%s\" found for publication \"%s\" and relation \"%s\"", s, pub->name, get_rel_name(relid));
+				}
+
+				ReleaseSysCache(rf_tuple);
+			}
 		}
 
 		list_free(pubids);
@@ -620,5 +707,10 @@ rel_sync_cache_publication_cb(Datum arg, int cacheid, uint32 hashvalue)
 	 */
 	hash_seq_init(&status, RelationSyncCache);
 	while ((entry = (RelationSyncEntry *) hash_seq_search(&status)) != NULL)
+	{
 		entry->replicate_valid = false;
+		if (list_length(entry->row_filter) > 0)
+			list_free(entry->row_filter);
+		entry->row_filter = NIL;
+	}
 }
diff --git a/src/include/catalog/pg_publication.h b/src/include/catalog/pg_publication.h
index 37e77b8..28962e6 100644
--- a/src/include/catalog/pg_publication.h
+++ b/src/include/catalog/pg_publication.h
@@ -86,6 +86,12 @@ typedef struct Publication
 	PublicationActions pubactions;
 } Publication;
 
+typedef struct PublicationRelationQual
+{
+	Relation	relation;
+	Node		*whereClause;
+} PublicationRelationQual;
+
 extern Publication *GetPublication(Oid pubid);
 extern Publication *GetPublicationByName(const char *pubname, bool missing_ok);
 extern List *GetRelationPublications(Oid relid);
@@ -94,7 +100,7 @@ extern List *GetAllTablesPublications(void);
 extern List *GetAllTablesPublicationRelations(void);
 
 extern bool is_publishable_relation(Relation rel);
-extern ObjectAddress publication_add_relation(Oid pubid, Relation targetrel,
+extern ObjectAddress publication_add_relation(Oid pubid, PublicationRelationQual *targetrel,
 						 bool if_not_exists);
 
 extern Oid	get_publication_oid(const char *pubname, bool missing_ok);
diff --git a/src/include/catalog/pg_publication_rel.h b/src/include/catalog/pg_publication_rel.h
index 033b600..585f855 100644
--- a/src/include/catalog/pg_publication_rel.h
+++ b/src/include/catalog/pg_publication_rel.h
@@ -29,8 +29,12 @@
 
 CATALOG(pg_publication_rel,6106)
 {
-	Oid			prpubid;		/* Oid of the publication */
-	Oid			prrelid;		/* Oid of the relation */
+	Oid				prpubid;		/* Oid of the publication */
+	Oid				prrelid;		/* Oid of the relation */
+
+#ifdef	CATALOG_VARLEN				/* variable-length fields start here */
+	pg_node_tree	prrowfilter;	/* nodeToString representation of row filter */
+#endif
 } FormData_pg_publication_rel;
 
 /* ----------------
@@ -45,8 +49,9 @@ typedef FormData_pg_publication_rel *Form_pg_publication_rel;
  * ----------------
  */
 
-#define Natts_pg_publication_rel				2
+#define Natts_pg_publication_rel				3
 #define Anum_pg_publication_rel_prpubid			1
 #define Anum_pg_publication_rel_prrelid			2
+#define	Anum_pg_publication_rel_prrowfilter		3
 
 #endif							/* PG_PUBLICATION_REL_H */
diff --git a/src/include/nodes/nodes.h b/src/include/nodes/nodes.h
index 74b094a..499d839 100644
--- a/src/include/nodes/nodes.h
+++ b/src/include/nodes/nodes.h
@@ -471,6 +471,7 @@ typedef enum NodeTag
 	T_PartitionRangeDatum,
 	T_PartitionCmd,
 	T_VacuumRelation,
+	T_PublicationTable,
 
 	/*
 	 * TAGS FOR REPLICATION GRAMMAR PARSE NODES (replnodes.h)
diff --git a/src/include/nodes/parsenodes.h b/src/include/nodes/parsenodes.h
index ac292bc..9800acf 100644
--- a/src/include/nodes/parsenodes.h
+++ b/src/include/nodes/parsenodes.h
@@ -3419,12 +3419,19 @@ typedef struct AlterTSConfigurationStmt
 } AlterTSConfigurationStmt;

+typedef struct PublicationTable
+{
+	NodeTag		type;
+	RangeVar	*relation;		/* relation to be published */
+	Node		*whereClause;	/* qualifications */
+} PublicationTable;
+
 typedef struct CreatePublicationStmt
 {
 	NodeTag		type;
 	char	   *pubname;		/* Name of of the publication */
 	List	   *options;		/* List of DefElem nodes */
-	List	   *tables;			/* Optional list of tables to add */
+	List	   *tables;			/* Optional list of PublicationTable to add */
 	bool		for_all_tables; /* Special publication for all tables in db */
 } CreatePublicationStmt;
 
@@ -3437,7 +3444,7 @@ typedef struct AlterPublicationStmt
 	List	   *options;		/* List of DefElem nodes */
 
 	/* parameters used for ALTER PUBLICATION ... ADD/DROP TABLE */
-	List	   *tables;			/* List of tables to add/drop */
+	List	   *tables;			/* List of PublicationTable to add/drop */
 	bool		for_all_tables; /* Special publication for all tables in db */
 	DefElemAction tableAction;	/* What action to perform with the tables */
 } AlterPublicationStmt;
diff --git a/src/include/parser/parse_node.h b/src/include/parser/parse_node.h
index 0230543..8e3c735 100644
--- a/src/include/parser/parse_node.h
+++ b/src/include/parser/parse_node.h
@@ -69,7 +69,8 @@ typedef enum ParseExprKind
 	EXPR_KIND_TRIGGER_WHEN,		/* WHEN condition in CREATE TRIGGER */
 	EXPR_KIND_POLICY,			/* USING or WITH CHECK expr in policy */
 	EXPR_KIND_PARTITION_EXPRESSION, /* PARTITION BY expression */
-	EXPR_KIND_CALL_ARGUMENT		/* procedure argument in CALL */
+	EXPR_KIND_CALL_ARGUMENT,	/* procedure argument in CALL */
+	EXPR_KIND_PUBLICATION_WHERE	/* WHERE condition for a table in PUBLICATION */
 } ParseExprKind;

diff --git a/src/include/replication/logicalrelation.h b/src/include/replication/logicalrelation.h
index d4250c2..32f1312 100644
--- a/src/include/replication/logicalrelation.h
+++ b/src/include/replication/logicalrelation.h
@@ -39,4 +39,6 @@ extern void logicalrep_rel_close(LogicalRepRelMapEntry *rel,
 extern void logicalrep_typmap_update(LogicalRepTyp *remotetyp);
 extern Oid	logicalrep_typmap_getid(Oid remoteid);
 
+extern EState *create_estate_for_relation(Relation rel);
+
 #endif							/* LOGICALRELATION_H */
diff --git a/src/test/subscription/t/001_rep_changes.pl b/src/test/subscription/t/001_rep_changes.pl
index e0104cd..d40ae03 100644
--- a/src/test/subscription/t/001_rep_changes.pl
+++ b/src/test/subscription/t/001_rep_changes.pl
@@ -3,7 +3,7 @@ use strict;
 use warnings;
 use PostgresNode;
 use TestLib;
-use Test::More tests => 16;
+use Test::More tests => 17;
 
 # Initialize publisher node
 my $node_publisher = get_new_node('publisher');
@@ -31,6 +31,8 @@ $node_publisher->safe_psql('postgres',
 	"CREATE TABLE tab_mixed (a int primary key, b text)");
 $node_publisher->safe_psql('postgres',
 	"INSERT INTO tab_mixed (a, b) VALUES (1, 'foo')");
+$node_publisher->safe_psql('postgres',
+	"CREATE TABLE tab_rowfilter (a int primary key, b text)");
 
 # Setup structure on subscriber
 $node_subscriber->safe_psql('postgres', "CREATE TABLE tab_notrep (a int)");
@@ -39,6 +41,8 @@ $node_subscriber->safe_psql('postgres', "CREATE TABLE tab_full (a int)");
 $node_subscriber->safe_psql('postgres', "CREATE TABLE tab_full2 (x text)");
 $node_subscriber->safe_psql('postgres',
 	"CREATE TABLE tab_rep (a int primary key)");
+$node_subscriber->safe_psql('postgres',
+	"CREATE TABLE tab_rowfilter (a int primary key, b text)");
 
 # different column count and order than on publisher
 $node_subscriber->safe_psql('postgres',
@@ -54,10 +58,12 @@ $node_publisher->safe_psql('postgres',
 );
 $node_publisher->safe_psql('postgres',
 	"ALTER PUBLICATION tap_pub_ins_only ADD TABLE tab_ins");
+$node_publisher->safe_psql('postgres',
+	"CREATE PUBLICATION tap_pub_row_filter FOR TABLE tab_rowfilter WHERE (a > 1000 AND b <> 'filtered')");
 
 my $appname = 'tap_sub';
 $node_subscriber->safe_psql('postgres',
-"CREATE SUBSCRIPTION tap_sub CONNECTION '$publisher_connstr application_name=$appname' PUBLICATION tap_pub, tap_pub_ins_only"
+"CREATE SUBSCRIPTION tap_sub CONNECTION '$publisher_connstr application_name=$appname' PUBLICATION tap_pub, tap_pub_ins_only, tap_pub_row_filter"
 );
 
 $node_publisher->wait_for_catchup($appname);
@@ -76,6 +82,25 @@ $result =
   $node_subscriber->safe_psql('postgres', "SELECT count(*) FROM tab_ins");
 is($result, qq(1002), 'check initial data was copied to subscriber');
 
+# row filter
+$node_publisher->safe_psql('postgres',
+	"INSERT INTO tab_rowfilter (a, b) VALUES (1, 'not replicated')");
+$node_publisher->safe_psql('postgres',
+	"INSERT INTO tab_rowfilter (a, b) VALUES (1500, 'filtered')");
+$node_publisher->safe_psql('postgres',
+	"INSERT INTO tab_rowfilter (a, b) VALUES (1980, 'not filtered')");
+$node_publisher->safe_psql('postgres',
+	"INSERT INTO tab_rowfilter (a, b) SELECT x, 'test ' || x FROM generate_series(990,1003) x");
+
+$node_publisher->wait_for_catchup($appname);
+
+$result =
+  $node_subscriber->safe_psql('postgres', "SELECT a, b FROM tab_rowfilter");
+is($result, qq(1980|not filtered
+1001|test 1001
+1002|test 1002
+1003|test 1003), 'check initial data was copied to subscriber');
+
 $node_publisher->safe_psql('postgres',
 	"INSERT INTO tab_ins SELECT generate_series(1,50)");
 $node_publisher->safe_psql('postgres', "DELETE FROM tab_ins WHERE a > 20");
-- 
2.7.4

---

From: David Fetter <[email-masked]>
Date: Thu, 1 Mar 2018 01:47:52 +0100

> independently. 0003 doesn't include row filtering on initial

Great feature!  I think a lot of people will like to have the option
of trading a little extra CPU on the pub side for a bunch of network
traffic and some work on the sub side.

I noticed that the WHERE clause applies to all tables in the
publication.  Is that actually the right thing?  I'm thinking of a
case where we have foo(id, ...) and bar(foo_id, ....).  To slice that
correctly, we'd want to do the ids in the foo table and the foo_ids in
the bar table.  In the system as written, that would entail, at least
potentially, writing a lot of publications by hand.

Something like
    WHERE (
        (table_1,..., table_N) HAS (/* WHERE clause here */) AND
        (table_N+1,..., table_M) HAS (/* WHERE clause here */) AND
        ...
    )

could be one way to specify.

I also noticed that in psql, \dRp+ doesn't show the WHERE clause,
which it probably should.

Does it need regression tests?

Best,
David.
-- 
David Fetter <david(at)fetter(dot)org> http://fetter.org/
Phone: +1 415 235 3778

Remember to vote!
Consider donating to Postgres: http://www.postgresql.org/about/donate

---

From: Craig Ringer <[email-masked]>
Date: Thu, 1 Mar 2018 08:54:58 +0800

> independently. 0003 doesn't include row filtering on initial

Good idea. I haven't read this yet, but one thing to make sure you've
handled is limiting the clause to referencing only the current tuple and
the catalogs. user-catalog tables are OK, too, anything that is
RelationIsAccessibleInLogicalDecoding().

This means only immutable functions may be invoked, since a stable or
volatile function might attempt to access a table. And views must be
prohibited or recursively checked. (We have tree walkers that would help
with this).

It might be worth looking at the current logic for CHECK expressions, since
the requirements are similar. In my opinion you could safely not bother
with allowing access to user catalog tables in the filter expressions and
limit them strictly to immutable functions and the tuple its self.

-- 
 Craig Ringer                   http://www.2ndQuadrant.com/
 PostgreSQL Development, 24x7 Support, Training & Services

---

From: Erik Rijkers <[email-masked]>
Date: Thu, 01 Mar 2018 16:27:11 +0100

> 001-Refactor-function-create_estate_for_relation.patch
> 0002-Rename-a-WHERE-node.patch
> 0003-Row-filtering-for-logical-replication.patch

> Comments?

Very, very useful.  I really do hope this patch survives the 
late-arrival-cull.

I built this functionality into a test program I have been using and in 
simple cascading replication tests it works well.

I did find what I think is a bug (a bug easy to avoid but also easy to 
run into):
The test I used was to cascade 3 instances (all on one machine) from 
A->B->C
I ran a pgbench session in instance A, and used:
   in A: alter publication pub0_6515 add table pgbench_accounts where 
(aid between 40000 and 60000-1);
   in B: alter publication pub1_6516 add table pgbench_accounts;

The above worked well, but when I did the same but used the filter in 
both publications:
   in A: alter publication pub0_6515 add table pgbench_accounts where 
(aid between 40000 and 60000-1);
   in B: alter publication pub1_6516 add table pgbench_accounts where 
(aid between 40000 and 60000-1);

then the replication only worked for (pgbench-)scale 1 (hence: very 
little data); with larger scales it became slow (taking many minutes 
where the above had taken less than 1 minute), and ended up using far 
too much memory (or blowing up/crashing altogether).  Something not 
quite right there.

Nevertheless, I am much in favour of acquiring this functionality as 
soon as possible.

Thanks,

Erik Rijkers

---

From: Euler Taveira <[email-masked]>
Date: Thu, 1 Mar 2018 12:41:04 -0300

> potentially, writing a lot of publications by hand.
I didn't make it clear in my previous email and I think you misread
the attached docs. Each table can have an optional WHERE clause. I'll
made it clear when I rewrite the tests. Something like:

CREATE PUBLICATION tap_pub FOR TABLE tab_rowfilter_1 WHERE (a > 1000
AND b <> 'filtered'), tab_rowfilter_2 WHERE (c % 2 = 0),
tab_rowfilter_3;

Such syntax will not block another future feature that will publish
only few columns of the table.

> which it probably should.
Yea, it could be added be I'm afraid of such long WHERE clauses.

> Does it need regression tests?
I included some tests just to demonstrate the feature but I'm planning
to add a separate test file for it.

---

From: David Fetter <[email-masked]>
Date: Thu, 1 Mar 2018 17:06:42 +0100

> made it clear when I rewrite the tests. Something like:

Sorry I misunderstood.

> AND b <> 'filtered'), tab_rowfilter_2 WHERE (c % 2 = 0),

That's great!

> Yea, it could be added be I'm afraid of such long WHERE clauses.

I think of + as signifying, "I am ready to get a LOT of output in
order to see more detail."  Perhaps that's just me.

> I included some tests just to demonstrate the feature but I'm
> planning to add a separate test file for it.

Excellent. This feature looks like a nice big chunk of the user-space
infrastructure needed for sharding, among other things.

Best,
David.

Remember to vote!

---

From: Erik Rijkers <[email-masked]>
Date: Thu, 01 Mar 2018 22:25:02 +0100

On 2018-03-01 16:27, Erik Rijkers wrote:
> I built this functionality into a test program I have been using and
> in simple cascading replication tests it works well.
> 

Attached is 'logrep_rowfilter.sh', a demonstration of above-described 
bug.

The program runs initdb for 3 instances in /tmp (using ports 6515, 6516, 
and 6517) and sets up logical replication from 1->2->3.

It can be made to work by removing de where-clause on the second 'create 
publication' ( i.e., outcomment the $where2 variable ).

---

From: Andres Freund <[email-masked]>
Date: Thu, 1 Mar 2018 13:27:56 -0800

Hi,

> Very, very useful.  I really do hope this patch survives the

FWIW, I don't think it'd be fair or prudent. There's definitely some
issues (see e.g. Craig's reply), and I don't see why this patch'd
deserve an exemption from the "nontrivial patches shouldn't be submitted
to the last CF" policy?

- Andres

---

From: David Steele <[email-masked]>
Date: Thu, 1 Mar 2018 16:51:39 -0500

Hi,

> deserve an exemption from the "nontrivial patches shouldn't be submitted

I'm unable to find this in the CF under the title or author name.  If it
didn't get entered then it is definitely out.

If it does have an entry, then I agree with Andres that it should be
pushed to the next CF.

-- 
-David
[email-masked]

---

From: Euler Taveira <[email-masked]>
Date: Thu, 1 Mar 2018 20:00:37 -0300

> deserve an exemption from the "nontrivial patches shouldn't be submitted
I forgot to mention but this feature is for v12. I know the rules and
that is why I didn't add it to the in progress CF.

---

From: Euler Taveira <[email-masked]>
Date: Thu, 1 Mar 2018 20:03:42 -0300

2018-03-01 18:25 GMT-03:00 Erik Rijkers <[email-masked]>:
> Attached is 'logrep_rowfilter.sh', a demonstration of above-described bug.
>
Thanks for testing. I will figure out what is happening. There are
some leaks around. I'll post another version when I fix some of those
bugs.

---

From: Euler Taveira <[email-masked]>
Date: Thu, 1 Mar 2018 20:15:37 -0300

2018-02-28 21:54 GMT-03:00 Craig Ringer <[email-masked]>:
> handled is limiting the clause to referencing only the current tuple and the
> catalogs. user-catalog tables are OK, too, anything that is
> the requirements are similar. In my opinion you could safely not bother with
> allowing access to user catalog tables in the filter expressions and limit
> them strictly to immutable functions and the tuple its self.
>
IIRC implementation is similar to RLS expressions. I'll check all of
these rules.

---

From: David Steele <[email-masked]>
Date: Thu, 1 Mar 2018 18:16:17 -0500

> that is why I didn't add it to the in progress CF.

That was the right thing to do, thank you!

-- 
-David
[email-masked]

---

From: Michael Paquier <[email-masked]>
Date: Tue, 2 Oct 2018 14:03:15 +0900

> That was the right thing to do, thank you!

This patch has been waiting on author for a couple of months and does
not apply anymore, so I am marking as returned with feedback.  If you
can rebase, please feel free to resubmit.
--
Michael

---

From: Euler Taveira <[email-masked]>
Date: Wed, 31 Oct 2018 21:29:59 -0300

> The attached patches add support for filtering rows in the publisher.
I rebased the patch. I added row filtering for initial
synchronization, pg_dump support and psql support. 0001 removes unused
code. 0002 reduces memory use. 0003 passes only structure member that
is used in create_estate_for_relation. 0004 reuses a parser node for
row filtering. 0005 is the feature. 0006 prints WHERE expression in
psql. 0007 adds pg_dump support. 0008 is only for debug purposes (I'm
not sure some of these messages will be part of the final patch).
0001, 0002, 0003 and 0008 are not mandatory for this feature.

Comments?

-- 
   Euler Taveira                                   Timbira -
http://www.timbira.com.br/
   PostgreSQL: Consultoria, Desenvolvimento, Suporte 24x7 e Treinamento
From b2e56eaa9e16246c8158ff2961a6a4b2acbd096b Mon Sep 17 00:00:00 2001
From: Euler Taveira <[email-masked]>
Date: Fri, 9 Mar 2018 18:39:22 +0000
Subject: [PATCH 1/8] Remove unused atttypmod column from initial table
 synchronization

 Since commit 7c4f52409a8c7d85ed169bbbc1f6092274d03920, atttypmod was
 added but not used. The removal is safe because COPY from publisher
 does not need such information.
---
 src/backend/replication/logical/tablesync.c | 7 +++----
 1 file changed, 3 insertions(+), 4 deletions(-)

diff --git a/src/backend/replication/logical/tablesync.c b/src/backend/replication/logical/tablesync.c
index 6e420d8..f285813 100644
--- a/src/backend/replication/logical/tablesync.c
+++ b/src/backend/replication/logical/tablesync.c
@@ -660,7 +660,7 @@ fetch_remote_table_info(char *nspname, char *relname,
 	StringInfoData cmd;
 	TupleTableSlot *slot;
 	Oid			tableRow[2] = {OIDOID, CHAROID};
-	Oid			attrRow[4] = {TEXTOID, OIDOID, INT4OID, BOOLOID};
+	Oid			attrRow[3] = {TEXTOID, OIDOID, BOOLOID};
 	bool		isnull;
 	int			natt;
 
@@ -704,7 +704,6 @@ fetch_remote_table_info(char *nspname, char *relname,
 	appendStringInfo(&cmd,
 					 "SELECT a.attname,"
 					 "       a.atttypid,"
-					 "       a.atttypmod,"
 					 "       a.attnum = ANY(i.indkey)"
 					 "  FROM pg_catalog.pg_attribute a"
 					 "  LEFT JOIN pg_catalog.pg_index i"
@@ -714,7 +713,7 @@ fetch_remote_table_info(char *nspname, char *relname,
 					 "   AND a.attrelid = %u"
 					 " ORDER BY a.attnum",
 					 lrel->remoteid, lrel->remoteid);
-	res = walrcv_exec(wrconn, cmd.data, 4, attrRow);
+	res = walrcv_exec(wrconn, cmd.data, 3, attrRow);
 
 	if (res->status != WALRCV_OK_TUPLES)
 		ereport(ERROR,
@@ -735,7 +734,7 @@ fetch_remote_table_info(char *nspname, char *relname,
 		Assert(!isnull);
 		lrel->atttyps[natt] = DatumGetObjectId(slot_getattr(slot, 2, &isnull));
 		Assert(!isnull);
-		if (DatumGetBool(slot_getattr(slot, 4, &isnull)))
+		if (DatumGetBool(slot_getattr(slot, 3, &isnull)))
 			lrel->attkeys = bms_add_member(lrel->attkeys, natt);
 
 		/* Should never happen. */
-- 
2.7.4

From 797a0e8d858b490df7a9e1526f76e49fe1e10215 Mon Sep 17 00:00:00 2001
From: Euler Taveira <[email-masked]>
Date: Fri, 9 Mar 2018 17:37:36 +0000
Subject: [PATCH 2/8] Store number of tuples in WalRcvExecResult

It seems to be a useful information while allocating memory for queries
that returns more than one row. It reduces memory allocation
for initial table synchronization.

While in it, since we have the number of columns, allocate only nfields
for cstrs instead of MaxTupleAttributeNumber.
---
 src/backend/replication/libpqwalreceiver/libpqwalreceiver.c | 9 ++++++---
 src/backend/replication/logical/tablesync.c                 | 5 ++---
 src/include/replication/walreceiver.h                       | 1 +
 3 files changed, 9 insertions(+), 6 deletions(-)

diff --git a/src/backend/replication/libpqwalreceiver/libpqwalreceiver.c b/src/backend/replication/libpqwalreceiver/libpqwalreceiver.c
index 1e1695e..2533e3a 100644
--- a/src/backend/replication/libpqwalreceiver/libpqwalreceiver.c
+++ b/src/backend/replication/libpqwalreceiver/libpqwalreceiver.c
@@ -865,6 +865,7 @@ libpqrcv_processTuples(PGresult *pgres, WalRcvExecResult *walres,
 				 errdetail("Expected %d fields, got %d fields.",
 						   nRetTypes, nfields)));
 
+	walres->ntuples = PQntuples(pgres);
 	walres->tuplestore = tuplestore_begin_heap(true, false, work_mem);
 
 	/* Create tuple descriptor corresponding to expected result. */
@@ -875,7 +876,7 @@ libpqrcv_processTuples(PGresult *pgres, WalRcvExecResult *walres,
 	attinmeta = TupleDescGetAttInMetadata(walres->tupledesc);
 
 	/* No point in doing more here if there were no tuples returned. */
-	if (PQntuples(pgres) == 0)
+	if (walres->ntuples == 0)
 		return;
 
 	/* Create temporary context for local allocations. */
@@ -884,15 +885,17 @@ libpqrcv_processTuples(PGresult *pgres, WalRcvExecResult *walres,
 									   ALLOCSET_DEFAULT_SIZES);
 
 	/* Process returned rows. */
-	for (tupn = 0; tupn < PQntuples(pgres); tupn++)
+	for (tupn = 0; tupn < walres->ntuples; tupn++)
 	{
-		char	   *cstrs[MaxTupleAttributeNumber];
+		char	**cstrs;
 
 		CHECK_FOR_INTERRUPTS();
 
 		/* Do the allocations in temporary context. */
 		oldcontext = MemoryContextSwitchTo(rowcontext);
 
+		cstrs = palloc(nfields * sizeof(char *));
+
 		/*
 		 * Fill cstrs with null-terminated strings of column values.
 		 */
diff --git a/src/backend/replication/logical/tablesync.c b/src/backend/replication/logical/tablesync.c
index f285813..e119781 100644
--- a/src/backend/replication/logical/tablesync.c
+++ b/src/backend/replication/logical/tablesync.c
@@ -720,9 +720,8 @@ fetch_remote_table_info(char *nspname, char *relname,
 				(errmsg("could not fetch table info for table \"%s.%s\": %s",
 						nspname, relname, res->err)));
 
-	/* We don't know the number of rows coming, so allocate enough space. */
-	lrel->attnames = palloc0(MaxTupleAttributeNumber * sizeof(char *));
-	lrel->atttyps = palloc0(MaxTupleAttributeNumber * sizeof(Oid));
+	lrel->attnames = palloc0(res->ntuples * sizeof(char *));
+	lrel->atttyps = palloc0(res->ntuples * sizeof(Oid));
 	lrel->attkeys = NULL;
 
 	natt = 0;
diff --git a/src/include/replication/walreceiver.h b/src/include/replication/walreceiver.h
index 5913b58..62f63f9 100644
--- a/src/include/replication/walreceiver.h
+++ b/src/include/replication/walreceiver.h
@@ -197,6 +197,7 @@ typedef struct WalRcvExecResult
 	char	   *err;
 	Tuplestorestate *tuplestore;
 	TupleDesc	tupledesc;
+	int			ntuples;
 } WalRcvExecResult;
 
 /* libpqwalreceiver hooks */
-- 
2.7.4

From 6dd5414b8dcf7b94b0901c9dfbd50d68a4c33ba1 Mon Sep 17 00:00:00 2001
From: Euler Taveira <[email-masked]>
Date: Tue, 27 Feb 2018 02:21:03 +0000
Subject: [PATCH 3/8] Refactor function create_estate_for_relation

diff --git a/src/backend/replication/logical/worker.c b/src/backend/replication/logical/worker.c
index 277da69..fa2f0ad 100644
--- a/src/backend/replication/logical/worker.c
+++ b/src/backend/replication/logical/worker.c
@@ -187,7 +187,7 @@ ensure_transaction(void)
  * This is based on similar code in copy.c
  */
 static EState *
-create_estate_for_relation(LogicalRepRelMapEntry *rel)
+create_estate_for_relation(Relation rel)
 {
 	EState	   *estate;
 	ResultRelInfo *resultRelInfo;
@@ -197,13 +197,13 @@ create_estate_for_relation(LogicalRepRelMapEntry *rel)
 
 	rte = makeNode(RangeTblEntry);
 	rte->rtekind = RTE_RELATION;
-	rte->relid = RelationGetRelid(rel->localrel);
-	rte->relkind = rel->localrel->rd_rel->relkind;
+	rte->relid = RelationGetRelid(rel);
+	rte->relkind = rel->rd_rel->relkind;
 	rte->rellockmode = AccessShareLock;
 	ExecInitRangeTable(estate, list_make1(rte));

 	estate->es_result_relations = resultRelInfo;
 	estate->es_num_result_relations = 1;
@@ -607,7 +607,7 @@ apply_handle_insert(StringInfo s)
 	}

@@ -713,7 +713,7 @@ apply_handle_update(StringInfo s)
 	check_relation_updatable(rel);
 
@@ -831,7 +831,7 @@ apply_handle_delete(StringInfo s)

From 848bc00f5e5c7b16c768eca2055a56aaef579817 Mon Sep 17 00:00:00 2001
From: Euler Taveira <[email-masked]>
Date: Wed, 24 Jan 2018 17:01:31 -0200
Subject: [PATCH 4/8] Rename a WHERE node

diff --git a/src/backend/parser/gram.y b/src/backend/parser/gram.y
index 6d23bfb..756f0dd 100644
--- a/src/backend/parser/gram.y
+++ b/src/backend/parser/gram.y
@@ -470,7 +470,7 @@ static Node *makeRecursiveViewSelect(char *relname, List *aliases, Node *query);
 %type <node>	def_arg columnElem where_clause where_or_current_clause
 				a_expr b_expr c_expr AexprConst indirection_el opt_slice_bound
 				columnref in_expr having_clause func_table xmltable array_expr
-				ExclusionWhereClause operator_def_arg
+				OptWhereClause operator_def_arg
 %type <list>	rowsfrom_item rowsfrom_list opt_col_def_list
 %type <boolean> opt_ordinality
 %type <list>	ExclusionConstraintList ExclusionConstraintElem
@@ -3742,7 +3742,7 @@ ConstraintElem:
 					$$ = (Node *)n;
 				}
 			| EXCLUDE access_method_clause '(' ExclusionConstraintList ')'
-				opt_c_include opt_definition OptConsTableSpace  ExclusionWhereClause
+				opt_c_include opt_definition OptConsTableSpace  OptWhereClause
 				ConstraintAttributeSpec
 				{
 					Constraint *n = makeNode(Constraint);
@@ -3844,7 +3844,7 @@ ExclusionConstraintElem: index_elem WITH any_operator
 			}
 		;
 
-ExclusionWhereClause:
+OptWhereClause:

From 80f710ffe42329d321e803cffc45314f35eda6c2 Mon Sep 17 00:00:00 2001
From: Euler Taveira <[email-masked]>
Date: Tue, 27 Feb 2018 04:03:13 +0000
Subject: [PATCH 5/8] Row filtering for logical replication

When you define or modify a publication you optionally can filter rows
to be published using a WHERE condition. This condition is any
expression that evaluates to boolean. Only those rows that
satisfy the WHERE condition will be sent to subscribers.
---
 doc/src/sgml/ref/alter_publication.sgml     |   9 ++-
 doc/src/sgml/ref/create_publication.sgml    |  14 +++-
 src/backend/catalog/pg_publication.c        |  46 +++++++++--
 src/backend/commands/publicationcmds.c      |  69 +++++++++++-----
 src/backend/parser/gram.y                   |  26 ++++--
 src/backend/parser/parse_agg.c              |  10 +++
 src/backend/parser/parse_expr.c             |   4 +
 src/backend/parser/parse_func.c             |   2 +
 src/backend/replication/logical/proto.c     |   2 +-
 src/backend/replication/logical/relation.c  |  13 +++
 src/backend/replication/logical/tablesync.c | 119 +++++++++++++++++++++++++---
 src/backend/replication/logical/worker.c    |   2 +-
 src/backend/replication/pgoutput/pgoutput.c |  97 ++++++++++++++++++++++-
 src/include/catalog/pg_publication.h        |   8 +-
 src/include/catalog/pg_publication_rel.h    |   8 +-
 src/include/nodes/nodes.h                   |   1 +
 src/include/nodes/parsenodes.h              |  11 ++-
 src/include/parser/parse_node.h             |   3 +-
 src/include/replication/logicalproto.h      |   2 +
 src/include/replication/logicalrelation.h   |   2 +
 src/test/regress/expected/misc_sanity.out   |   3 +-
 src/test/subscription/t/010_row_filter.pl   |  97 +++++++++++++++++++++++
 22 files changed, 492 insertions(+), 56 deletions(-)
 create mode 100644 src/test/subscription/t/010_row_filter.pl

index 534e598..5984915 100644
 
  <refsynopsisdiv>
 <synopsis>
-ALTER PUBLICATION <replaceable class="parameter">name</replaceable> ADD TABLE [ ONLY ] <replaceable class="parameter">table_name</replaceable> [ * ] [, ...]
-ALTER PUBLICATION <replaceable class="parameter">name</replaceable> SET TABLE [ ONLY ] <replaceable class="parameter">table_name</replaceable> [ * ] [, ...]
+ALTER PUBLICATION <replaceable class="parameter">name</replaceable> ADD TABLE [ ONLY ] <replaceable class="parameter">table_name</replaceable> [ * ] [ WHERE ( <replaceable class="parameter">expression</replaceable> ) ] [, ...]
+ALTER PUBLICATION <replaceable class="parameter">name</replaceable> SET TABLE [ ONLY ] <replaceable class="parameter">table_name</replaceable> [ * ] [ WHERE ( <replaceable class="parameter">expression</replaceable> ) ] [, ...]
 ALTER PUBLICATION <replaceable class="parameter">name</replaceable> DROP TABLE [ ONLY ] <replaceable class="parameter">table_name</replaceable> [ * ] [, ...]
 ALTER PUBLICATION <replaceable class="parameter">name</replaceable> SET ( <replaceable class="parameter">publication_parameter</replaceable> [= <replaceable class="parameter">value</replaceable>] [, ... ] )
 ALTER PUBLICATION <replaceable class="parameter">name</replaceable> OWNER TO { <replaceable>new_owner</replaceable> | CURRENT_USER | SESSION_USER }
@@ -91,7 +91,10 @@ ALTER PUBLICATION <replaceable class="parameter">name</replaceable> RENAME TO <r
       table name, only that table is affected.  If <literal>ONLY</literal> is not
       specified, the table and all its descendant tables (if any) are
       affected.  Optionally, <literal>*</literal> can be specified after the table
-      name to explicitly indicate that descendant tables are included.
+      name to explicitly indicate that descendant tables are included. If the
+      optional <literal>WHERE</literal> clause is specified, rows that do not
+      satisfy the <replaceable class="parameter">expression</replaceable> will
+      not be published. Note that parentheses are required around the expression.
      </para>
     </listitem>
    </varlistentry>
diff --git a/doc/src/sgml/ref/create_publication.sgml b/doc/src/sgml/ref/create_publication.sgml
index 99f87ca..d5fed30 100644
--- a/doc/src/sgml/ref/create_publication.sgml
+++ b/doc/src/sgml/ref/create_publication.sgml
@@ -22,7 +22,7 @@ PostgreSQL documentation
  <refsynopsisdiv>
 <synopsis>
 CREATE PUBLICATION <replaceable class="parameter">name</replaceable>
-    [ FOR TABLE [ ONLY ] <replaceable class="parameter">table_name</replaceable> [ * ] [, ...]
+    [ FOR TABLE [ ONLY ] <replaceable class="parameter">table_name</replaceable> [ * ] [ WHERE ( <replaceable class="parameter">expression</replaceable> ) ] [, ...]
       | FOR ALL TABLES ]
     [ WITH ( <replaceable class="parameter">publication_parameter</replaceable> [= <replaceable class="parameter">value</replaceable>] [, ... ] ) ]
 
@@ -68,7 +68,10 @@ CREATE PUBLICATION <replaceable class="parameter">name</replaceable>
       that table is added to the publication.  If <literal>ONLY</literal> is not
       specified, the table and all its descendant tables (if any) are added.
       Optionally, <literal>*</literal> can be specified after the table name to
-      explicitly indicate that descendant tables are included.
+      explicitly indicate that descendant tables are included. If the optional
+      <literal>WHERE</literal> clause is specified, rows that do not satisfy
+      the <replaceable class="parameter">expression</replaceable> will not be
+      published. Note that parentheses are required around the expression.
      </para>
 
      <para>
 
   <para>
index 3ecf6d5..f9f18a6 100644

 
@@ -173,10 +180,27 @@ publication_add_relation(Oid pubid, Relation targetrel,
 		ereport(ERROR,
 
+										AccessShareLock,
 
 	/* Form a tuple. */
 	memset(values, 0, sizeof(values));
@@ -187,6 +211,12 @@ publication_add_relation(Oid pubid, Relation targetrel,
 	values[Anum_pg_publication_rel_prrelid - 1] =
 		ObjectIdGetDatum(relid);

 	/* Insert tuple into catalog. */
@@ -203,11 +233,17 @@ publication_add_relation(Oid pubid, Relation targetrel,
 	ObjectAddressSet(referenced, RelationRelationId, relid);
 	recordDependencyOn(&myself, &referenced, DEPENDENCY_AUTO);

 
 	return myself;
 }
diff --git a/src/backend/commands/publicationcmds.c b/src/backend/commands/publicationcmds.c
index 6f7762a..d4fca7f 100644
--- a/src/backend/commands/publicationcmds.c
+++ b/src/backend/commands/publicationcmds.c
@@ -338,6 +338,27 @@ AlterPublicationTables(AlterPublicationStmt *stmt, Relation rel,

+	/*
 
 	if (stmt->tableAction == DEFELEM_ADD)
@@ -359,9 +380,9 @@ AlterPublicationTables(AlterPublicationStmt *stmt, Relation rel,
 
 			foreach(newlc, rels)
 			{
 
-				if (RelationGetRelid(newrel) == oldrelid)
+				if (RelationGetRelid(newrel->relation) == oldrelid)
 				{
 					found = true;
 					break;
@@ -370,7 +391,8 @@ AlterPublicationTables(AlterPublicationStmt *stmt, Relation rel,
 
 			if (!found)
 			{
 
 				delrels = lappend(delrels, oldrel);
@@ -493,16 +515,18 @@ OpenTableList(List *tables)
 	List	   *relids = NIL;
 	List	   *rels = NIL;
 	ListCell   *lc;
+	PublicationRelationQual	*relqual;
 
 	/*
 
 		CHECK_FOR_INTERRUPTS();
 
@@ -521,7 +545,10 @@ OpenTableList(List *tables)
 
 		if (recurse)
@@ -551,7 +578,11 @@ OpenTableList(List *tables)
 
@@ -572,10 +603,12 @@ CloseTableList(List *rels)
 
 	foreach(lc, rels)
 	{

 /*
@@ -591,13 +624,13 @@ PublicationAddTables(Oid pubid, List *rels, bool if_not_exists,
 
 	foreach(lc, rels)
 	{

 		obj = publication_add_relation(pubid, rel, if_not_exists);
 		if (stmt)
@@ -623,8 +656,8 @@ PublicationDropTables(Oid pubid, List *rels, bool missing_ok)
 
 	foreach(lc, rels)
 	{
 
@@ -636,7 +669,7 @@ PublicationDropTables(Oid pubid, List *rels, bool missing_ok)
 
index 756f0dd..edf1fef 100644
@@ -398,13 +398,13 @@ static Node *makeRecursiveViewSelect(char *relname, List *aliases, Node *query);

 %type <list>	opt_fdw_options fdw_options
@@ -9526,7 +9526,7 @@ opt_publication_for_tables:
 		;
 
 publication_for_tables:
-			FOR TABLE relation_expr_list
+			FOR TABLE publication_table_list
 				{
 					$$ = (Node *) $3;
 				}
@@ -9557,7 +9557,7 @@ AlterPublicationStmt:
 					n->options = $5;
 					$$ = (Node *)n;
 				}
-			| ALTER PUBLICATION name ADD_P TABLE relation_expr_list
+			| ALTER PUBLICATION name ADD_P TABLE publication_table_list
 				{
 					AlterPublicationStmt *n = makeNode(AlterPublicationStmt);
 					n->pubname = $3;
@@ -9565,7 +9565,7 @@ AlterPublicationStmt:
 					n->tableAction = DEFELEM_ADD;
 					$$ = (Node *)n;
 				}
-			| ALTER PUBLICATION name SET TABLE relation_expr_list
+			| ALTER PUBLICATION name SET TABLE publication_table_list
 				{
 					AlterPublicationStmt *n = makeNode(AlterPublicationStmt);
 					n->pubname = $3;
@@ -9573,7 +9573,7 @@ AlterPublicationStmt:
 					n->tableAction = DEFELEM_SET;
 					$$ = (Node *)n;
 				}
-			| ALTER PUBLICATION name DROP TABLE relation_expr_list
+			| ALTER PUBLICATION name DROP TABLE publication_table_list
 				{
 					AlterPublicationStmt *n = makeNode(AlterPublicationStmt);
 					n->pubname = $3;
@@ -9583,6 +9583,20 @@ AlterPublicationStmt:
 				}
 		;
 
+publication_table_list:
index 61727e1..128d0b9 100644
 
 			break;
+		case EXPR_KIND_PUBLICATION_WHERE:
+			if (isAgg)
+				err = _("aggregate functions are not allowed in publication WHERE expressions");
+			else
+				err = _("grouping operations are not allowed in publication WHERE expressions");
+
+			break;
 
 			/*
+			err = _("window functions are not allowed in publication WHERE expressions");
+			break;
 
 			/*
 			 * There is intentionally no default: case here, so that the
diff --git a/src/backend/parser/parse_expr.c b/src/backend/parser/parse_expr.c
index 385e54a..55cc385 100644
--- a/src/backend/parser/parse_expr.c
+++ b/src/backend/parser/parse_expr.c
@@ -1848,6 +1848,8 @@ transformSubLink(ParseState *pstate, SubLink *sublink)
 			break;
 		case EXPR_KIND_CALL_ARGUMENT:
 			err = _("cannot use subquery in CALL argument");
+		case EXPR_KIND_PUBLICATION_WHERE:
+			err = _("cannot use subquery in publication WHERE expression");
 			break;
 
 			/*
@@ -3475,6 +3477,8 @@ ParseExprKindName(ParseExprKind exprKind)
 			return "PARTITION BY";
 		case EXPR_KIND_CALL_ARGUMENT:
 			return "CALL";
+		case EXPR_KIND_PUBLICATION_WHERE:
+			return "publication expression";
 
 			/*
 			 * There is intentionally no default: case here, so that the
diff --git a/src/backend/parser/parse_func.c b/src/backend/parser/parse_func.c
index 4425715..e997ea0 100644
--- a/src/backend/parser/parse_func.c
+++ b/src/backend/parser/parse_func.c
@@ -2369,6 +2369,8 @@ check_srf_call_placement(ParseState *pstate, Node *last_srf, int location)
 			break;
 		case EXPR_KIND_CALL_ARGUMENT:
 			err = _("set-returning functions are not allowed in CALL arguments");
+		case EXPR_KIND_PUBLICATION_WHERE:
+			err = _("set-returning functions are not allowed in publication WHERE expressions");
 			break;
 
 			/*
diff --git a/src/backend/replication/logical/proto.c b/src/backend/replication/logical/proto.c
index 1945171..7ce9378 100644
--- a/src/backend/replication/logical/proto.c
+++ b/src/backend/replication/logical/proto.c
@@ -378,7 +378,7 @@ logicalrep_write_rel(StringInfo out, Relation rel)
 LogicalRepRelation *
 logicalrep_read_rel(StringInfo in)
 {
-	LogicalRepRelation *rel = palloc(sizeof(LogicalRepRelation));
+	LogicalRepRelation *rel = palloc0(sizeof(LogicalRepRelation));
 
 	rel->remoteid = pq_getmsgint(in, 4);
 
diff --git a/src/backend/replication/logical/relation.c b/src/backend/replication/logical/relation.c
index 1f20df5..8cbb394 100644
--- a/src/backend/replication/logical/relation.c
+++ b/src/backend/replication/logical/relation.c
@@ -140,6 +140,16 @@ logicalrep_relmap_free_entry(LogicalRepRelMapEntry *entry)
 	}
 	bms_free(remoterel->attkeys);
 
+	if (remoterel->nrowfilters > 0)
+	{
+		int		i;
+
+		for (i = 0; i < remoterel->nrowfilters; i++)
+			pfree(remoterel->rowfiltercond[i]);
+
+		pfree(remoterel->rowfiltercond);
+	}
+
 	if (entry->attrmap)
 		pfree(entry->attrmap);
 }
@@ -187,6 +197,9 @@ logicalrep_relmap_update(LogicalRepRelation *remoterel)
 	}
 	entry->remoterel.replident = remoterel->replident;
 	entry->remoterel.attkeys = bms_copy(remoterel->attkeys);
+	entry->remoterel.rowfiltercond = palloc(remoterel->nrowfilters * sizeof(char *));
+	for (i = 0; i < remoterel->nrowfilters; i++)
+		entry->remoterel.rowfiltercond[i] = pstrdup(remoterel->rowfiltercond[i]);
 	MemoryContextSwitchTo(oldctx);
 }
 
diff --git a/src/backend/replication/logical/tablesync.c b/src/backend/replication/logical/tablesync.c
index e119781..fa7c111 100644
--- a/src/backend/replication/logical/tablesync.c
+++ b/src/backend/replication/logical/tablesync.c
@@ -661,8 +661,14 @@ fetch_remote_table_info(char *nspname, char *relname,
 	TupleTableSlot *slot;
 	Oid			tableRow[2] = {OIDOID, CHAROID};
 	Oid			attrRow[3] = {TEXTOID, OIDOID, BOOLOID};
+	Oid			rowfilterRow[1] = {TEXTOID};
 	bool		isnull;
-	int			natt;
+	int			n;
+	ListCell   *lc;
+	bool		first;
+
+	/* Avoid trashing relation map cache */
+	memset(lrel, 0, sizeof(LogicalRepRelation));
 
 	lrel->nspname = nspname;
 	lrel->relname = relname;
@@ -724,20 +730,20 @@ fetch_remote_table_info(char *nspname, char *relname,
 	lrel->atttyps = palloc0(res->ntuples * sizeof(Oid));
 	lrel->attkeys = NULL;
 
-	natt = 0;
+	n = 0;
 	slot = MakeSingleTupleTableSlot(res->tupledesc);
 	while (tuplestore_gettupleslot(res->tuplestore, true, false, slot))
 	{
-		lrel->attnames[natt] =
+		lrel->attnames[n] =
 			TextDatumGetCString(slot_getattr(slot, 1, &isnull));
 		Assert(!isnull);
-		lrel->atttyps[natt] = DatumGetObjectId(slot_getattr(slot, 2, &isnull));
+		lrel->atttyps[n] = DatumGetObjectId(slot_getattr(slot, 2, &isnull));
 		Assert(!isnull);
 		if (DatumGetBool(slot_getattr(slot, 3, &isnull)))
-			lrel->attkeys = bms_add_member(lrel->attkeys, natt);
+			lrel->attkeys = bms_add_member(lrel->attkeys, n);
 
 		/* Should never happen. */
-		if (++natt >= MaxTupleAttributeNumber)
+		if (++n >= MaxTupleAttributeNumber)
 			elog(ERROR, "too many columns in remote table \"%s.%s\"",
 				 nspname, relname);
 
@@ -745,7 +751,54 @@ fetch_remote_table_info(char *nspname, char *relname,
 	}
 	ExecDropSingleTupleTableSlot(slot);
 
-	lrel->natts = natt;
+	lrel->natts = n;
+
+	walrcv_clear_result(res);
+
+	/* Fetch row filtering info */
+	resetStringInfo(&cmd);
+	appendStringInfo(&cmd, "SELECT pg_get_expr(prrowfilter, prrelid) FROM pg_publication p INNER JOIN pg_publication_rel pr ON (p.oid = pr.prpubid) WHERE pr.prrelid = %u AND p.pubname IN (", MyLogicalRepWorker->relid);
+
+	first = true;
+	foreach(lc, MySubscription->publications)
+	{
+		char	*pubname = strVal(lfirst(lc));
+
+		if (first)
+			first = false;
+		else
+			appendStringInfoString(&cmd, ", ");
+
+		appendStringInfoString(&cmd, quote_literal_cstr(pubname));
+	}
+	appendStringInfoChar(&cmd, ')');
+
+	res = walrcv_exec(wrconn, cmd.data, 1, rowfilterRow);
+
+	if (res->status != WALRCV_OK_TUPLES)
+		ereport(ERROR,
+				(errmsg("could not fetch row filter info for table \"%s.%s\" from publisher: %s",
+						nspname, relname, res->err)));
+
+	lrel->rowfiltercond = palloc0(res->ntuples * sizeof(char *));
+
+	n = 0;
+	slot = MakeSingleTupleTableSlot(res->tupledesc);
+	while (tuplestore_gettupleslot(res->tuplestore, true, false, slot))
+	{
+		Datum rf = slot_getattr(slot, 1, &isnull);
+
+		if (!isnull)
+		{
+			char *p = TextDatumGetCString(rf);
+			lrel->rowfiltercond[n++] = p;
+		}
+
+		ExecClearTuple(slot);
+	}
+	ExecDropSingleTupleTableSlot(slot);
+
+	lrel->nrowfilters = n;
 
 	walrcv_clear_result(res);
 	pfree(cmd.data);
@@ -778,10 +831,57 @@ copy_table(Relation rel)
 	relmapentry = logicalrep_rel_open(lrel.remoteid, NoLock);
 	Assert(rel == relmapentry->localrel);
 
+	/* list of columns for COPY */
+	attnamelist = make_copy_attnamelist(relmapentry);
+
 	/* Start copy on the publisher. */
 	initStringInfo(&cmd);
-	appendStringInfo(&cmd, "COPY %s TO STDOUT",
-					 quote_qualified_identifier(lrel.nspname, lrel.relname));
+	/*
+	 * If publication has any row filter, build a SELECT query with OR'ed row
+	 * filters for COPY.
+	 * If no row filters are available, use COPY for all
+	 * table contents.
+	 */
+	if (lrel.nrowfilters > 0)
+	{
+		ListCell   *lc;
+		bool		first;
+		int			i;
+
+		appendStringInfoString(&cmd, "COPY (SELECT ");
+		/* list of attribute names */
+		first = true;
+		foreach(lc, attnamelist)
+		{
+			char	*col = strVal(lfirst(lc));
+
+			if (first)
+				first = false;
+			else
+				appendStringInfoString(&cmd, ", ");
+			appendStringInfo(&cmd, "%s", quote_identifier(col));
+		}
+		appendStringInfo(&cmd, " FROM %s",
+						 quote_qualified_identifier(lrel.nspname, lrel.relname));
+		appendStringInfoString(&cmd, " WHERE ");
+		/* list of OR'ed filters */
+		first = true;
+		for (i = 0; i < lrel.nrowfilters; i++)
+		{
+			if (first)
+				first = false;
+			else
+				appendStringInfoString(&cmd, " OR ");
+			appendStringInfo(&cmd, "%s", lrel.rowfiltercond[i]);
+		}
+
+		appendStringInfoString(&cmd, ") TO STDOUT");
+	}
+	else
+	{
+		appendStringInfo(&cmd, "COPY %s TO STDOUT",
+						 quote_qualified_identifier(lrel.nspname, lrel.relname));
+	}
 	res = walrcv_exec(wrconn, cmd.data, 0, NULL);
 	pfree(cmd.data);
 	if (res->status != WALRCV_OK_COPY_OUT)
@@ -796,7 +896,6 @@ copy_table(Relation rel)
 	addRangeTableEntryForRelation(pstate, rel, AccessShareLock,
 								  NULL, false, false);
 
-	attnamelist = make_copy_attnamelist(relmapentry);
 	cstate = BeginCopyFrom(pstate, rel, NULL, false, copy_read_data, attnamelist, NIL);
 
 	/* Do the copy */
diff --git a/src/backend/replication/logical/worker.c b/src/backend/replication/logical/worker.c
index fa2f0ad..f28a74f 100644
--- a/src/backend/replication/logical/worker.c
+++ b/src/backend/replication/logical/worker.c
@@ -186,7 +186,7 @@ ensure_transaction(void)
  *
  * This is based on similar code in copy.c
  */
-static EState *
+EState *
 create_estate_for_relation(Relation rel)
 {
 	EState	   *estate;
diff --git a/src/backend/replication/pgoutput/pgoutput.c b/src/backend/replication/pgoutput/pgoutput.c
index 86e0951..1f4a3d3 100644
--- a/src/backend/replication/pgoutput/pgoutput.c
+++ b/src/backend/replication/pgoutput/pgoutput.c
@@ -12,13 +12,23 @@
  */
 #include "postgres.h"

 
+#include "utils/builtins.h"
 #include "utils/inval.h"
 #include "utils/int8.h"
 #include "utils/memutils.h"
@@ -58,6 +68,7 @@ typedef struct RelationSyncEntry
 	bool		schema_sent;	/* did we send the schema? */
 	bool		replicate_valid;
 	PublicationActions pubactions;
+	List		*row_filter;
 } RelationSyncEntry;
 
 /* Map used to remember which relation schemas we sent. */
@@ -329,6 +340,63 @@ pgoutput_change(LogicalDecodingContext *ctx, ReorderBufferTXN *txn,
 			Assert(false);
 	}
 
+		ExecStoreHeapTuple(new_tuple ? new_tuple : old_tuple, ecxt->ecxt_scantuple, false);
 
@@ -564,10 +632,14 @@ get_rel_sync_entry(PGOutputData *data, Oid relid)
 		 */
 		entry->pubactions.pubinsert = entry->pubactions.pubupdate =
 			entry->pubactions.pubdelete = entry->pubactions.pubtruncate = false;
+		entry->row_filter = NIL;

 			if (pub->alltables || list_member_oid(pubids, pub->oid))
 			{
@@ -577,9 +649,23 @@ get_rel_sync_entry(PGOutputData *data, Oid relid)
 				entry->pubactions.pubtruncate |= pub->pubactions.pubtruncate;
 			}
 
-				entry->pubactions.pubdelete && entry->pubactions.pubtruncate)
 
 		list_free(pubids);
@@ -654,5 +740,10 @@ rel_sync_cache_publication_cb(Datum arg, int cacheid, uint32 hashvalue)
 	 */
 	hash_seq_init(&status, RelationSyncCache);
 	while ((entry = (RelationSyncEntry *) hash_seq_search(&status)) != NULL)
+	{
 		entry->replicate_valid = false;
+		if (list_length(entry->row_filter) > 0)
+			list_free(entry->row_filter);
+		entry->row_filter = NIL;
+	}
 }
diff --git a/src/include/catalog/pg_publication.h b/src/include/catalog/pg_publication.h
index a5d5570..e78222e 100644
--- a/src/include/catalog/pg_publication.h
+++ b/src/include/catalog/pg_publication.h
@@ -76,6 +76,12 @@ typedef struct Publication
 	PublicationActions pubactions;
 } Publication;
 
@@ -84,7 +90,7 @@ extern List *GetAllTablesPublications(void);

index d97b0fe..f499253 100644
@@ -28,8 +28,12 @@
 CATALOG(pg_publication_rel,6106,PublicationRelRelationId)
 
 /* ----------------
diff --git a/src/include/nodes/nodes.h b/src/include/nodes/nodes.h
index cac6ff0..26b79d7 100644
--- a/src/include/nodes/nodes.h
+++ b/src/include/nodes/nodes.h
@@ -475,6 +475,7 @@ typedef enum NodeTag
 	T_PartitionRangeDatum,
 	T_PartitionCmd,
 	T_VacuumRelation,
+	T_PublicationTable,
 
 	/*
 	 * TAGS FOR REPLICATION GRAMMAR PARSE NODES (replnodes.h)
diff --git a/src/include/nodes/parsenodes.h b/src/include/nodes/parsenodes.h
index aa4a0db..8ac4d81 100644
--- a/src/include/nodes/parsenodes.h
+++ b/src/include/nodes/parsenodes.h
@@ -3448,12 +3448,19 @@ typedef struct AlterTSConfigurationStmt
 } AlterTSConfigurationStmt;

 	char	   *pubname;		/* Name of the publication */
 
@@ -3466,7 +3473,7 @@ typedef struct AlterPublicationStmt
 	List	   *options;		/* List of DefElem nodes */

diff --git a/src/include/replication/logicalproto.h b/src/include/replication/logicalproto.h
index 8192f79..75be2f0 100644
--- a/src/include/replication/logicalproto.h
+++ b/src/include/replication/logicalproto.h
@@ -50,6 +50,8 @@ typedef struct LogicalRepRelation
 	Oid		   *atttyps;		/* column types */
 	char		replident;		/* replica identity */
 	Bitmapset  *attkeys;		/* Bitmap of key columns */
+	char	  **rowfiltercond;	/* condition for row filtering */
+	int			nrowfilters;	/* number of row filters */
 } LogicalRepRelation;
 
 /* Type mapping info */
diff --git a/src/include/replication/logicalrelation.h b/src/include/replication/logicalrelation.h
index 73e4805..dd54295 100644
--- a/src/include/replication/logicalrelation.h
+++ b/src/include/replication/logicalrelation.h
@@ -39,4 +39,6 @@ extern void logicalrep_rel_close(LogicalRepRelMapEntry *rel,
 extern void logicalrep_typmap_update(LogicalRepTyp *remotetyp);
 extern char *logicalrep_typmap_gettypname(Oid remoteid);
 
+extern EState *create_estate_for_relation(Relation rel);
+
 #endif							/* LOGICALRELATION_H */
diff --git a/src/test/regress/expected/misc_sanity.out b/src/test/regress/expected/misc_sanity.out
index 2d3522b..62eabbd 100644
--- a/src/test/regress/expected/misc_sanity.out
+++ b/src/test/regress/expected/misc_sanity.out
@@ -105,5 +105,6 @@ ORDER BY 1, 2;
  pg_index                | indpred       | pg_node_tree
  pg_largeobject          | data          | bytea
  pg_largeobject_metadata | lomacl        | aclitem[]
-(11 rows)
+ pg_publication_rel      | prrowfilter   | pg_node_tree
+(12 rows)
 
diff --git a/src/test/subscription/t/010_row_filter.pl b/src/test/subscription/t/010_row_filter.pl
new file mode 100644
index 0000000..6c174fa
--- /dev/null
+++ b/src/test/subscription/t/010_row_filter.pl
@@ -0,0 +1,97 @@
+# Teste logical replication behavior with row filtering
+use strict;
+use warnings;
+use PostgresNode;
+use TestLib;
+use Test::More tests => 4;
+
+# create publisher node
+my $node_publisher = get_new_node('publisher');
+$node_publisher->init(allows_streaming => 'logical');
+$node_publisher->start;
+
+# create subscriber node
+my $node_subscriber = get_new_node('subscriber');
+$node_subscriber->init(allows_streaming => 'logical');
+$node_subscriber->start;
+
+# setup structure on publisher
+$node_publisher->safe_psql('postgres',
+	"CREATE TABLE tab_rowfilter_1 (a int primary key, b text)");
+$node_publisher->safe_psql('postgres',
+	"CREATE TABLE tab_rowfilter_2 (c int primary key)");
+$node_publisher->safe_psql('postgres',
+	"CREATE TABLE tab_rowfilter_3 (a int primary key, b boolean)");
+
+# setup structure on subscriber
+$node_subscriber->safe_psql('postgres',
+	"CREATE TABLE tab_rowfilter_1 (a int primary key, b text)");
+$node_subscriber->safe_psql('postgres',
+	"CREATE TABLE tab_rowfilter_2 (c int primary key)");
+$node_subscriber->safe_psql('postgres',
+	"CREATE TABLE tab_rowfilter_3 (a int primary key, b boolean)");
+
+# setup logical replication
+$node_publisher->safe_psql('postgres',
+	"CREATE PUBLICATION tap_pub_1 FOR TABLE tab_rowfilter_1 WHERE (a > 1000 AND b <> 'filtered')");
+
+my $result = $node_publisher->psql('postgres',
+	"ALTER PUBLICATION tap_pub_1 DROP TABLE tab_rowfilter_1 WHERE (a > 1000 AND b <> 'filtered')");
+is($result, 3, "syntax error for ALTER PUBLICATION DROP TABLE");
+
+$node_publisher->safe_psql('postgres',
+	"ALTER PUBLICATION tap_pub_1 ADD TABLE tab_rowfilter_2 WHERE (c % 2 = 0)");
+
+$node_publisher->safe_psql('postgres',
+	"ALTER PUBLICATION tap_pub_1 SET TABLE tab_rowfilter_1 WHERE (a > 1000 AND b <> 'filtered'), tab_rowfilter_2 WHERE (c % 2 = 0), tab_rowfilter_3");
+
+$node_publisher->safe_psql('postgres',
+	"CREATE PUBLICATION tap_pub_2 FOR TABLE tab_rowfilter_2 WHERE (c % 3 = 0)");
+
+# test row filtering
+$node_publisher->safe_psql('postgres',
+	"INSERT INTO tab_rowfilter_1 (a, b) VALUES (1, 'not replicated')");
+$node_publisher->safe_psql('postgres',
+	"INSERT INTO tab_rowfilter_1 (a, b) VALUES (1500, 'filtered')");
+$node_publisher->safe_psql('postgres',
+	"INSERT INTO tab_rowfilter_1 (a, b) VALUES (1980, 'not filtered')");
+$node_publisher->safe_psql('postgres',
+	"INSERT INTO tab_rowfilter_1 (a, b) SELECT x, 'test ' || x FROM generate_series(990,1003) x");
+$node_publisher->safe_psql('postgres',
+	"INSERT INTO tab_rowfilter_2 (c) SELECT generate_series(1, 10)");
+$node_publisher->safe_psql('postgres',
+	"INSERT INTO tab_rowfilter_3 (a, b) SELECT x, (x % 3 = 0) FROM generate_series(1, 10) x");
+
+my $publisher_connstr = $node_publisher->connstr . ' dbname=postgres';
+my $appname = 'tap_sub';
+$node_subscriber->safe_psql('postgres',
+"CREATE SUBSCRIPTION tap_sub CONNECTION '$publisher_connstr application_name=$appname' PUBLICATION tap_pub_1, tap_pub_2"
+);
+
+$node_publisher->wait_for_catchup($appname);
+
+# wait for initial table sync to finish
+my $synced_query =
+"SELECT count(1) = 0 FROM pg_subscription_rel WHERE srsubstate NOT IN ('r', 's');";
+$node_subscriber->poll_query_until('postgres', $synced_query)
+  or die "Timed out while waiting for subscriber to synchronize data";
+
+#$node_publisher->wait_for_catchup($appname);
+
+$result =
+  $node_subscriber->safe_psql('postgres', "SELECT a, b FROM tab_rowfilter_1");
+is($result, qq(1980|not filtered
+1001|test 1001
+1002|test 1002
+1003|test 1003), 'check filtered data was copied to subscriber');
+
+$result =
+  $node_subscriber->safe_psql('postgres', "SELECT count(c), min(c), max(c) FROM tab_rowfilter_2");
+is($result, qq(7|2|10), 'check filtered data was copied to subscriber');
+
+$result =
+  $node_subscriber->safe_psql('postgres', "SELECT count(a) FROM tab_rowfilter_3");
+is($result, qq(10), 'check filtered data was copied to subscriber');
+
+$node_subscriber->stop('fast');
+$node_publisher->stop('fast');
-- 
2.7.4

From 7bfdda04dbfdb7acba84296bdf15fe49bafe2d7c Mon Sep 17 00:00:00 2001
From: Euler Taveira <[email-masked]>
Date: Thu, 17 May 2018 20:52:28 +0000
Subject: [PATCH 6/8] Print publication WHERE condition in psql

---
 src/bin/psql/describe.c | 7 ++++++-
 1 file changed, 6 insertions(+), 1 deletion(-)

diff --git a/src/bin/psql/describe.c b/src/bin/psql/describe.c
index 4ca0db1..7aab8b9 100644
--- a/src/bin/psql/describe.c
+++ b/src/bin/psql/describe.c
@@ -5445,7 +5445,8 @@ describePublications(const char *pattern)
 		if (!puballtables)
 		{
 			printfPQExpBuffer(&buf,
-							  "SELECT n.nspname, c.relname\n"
+							  "SELECT n.nspname, c.relname,\n"
+							  "  pg_get_expr(pr.prrowfilter, c.oid)\n"
 							  "FROM pg_catalog.pg_class c,\n"
 							  "     pg_catalog.pg_namespace n,\n"
 							  "     pg_catalog.pg_publication_rel pr\n"
@@ -5475,6 +5476,10 @@ describePublications(const char *pattern)
 								  PQgetvalue(tabres, j, 0),
 								  PQgetvalue(tabres, j, 1));
 
+				if (!PQgetisnull(tabres, j, 2))
+					appendPQExpBuffer(&buf, "  WHERE %s",
+									PQgetvalue(tabres, j, 2));
+
 				printTableAddFooter(&cont, buf.data);
 			}
 			PQclear(tabres);
-- 
2.7.4

From be36719198f6ad3f90164510601b35118870c389 Mon Sep 17 00:00:00 2001
From: Euler Taveira <[email-masked]>
Date: Sat, 15 Sep 2018 02:52:00 +0000
Subject: [PATCH 7/8] Publication where condition support for pg_dump

---
 src/bin/pg_dump/pg_dump.c | 15 +++++++++++++--
 src/bin/pg_dump/pg_dump.h |  1 +
 2 files changed, 14 insertions(+), 2 deletions(-)

diff --git a/src/bin/pg_dump/pg_dump.c b/src/bin/pg_dump/pg_dump.c
index c8d01ed..4c63694 100644
--- a/src/bin/pg_dump/pg_dump.c
+++ b/src/bin/pg_dump/pg_dump.c
@@ -3911,6 +3911,7 @@ getPublicationTables(Archive *fout, TableInfo tblinfo[], int numTables)
 	int			i_tableoid;
 	int			i_oid;
 	int			i_pubname;
+	int			i_pubrelqual;
 	int			i,
 				j,
 				ntups;
@@ -3944,7 +3945,8 @@ getPublicationTables(Archive *fout, TableInfo tblinfo[], int numTables)
 
 		/* Get the publication membership for the table. */
 		appendPQExpBuffer(query,
-						  "SELECT pr.tableoid, pr.oid, p.pubname "
+						  "SELECT pr.tableoid, pr.oid, p.pubname, "
+						  "pg_catalog.pg_get_expr(pr.prrowfilter, pr.prrelid) AS pubrelqual "
 						  "FROM pg_publication_rel pr, pg_publication p "
 						  "WHERE pr.prrelid = '%u'"
 						  "  AND p.oid = pr.prpubid",
@@ -3965,6 +3967,7 @@ getPublicationTables(Archive *fout, TableInfo tblinfo[], int numTables)
 		i_tableoid = PQfnumber(res, "tableoid");
 		i_oid = PQfnumber(res, "oid");
 		i_pubname = PQfnumber(res, "pubname");
+		i_pubrelqual = PQfnumber(res, "pubrelqual");
 
 		pubrinfo = pg_malloc(ntups * sizeof(PublicationRelInfo));
 
@@ -3980,6 +3983,11 @@ getPublicationTables(Archive *fout, TableInfo tblinfo[], int numTables)
 			pubrinfo[j].pubname = pg_strdup(PQgetvalue(res, j, i_pubname));
 			pubrinfo[j].pubtable = tbinfo;
 
+			if (PQgetisnull(res, j, i_pubrelqual))
+				pubrinfo[j].pubrelqual = NULL;
+			else
+				pubrinfo[j].pubrelqual = pg_strdup(PQgetvalue(res, j, i_pubrelqual));
+
 			/* Decide whether we want to dump it */
 			selectDumpablePublicationTable(&(pubrinfo[j].dobj), fout);
 		}
@@ -4008,8 +4016,11 @@ dumpPublicationTable(Archive *fout, PublicationRelInfo *pubrinfo)
 
 	appendPQExpBuffer(query, "ALTER PUBLICATION %s ADD TABLE ONLY",
 					  fmtId(pubrinfo->pubname));
-	appendPQExpBuffer(query, " %s;\n",
+	appendPQExpBuffer(query, " %s",
 					  fmtQualifiedDumpable(tbinfo));
+	if (pubrinfo->pubrelqual)
+		appendPQExpBuffer(query, " WHERE %s", pubrinfo->pubrelqual);
+	appendPQExpBufferStr(query, ";\n");
 
 	/*
 	 * There is no point in creating drop query as drop query as the drop is
diff --git a/src/bin/pg_dump/pg_dump.h b/src/bin/pg_dump/pg_dump.h
index 685ad78..c2dfae6 100644
--- a/src/bin/pg_dump/pg_dump.h
+++ b/src/bin/pg_dump/pg_dump.h
@@ -609,6 +609,7 @@ typedef struct _PublicationRelInfo
 	DumpableObject dobj;
 	TableInfo  *pubtable;
 	char	   *pubname;
+	char	   *pubrelqual;
 } PublicationRelInfo;
 
 /*
-- 
2.7.4

From f1571f8b607333efe3f4a70b5dc73dd069e89573 Mon Sep 17 00:00:00 2001
From: Euler Taveira <[email-masked]>
Date: Wed, 14 Mar 2018 00:53:17 +0000
Subject: [PATCH 8/8] Debug for row filtering

---
 src/backend/commands/publicationcmds.c      | 11 +++++
 src/backend/replication/logical/tablesync.c |  1 +
 src/backend/replication/pgoutput/pgoutput.c | 66 +++++++++++++++++++++++++++++
 3 files changed, 78 insertions(+)

diff --git a/src/backend/commands/publicationcmds.c b/src/backend/commands/publicationcmds.c
index d4fca7f..27f1102 100644
--- a/src/backend/commands/publicationcmds.c
+++ b/src/backend/commands/publicationcmds.c
@@ -327,6 +327,7 @@ AlterPublicationTables(AlterPublicationStmt *stmt, Relation rel,
 	Oid			pubid = HeapTupleGetOid(tup);
 	List	   *rels = NIL;
 	Form_pg_publication pubform = (Form_pg_publication) GETSTRUCT(tup);
+	ListCell	*xpto;
 
 	/* Check that user is allowed to manipulate the publication tables. */
 	if (pubform->puballtables)
@@ -338,6 +339,16 @@ AlterPublicationTables(AlterPublicationStmt *stmt, Relation rel,

+	foreach(xpto, stmt->tables)
+	{
+		PublicationTable *t = lfirst(xpto);
+
+		if (t->whereClause == NULL)
+			elog(DEBUG3, "publication \"%s\" has no WHERE clause", NameStr(pubform->pubname));
+		else
+			elog(DEBUG3, "publication \"%s\" has WHERE clause", NameStr(pubform->pubname));
+	}
+
 	/*
 	 * ALTER PUBLICATION ... DROP TABLE cannot contain a WHERE clause.  Use
 	 * publication_table_list node (that accepts a WHERE clause) but forbid the
diff --git a/src/backend/replication/logical/tablesync.c b/src/backend/replication/logical/tablesync.c
index fa7c111..e0eb73c 100644
--- a/src/backend/replication/logical/tablesync.c
+++ b/src/backend/replication/logical/tablesync.c
@@ -882,6 +882,7 @@ copy_table(Relation rel)
 		appendStringInfo(&cmd, "COPY %s TO STDOUT",
 						 quote_qualified_identifier(lrel.nspname, lrel.relname));
 	}
+	elog(DEBUG2, "COPY for initial synchronization: %s", cmd.data);
 	res = walrcv_exec(wrconn, cmd.data, 0, NULL);
 	pfree(cmd.data);
 	if (res->status != WALRCV_OK_COPY_OUT)
diff --git a/src/backend/replication/pgoutput/pgoutput.c b/src/backend/replication/pgoutput/pgoutput.c
index 1f4a3d3..c7f0e32 100644
--- a/src/backend/replication/pgoutput/pgoutput.c
+++ b/src/backend/replication/pgoutput/pgoutput.c
@@ -31,6 +31,7 @@
 #include "utils/builtins.h"
 #include "utils/inval.h"
 #include "utils/int8.h"
+#include "utils/lsyscache.h"
 #include "utils/memutils.h"
 #include "utils/syscache.h"
 #include "utils/varlena.h"
@@ -316,6 +317,10 @@ pgoutput_change(LogicalDecodingContext *ctx, ReorderBufferTXN *txn,
 	MemoryContext old;
 	RelationSyncEntry *relentry;
 
+	Form_pg_class	class_form;
+	char			*schemaname;
+	char			*tablename;
+
 	if (!is_publishable_relation(relation))
 		return;
 
@@ -340,6 +345,17 @@ pgoutput_change(LogicalDecodingContext *ctx, ReorderBufferTXN *txn,
 			Assert(false);
 	}
 
+	class_form = RelationGetForm(relation);
+	schemaname = get_namespace_name(class_form->relnamespace);
+	tablename = NameStr(class_form->relname);
+
+	if (change->action == REORDER_BUFFER_CHANGE_INSERT)
+		elog(DEBUG1, "INSERT \"%s\".\"%s\" txid: %u", schemaname, tablename, txn->xid);
+	else if (change->action == REORDER_BUFFER_CHANGE_UPDATE)
+		elog(DEBUG1, "UPDATE \"%s\".\"%s\" txid: %u", schemaname, tablename, txn->xid);
+	else if (change->action == REORDER_BUFFER_CHANGE_DELETE)
+		elog(DEBUG1, "DELETE \"%s\".\"%s\" txid: %u", schemaname, tablename, txn->xid);
+
 	/* ... then check row filter */
 	if (list_length(relentry->row_filter) > 0)
 	{
@@ -356,6 +372,42 @@ pgoutput_change(LogicalDecodingContext *ctx, ReorderBufferTXN *txn,
 		tupdesc = RelationGetDescr(relation);
 		estate = create_estate_for_relation(relation);
 
+#ifdef	_NOT_USED
+		if (old_tuple)
+		{
+			int i;
+
+			for (i = 0; i < tupdesc->natts; i++)
+			{
+				Form_pg_attribute	attr;
+				HeapTuple			type_tuple;
+				Oid					typoutput;
+				bool				typisvarlena;
+				bool				isnull;
+				Datum				val;
+				char				*outputstr = NULL;
+
+				attr = TupleDescAttr(tupdesc, i);
+
+				/* Figure out type name */
+				type_tuple = SearchSysCache1(TYPEOID, ObjectIdGetDatum(attr->atttypid));
+				if (HeapTupleIsValid(type_tuple))
+				{
+					/* Get information needed for printing values of a type */
+					getTypeOutputInfo(attr->atttypid, &typoutput, &typisvarlena);
+
+					val = heap_getattr(old_tuple, i + 1, tupdesc, &isnull);
+					if (!isnull)
+					{
+						outputstr = OidOutputFunctionCall(typoutput, val);
+						elog(DEBUG2, "row filter: REPLICA IDENTITY %s: %s", NameStr(attr->attname), outputstr);
+						pfree(outputstr);
+					}
+				}
+			}
+		}
+#endif
+
 		/* prepare context per tuple */
 		ecxt = GetPerTupleExprContext(estate);
 		oldcxt = MemoryContextSwitchTo(estate->es_query_cxt);
@@ -371,6 +423,7 @@ pgoutput_change(LogicalDecodingContext *ctx, ReorderBufferTXN *txn,
 			Oid			expr_type;
 			Datum		res;
 			bool		isnull;
+			char		*s = NULL;
 
 			row_filter = (Node *) lfirst(lc);
 
@@ -381,14 +434,24 @@ pgoutput_change(LogicalDecodingContext *ctx, ReorderBufferTXN *txn,
 			expr_state = ExecInitExpr(expr, NULL);
 			res = ExecEvalExpr(expr_state, ecxt, &isnull);
 
+			elog(DEBUG3, "row filter: result: %s ; isnull: %s", (DatumGetBool(res)) ? "true" : "false", (isnull) ? "true" : "false");
+
 			/* if tuple does not match row filter, bail out */
 			if (!DatumGetBool(res) || isnull)
 			{
+				s = TextDatumGetCString(DirectFunctionCall2(pg_get_expr, CStringGetTextDatum(nodeToString(row_filter)), ObjectIdGetDatum(relentry->relid)));
+				elog(DEBUG2, "row filter \"%s\" was not matched", s);
+				pfree(s);
+
 				MemoryContextSwitchTo(oldcxt);
 				ExecDropSingleTupleTableSlot(ecxt->ecxt_scantuple);
 				FreeExecutorState(estate);
 				return;
 			}
+
+			s = TextDatumGetCString(DirectFunctionCall2(pg_get_expr, CStringGetTextDatum(nodeToString(row_filter)), ObjectIdGetDatum(relentry->relid)));
+			elog(DEBUG2, "row filter \"%s\" was matched", s);
+			pfree(s);
 		}
 
 		MemoryContextSwitchTo(oldcxt);
@@ -659,9 +722,12 @@ get_rel_sync_entry(PGOutputData *data, Oid relid)
 				{
 					MemoryContext oldctx = MemoryContextSwitchTo(CacheMemoryContext);
 					char	*s = TextDatumGetCString(rf_datum);
+					char	*t = TextDatumGetCString(DirectFunctionCall2(pg_get_expr, rf_datum, ObjectIdGetDatum(entry->relid)));
 					Node	*rf_node = stringToNode(s);
 					entry->row_filter = lappend(entry->row_filter, rf_node);
 					MemoryContextSwitchTo(oldctx);
+
+					elog(DEBUG2, "row filter \"%s\" found for publication \"%s\" and relation \"%s\"", t, pub->name, get_rel_name(relid));
 				}
 
 				ReleaseSysCache(rf_tuple);
-- 
2.7.4

---

From: Erik Rijkers <[email-masked]>
Date: Thu, 01 Nov 2018 08:56:48 +0100

>> The attached patches add support for filtering rows in the publisher.

I ran pgbench-over-logical-replication with a WHERE-clause and could not 
get this to do a correct replication.  Below is the output of the 
attached test program.

$ ./logrep_rowfilter.sh
-- 
/home/aardvark/pg_stuff/pg_installations/pgsql.logrep_rowfilter/bin.fast/initdb 
--pgdata=/tmp/cascade/instance1/data --encoding=UTF8 --pwfile=/tmp/bugs
-- 
/home/aardvark/pg_stuff/pg_installations/pgsql.logrep_rowfilter/bin.fast/initdb 
--pgdata=/tmp/cascade/instance2/data --encoding=UTF8 --pwfile=/tmp/bugs
-- 
/home/aardvark/pg_stuff/pg_installations/pgsql.logrep_rowfilter/bin.fast/initdb 
--pgdata=/tmp/cascade/instance3/data --encoding=UTF8 --pwfile=/tmp/bugs
sleep 3s
dropping old tables...
creating tables...
generating data...
100000 of 100000 tuples (100%) done (elapsed 0.09 s, remaining 0.00 s)
vacuuming...
creating primary keys...
done.
create publication pub_6515_to_6516;
alter publication pub_6515_to_6516 add table pgbench_accounts where (aid 
between 40000 and 60000-1) ; --> where 1
alter publication pub_6515_to_6516 add table pgbench_branches;
alter publication pub_6515_to_6516 add table pgbench_tellers;
alter publication pub_6515_to_6516 add table pgbench_history;
create publication pub_6516_to_6517;
alter publication pub_6516_to_6517 add table pgbench_accounts ; -- where 
(aid between 40000 and 60000-1) ; --> where 2
alter publication pub_6516_to_6517 add table pgbench_branches;
alter publication pub_6516_to_6517 add table pgbench_tellers;
alter publication pub_6516_to_6517 add table pgbench_history;

create subscription pub_6516_from_6515 connection 'port=6515 
application_name=rowfilter'
        publication pub_6515_to_6516 with(enabled=false);
alter subscription pub_6516_from_6515 enable;
create subscription pub_6517_from_6516 connection 'port=6516 
application_name=rowfilter'
        publication pub_6516_to_6517 with(enabled=false);
alter subscription pub_6517_from_6516 enable;
-- pgbench -p 6515 -c 16 -j 8 -T 5 -n postgres    #  scale 1
transaction type: <builtin: TPC-B (sort of)>
scaling factor: 1
query mode: simple
number of clients: 16
number of threads: 8
duration: 5 s
number of transactions actually processed: 80
latency average = 1178.106 ms
tps = 13.581120 (including connections establishing)
tps = 13.597443 (excluding connections establishing)

        accounts  branches   tellers   history
        --------- --------- --------- ---------
6515   6546b1f0f 2d328ed28 7406473b0 7c1351523    e8c07347b
6516   6546b1f0f 2d328ed28 d41d8cd98 d41d8cd98    e7235f541
6517   f7c0791c8 d9c63e471 d41d8cd98 d41d8cd98    30892eea1   NOK

6515   6546b1f0f 2d328ed28 7406473b0 7c1351523    e8c07347b
6516   6546b1f0f 2d328ed28 7406473b0 5a54cf7c5    191ae1af3
6517   6546b1f0f 2d328ed28 7406473b0 5a54cf7c5    191ae1af3   NOK

6515   6546b1f0f 2d328ed28 7406473b0 7c1351523    e8c07347b
6516   6546b1f0f 2d328ed28 7406473b0 5a54cf7c5    191ae1af3
6517   6546b1f0f 2d328ed28 7406473b0 5a54cf7c5    191ae1af3   NOK

[...]

I let that run for 10 minutes or so but that pgbench_history table 
md5-values (of ports 6516 and 6517) do not change anymore, which shows 
that it is and remains different from the original pgbench_history table 
in 6515.

When there is a where-clause this goes *always* wrong.

Without a where-clause all logical replication tests were OK.  Perhaps 
the error is not in our patch but something in logical replication.

Attached is the test program (will need some tweaking of PATHs, 
PG-variables (PGPASSFILE) etc).  This is the same program I used in 
march when you first posted a version of this patch alhough the error is 
different.

thanks,

Erik Rijkers

---

From: Erik Rijkers <[email-masked]>
Date: Thu, 01 Nov 2018 09:30:36 +0100

On 2018-11-01 08:56, Erik Rijkers wrote:
> I ran pgbench-over-logical-replication with a WHERE-clause and could
> not get this to do a correct replication.  Below is the output of the
> attached test program.
> 
> 
> $ ./logrep_rowfilter.sh

I have noticed that the failure to replicate correctly can be avoided by 
putting a wait state of (on my machine) at least 3 seconds between the 
setting up of the subscription and the start of pgbench.  See the bash 
program I attached in my previous mail.  The bug can be avoided by a 
'sleep 5' just before the start of the actual pgbench run.

So it seems this bug is due to some timing error in your patch (or 
possibly in logical replication itself).

Erik Rijkers

---

From: Euler Taveira <[email-masked]>
Date: Thu, 1 Nov 2018 22:59:36 -0300

> > not get this to do a correct replication.  Below is the output of the
Erik, thanks for testing.

> possibly in logical replication itself).
It is a bug in the new synchronization code. I'm doing some code
cleanup/review and will post a new patchset after I finish it. If you
want to give it a try again, apply the following patch.

diff --git a/src/backend/replication/logical/tablesync.c
b/src/backend/replication/logical/tablesync.c
index e0eb73c..4797e0b 100644
--- a/src/backend/replication/logical/tablesync.c
+++ b/src/backend/replication/logical/tablesync.c
@@ -757,7 +757,7 @@ fetch_remote_table_info(char *nspname, char *relname,

        /* Fetch row filtering info */
        resetStringInfo(&cmd);
-       appendStringInfo(&cmd, "SELECT pg_get_expr(prrowfilter,
prrelid) FROM pg_publication p INNER JOIN pg_publication_rel pr ON
(p.oid = pr.prpubid) WHERE pr.prrelid = %u AND p.pubname IN (",
MyLogicalRepWorker->relid);
+       appendStringInfo(&cmd, "SELECT pg_get_expr(prrowfilter,
prrelid) FROM pg_publication p INNER JOIN pg_publication_rel pr ON
(p.oid = pr.prpubid) WHERE pr.prrelid = %u AND p.pubname IN (",
lrel->remoteid);

---

From: Erik Rijkers <[email-masked]>
Date: Fri, 02 Nov 2018 07:55:05 +0100

> +++ b/src/backend/replication/logical/tablesync.c

That does indeed fix it.

Thank you,

Erik Rijkers

---

From: Hironobu SUZUKI <[email-masked]>
Date: Wed, 21 Nov 2018 05:51:34 +0000

Hi,

I reviewed your patches and I found a bug when I tested ALTER 
PUBLICATION statement.

In short, ALTER PUBLICATION SET with a WHERE clause does not applied new 
WHERE clause.

I describe the outline of the test I did and my conclusion.

[TEST]
I show the test case I tried in below.

(1)Publisher and Subscriber

I executed each statement on the publisher and the subscriber.

```
testdb=# CREATE PUBLICATION pub_testdb_t FOR TABLE t WHERE (id > 10);
CREATE PUBLICATION
```

```
testdb=# CREATE SUBSCRIPTION sub_testdb_t CONNECTION 'dbname=testdb 
port=5432 user=postgres' PUBLICATION pub_testdb_t;
NOTICE:  created replication slot "sub_testdb_t" on publisher
CREATE SUBSCRIPTION
```

(2)Publisher

I executed these statements shown below.

testdb=# INSERT INTO t VALUES (1,1);
INSERT 0 1
testdb=# INSERT INTO t VALUES (11,11);
INSERT 0 1

(3)Subscriber

I confirmed that the CREATE PUBLICATION statement worked well.

```
testdb=# SELECT * FROM t;
  id | data
----+------
  11 |   11
(1 row)
```

(4)Publisher
After that, I executed ALTER PUBLICATION with a WHERE clause and 
inserted a new row.

```
testdb=# ALTER  PUBLICATION pub_testdb_t SET TABLE t WHERE (id > 5);
ALTER PUBLICATION

testdb=# INSERT INTO t VALUES (7,7);
INSERT 0 1

testdb=# SELECT * FROM t;
  id | data
----+------
   1 |    1
  11 |   11
   7 |    7
(3 rows)
```

(5)Subscriber
I confirmed that the change of WHERE clause set by ALTER PUBLICATION 
statement was ignored.

```
testdb=# SELECT * FROM t;
  id | data
----+------
  11 |   11
(1 row)
```

[Conclusion]
I think AlterPublicationTables()@publicationcmds.c has a bug.

In the foreach(oldlc, oldrelids) loop, oldrel must be appended to 
delrels if oldrel or newrel has a WHERE clause. However, the current 
implementation does not, therefore, old WHERE clause is not deleted and 
the new WHERE clause is ignored.

This is my speculation. It may not be correct, but , at least, it is a 
fact that ALTER PUBLICATION with a WHERE clause is not functioned in my 
environment and my operation described in above.

Best regards,

---

From: Petr Jelinek <[email-masked]>
Date: Fri, 23 Nov 2018 00:03:27 +0100

Hi,

I think there are two main topics that still need to be discussed about
this patch.

Firstly, I am not sure if it's wise to allow UDFs in the filter clause
for the table. The reason for that is that we can't record all necessary
dependencies there because the functions are black box for parser. That
means if somebody drops object that an UDF used in replication filter
depends on, that function will start failing. But unlike for user
sessions it will start failing during decoding (well processing in
output plugin). And that's not recoverable by reading the missing
object, the only way to get out of that is either to move slot forward
which means losing part of replication stream and need for manual resync
or full rebuild of replication. Neither of which are good IMHO.

Secondly, do we want to at least notify user on filters (or maybe even
disallow them) with combination of action + column where column value
will not be logged? I mean for example we do this when processing the

> +		ExecStoreHeapTuple(new_tuple ? new_tuple : old_tuple, ecxt->ecxt_scantuple, false);

But if user has expression on column which is not part of replica
identity that expression will always return NULL for DELETEs because
only replica identity is logged with actual values and everything else
in NULL in old_tuple. So if publication replicates deletes we should
check for this somehow.

Btw about code (you already fixed the wrong reloid in sync so skipping
that).

> +		cstrs = palloc(nfields * sizeof(char *));

Not really sure that this is actually worth it given that we have to
allocate and free this in a loop now while before it was just sitting on
a stack.

> +		entry->row_filter = NIL;

Won't this leak memory? The list_free only frees the list cells, but not
the nodes you stored there before.

Also I think we should document here that the expression is run with the
session environment of the replication connection (so that it's more
obvious that things like CURRENT_USER will not return user which changed
tuple but the replication user).

It would be nice if 0006 had regression test and 0007 TAP test.

-- 
  Petr Jelinek                  http://www.2ndQuadrant.com/
  PostgreSQL Development, 24x7 Support, Training & Services

---

From: Stephen Frost <[email-masked]>
Date: Thu, 22 Nov 2018 21:02:35 -0500

Greetings,

> IIRC implementation is similar to RLS expressions. I'll check all of

Given the similarity to RLS and the nearby discussion about allowing
non-superusers to create subscriptions, and probably publications later,
I wonder if we shouldn't be somehow associating this with RLS policies
instead of having the publication filtering be entirely independent..

Thanks!

Stephen

---

From: Petr Jelinek <[email-masked]>
Date: Fri, 23 Nov 2018 15:40:18 +0100

> instead of having the publication filtering be entirely independent..
I do see the appeal here, if you consider logical replication to be a
streaming select it probably applies well.

But given that this is happening inside output plugin which does not
have full executor setup and has catalog-only snapshot I am not sure how
feasible it is to try to merge these two things. As per my previous
email it's possible that we'll have to be stricter about what we allow
in expressions here.

The other issue with merging this is that the use-case for filtering out
the data in logical replication is not necessarily about security, but
often about sending only relevant data. So it makes sense to have filter
on publication without RLS enabled on table and if we'd force that, we'd
limit usefulness of this feature.

We definitely want to eventually create subscriptions as non-superuser
but that has zero effect on this as everything here is happening on
different server than where subscription lives (we already allow
creation of publications with just CREATE privilege on database and
ownership of the table).

---

From: Euler Taveira <[email-masked]>
Date: Fri, 23 Nov 2018 12:17:12 -0300

> email it's possible that we'll have to be stricter about what we allow
This feature should be as simple as possible. I don't want to
introduce a huge overhead just for filtering some data. Data sharding
generally uses simple expressions.

> limit usefulness of this feature.
Use the same infrastructure as RLS could be a good idea but use RLS
for row filtering is not. RLS is complex.

---

From: Euler Taveira <[email-masked]>
Date: Fri, 23 Nov 2018 13:15:08 -0300

> or full rebuild of replication. Neither of which are good IMHO.
It is a foot gun but there are several ways to do bad things in
postgres. CREATE PUBLICATION is restricted to superusers and role with
CREATE privilege in current database. AFAICS a role with CREATE
privilege cannot drop objects whose owner is not himself. I wouldn't
like to disallow UDFs in row filtering expressions just because
someone doesn't set permissions correctly. Do you have any other case
in mind?

> > +             ExecStoreHeapTuple(new_tuple ? new_tuple : old_tuple, ecxt->ecxt_scantuple, false);
We could emit a LOG message. That could possibly be an option but it
could be too complex for the first version.

> in NULL in old_tuple. So if publication replicates deletes we should
In this case, we should document this behavior. That is a recurring
question in wal2json issues. Besides that we should explain that
UPDATE/DELETE tuples doesn't log all columns (people think the
behavior is equivalent to triggers; it is not unless you set REPLICA
IDENTITY FULL).

> allocate and free this in a loop now while before it was just sitting on
That is a experimentation code that should be in a separate patch.
Don't you think low memory use is a good goal? I also think that
MaxTupleAttributeNumber is an extreme value. I didn't some preliminary
tests and didn't notice overheads. I'll leave these modifications in a
separate patch.

> the nodes you stored there before.
Good catch. It should be list_free_deep.

> tuple but the replication user).
Sure.

> It would be nice if 0006 had regression test and 0007 TAP test.
Sure.

Besides the problem presented by Hironobu-san, I'm doing some cleanup
and improving docs. I also forget to declare pg_publication_rel TOAST
table.

Thanks for your review.

---

From: Alvaro Herrera <[email-masked]>
Date: Fri, 23 Nov 2018 13:19:33 -0300

> Good catch. It should be list_free_deep.

Actually, if the nodes have more structure (say you palloc one list
item, but that list item also contains pointers to a Node) then a
list_free_deep won't be enough either.  I'd suggest to create a bespoke
memory context, which you can delete afterwards.

-- 
Álvaro Herrera                https://www.2ndQuadrant.com/
PostgreSQL Development, 24x7 Support, Remote DBA, Training & Services

---

From: David Fetter <[email-masked]>
Date: Fri, 23 Nov 2018 17:39:14 +0100

On Fri, Nov 23, 2018 at 12:03:27AM +0100, Petr Jelinek wrote:
> dependencies there because the functions are black box for parser.

Some UDFs are not a black box for the parser, namely ones written in
SQL. Would it make sense at least not to foreclose the non-(black box)
option?

Best,
David.

Remember to vote!

---

From: Petr Jelinek <[email-masked]>
Date: Fri, 23 Nov 2018 18:53:42 +0100

> SQL. Would it make sense at least not to foreclose the non-(black box)

Yeah inlinable SQL functions should be fine, we just need the ability to
extract dependencies.

---

From: Petr Jelinek <[email-masked]>
Date: Fri, 23 Nov 2018 18:54:58 +0100

> someone doesn't set permissions correctly. Do you have any other case

I don't think this has anything to do with security. Stupid example:

user1: CREATE EXTENSION citext;

user2: CREATE FUNCTION myfilter(col1 text, col2 text) returns boolean
language plpgsql as
$$BEGIN
RETURN col1::citext = col2::citext;
END;$$

user2: ALTER PUBLICATION mypub ADD TABLE mytab WHERE (myfilter(a,b));

[... replication happening ...]

user1: DROP EXTENSION citext;

And now replication is broken and unrecoverable without data loss.
Recreating extension will not help because the changes happening in
meantime will not see it in the historical snapshot.

I don't think it's okay to do completely nothing about this.

> could be too complex for the first version.

Well, it needs walker which extracts Vars from the expression and checks
them against replica identity columns. We already have a way to fetch
replica identity columns and the walker could be something like
simplified version of the find_expr_references_walker used by the
recordDependencyOnSingleRelExpr (I don't think there is anything ready
made already).

> tests and didn't notice overheads. I'll leave these modifications in a

It's static memory and it's a few KB of it (it's just single array of
pointers, not array of data, and does not depend on the number of rows).
Palloc will definitely need more CPU cycles.

---

From: Fabrízio de Royes Mello <[email-masked]>
Date: Fri, 23 Nov 2018 16:05:21 -0200

> I don't think it's okay to do completely nothing about this.

If carefully documented I see no problem with it... we already have an
analogous problem with functional indexes.

Regards,

--
   Fabrízio de Royes Mello         Timbira - http://www.timbira.com.br/
   PostgreSQL: Consultoria, Desenvolvimento, Suporte 24x7 e Treinamento

---

From: Petr Jelinek <[email-masked]>
Date: Fri, 23 Nov 2018 19:13:53 +0100

On 23/11/2018 19:05, Fabrízio de Royes Mello wrote:
> On Fri, Nov 23, 2018 at 3:55 PM Petr Jelinek
> <[email-masked] <mailto:[email-masked]>> wrote:
>>
>> > <[email-masked] <mailto:[email-masked]>>
> escreveu:

The difference is that with functional indexes you can recreate the
missing object and everything is okay again. With logical replication
recreating the object will not help.

---

From: Fabrízio de Royes Mello <[email-masked]>
Date: Fri, 23 Nov 2018 16:29:16 -0200

> recreating the object will not help.

In this case with logical replication you should rsync the object. That is
the price of misunderstanding / bad use of the new feature.

As usual, there are no free beer ;-)

Regards,

---

From: Stephen Frost <[email-masked]>
Date: Fri, 23 Nov 2018 14:03:30 -0500

Greetings,

> As usual, there are no free beer ;-)

There's also certainly no shortage of other ways to break logical
replication, including ways that would also be hard to recover from
today other than doing a full resync.

What that seems to indicate, to me at least, is that it'd be awful nice
to have a way to resync the data which doesn't necessairly involve
transferring all of it over again.

Of course, it'd be nice if we could track those dependencies too,
but that's yet another thing.

In short, I'm not sure that I agree with the idea that we shouldn't
allow this and instead I'd rather we realize it and put the logical
replication into some kind of an error state that requires a resync.

Thanks!

Stephen

---

From: Petr Jelinek <[email-masked]>
Date: Fri, 23 Nov 2018 20:14:12 +0100

On 23/11/2018 19:29, Fabrízio de Royes Mello wrote:
> 
> On Fri, Nov 23, 2018 at 4:13 PM Petr Jelinek
> In this case with logical replication you should rsync the object. That
> is the price of misunderstanding / bad use of the new feature.
> 

Yeah but you have to resync whole subscription, not just single table
(removing table from the publication will also not help), that's pretty
severe punishment. What if you have triggers downstream that do
calculations or logging which you can't recover by simply rebuilding
replica? I think it's better to err on the side of no data loss.

We could also try to figure out a way to recover from this that does not
require resync, ie perhaps we could somehow temporarily force evaluation
of the expression to have current snapshot.

---

From: Tomas Vondra <[email-masked]>
Date: Fri, 23 Nov 2018 21:00:46 +0100

> today other than doing a full resync.

Sure, but that seems more like an argument against creating additional
ones (and for preventing those that already exist). I'm not sure this
particular feature is where we should draw the line, though.

> What that seems to indicate, to me at least, is that it'd be awful
> nice to have a way to resync the data which doesn't necessairly
> involve transferring all of it over again.
> 

Yep, that seems like a good idea in general. Both here and for
functional indexes (although I suppose sure is a technical reason why it
wasn't implemented right away for them).

> replication into some kind of an error state that requires a resync.

That would still mean a need to resync the data to recover, so I'm not
sure it's really an improvement. And I suppose it'd require tracking the
dependencies, because how else would you mark the subscription as
requiring a resync? At which point we could decline the DROP without a
CASCADE, just like we do elsewhere, no?

regards

-- 
Tomas Vondra                  http://www.2ndQuadrant.com
PostgreSQL Development, 24x7 Support, Remote DBA, Training & Services

---

From: Tomas Vondra <[email-masked]>
Date: Fri, 23 Nov 2018 21:02:15 +0100

> replica? I think it's better to err on the side of no data loss.

Yeah, having to resync everything because you accidentally dropped a
function is quite annoying. Of course, you should notice that while
testing the upgrade in a testing environment, but still ...

> of the expression to have current snapshot.

That seems like huge a can of worms ...

cheers

---

From: Stephen Frost <[email-masked]>
Date: Fri, 14 Dec 2018 10:38:36 -0500

Greetings,

> email it's possible that we'll have to be stricter about what we allow

I can certainly understand the concern about trying to combine the
implementation of this with that of RLS; perhaps that isn't a good fit
due to the additional constraints put on logical decoding.

That said, I still think it might make sense to consider these filters
for logical decoding to be policies and, ideally, to allow users to use
the same policy for both.

In the end, the idea of having to build a single large and complex
'create publication' command which has a bunch of tables, each with
their own filter clauses, just strikes me as pretty painful.

> limit usefulness of this feature.

I definitely have a serious problem if we are going to say that you
can't use this filtering for security-sensitive cases.

> creation of publications with just CREATE privilege on database and

What I wasn't clear about above was the idea that we might allow a user
other than the table owner to publish a given table, but that such a
publication should certanily only be allowed to include the rows which
that user has access to- as regulated by RLS.  If the RLS policy is too
complex to allow that then I would think we'd simply throw an error at
the create publication time and the would-be publisher would need to
figure that out with the table owner.

I'll admit that this might seem like a stretch, but what happens today?
Today, people write cronjobs to try to sync between tables with FDWs and
you don't need to own a table to use it as the target of a foreign
table.

I do think that we'll need to have some additional privileges around who
is allowed to create publications, I'm not entirely thrilled with that
being combined with the ability to create schemas; the two seem quite
different to me.

> generally uses simple expressions.

RLS often uses simple filters too.

> for row filtering is not. RLS is complex.

Right, this was along the lines I was thinking of- using the
infrastructure and the policy system, in particular.

Thanks!

Stephen

---

From: Stephen Frost <[email-masked]>
Date: Fri, 14 Dec 2018 10:56:45 -0500

Greetings,

> particular feature is where we should draw the line, though.

I was actually going in the other direction- we should allow it because
advanced users may know what they're doing better than we do and we
shouldn't prevent things just because they might be misused or
misunderstood by a user.

> wasn't implemented right away for them).

We don't track function dependencies in general and I could certainly
see cases where you really wouldn't want to do so, at least not in the
same way that we track FKs or similar.  I do wonder if maybe we didn't
track function dependencies because we didn't (yet) have create or
replace function and that now we should.  We don't track dependencies
inside a function either though.

> CASCADE, just like we do elsewhere, no?

I was actually thinking more along the lines of just simply marking the
publication/subscription as being in a 'failed' state when a failure
actually happens, and maybe even at that point basically throwing away
everything except the shell of the publication/subscription (so the user
can see that it failed and come in and properly drop it); I'm thinking
about this as perhaps similar to a transaction being aborted.

Thanks!

Stephen

---

From: Petr Jelinek <[email-masked]>
Date: Sat, 15 Dec 2018 15:13:41 +0100

> the same policy for both.

I am not against that as long as it's possible to have policy for
logical replication without having it for RLS and vice versa.

I also wonder if policies are flexible enough to allow for specifying
OLD and NEW - the replication filtering deals with DML, not with what's
visible, it might very well depend on differences between these (that's
something the current patch is missing as well BTW).

> can't use this filtering for security-sensitive cases.

I am saying it should not be tied to only security sensitive cases,
because it has use cases that have nothing to do with security (ie, I
don't want this to depend on RLS being enabled for a table).

> figure that out with the table owner.

My opinion is that this is useful, but not necessarily something v1
patch needs to solve. Having too many publications and subscriptions to
various places is not currently practical anyway due to decoding
duplicating all the work for every connection.

> infrastructure and the policy system, in particular.

Yeah that part is definitely worth investigating.

---

From: Petr Jelinek <[email-masked]>
Date: Sat, 15 Dec 2018 15:23:39 +0100

> shouldn't prevent things just because they might be misused or

That's all good, but we need good escape hatch for when things go south
and we don't have it and IMHO it's not as easy to have one as you might
think.

That's why I would do the simple and safe way first before allowing
more, otherwise we'll be discussing this for next couple of PG versions.

> inside a function either though.

Yeah we can't always have dependencies, it would break some perfectly
valid usage scenarios. Also it's not exactly clear to me how we'd track
dependencies of say plpython function...

> about this as perhaps similar to a transaction being aborted.

There are several problems with that. First this happens in historic
snapshot which can't write and on top of that we are in the middle of
error processing so we have our hands tied a bit, it's definitely going
to need bit of creative thinking to do this.

Second, and that's more soft issue (which is probably harder to solve)
what do we do with the slot and subscription. There is one failed
publication, but the subscription may be subscribed to 20 of them, do we
kill the whole subscription because of single failed publication? If we
don't do we continue replicating like nothing has happened but with data
in the failed publication missing (which can be considered data
loss/corruption from the view of user). If we stop replication, do we
clean the slot so that we don't keep back wal/catalog xmin forever
(which could lead to server stopping) or do we keep the slot so that
user can somehow fix the issue (reconfigure subscription to not care
about that publication for example) and continue replication without
further loss?

---

From: Stephen Frost <[email-masked]>
Date: Thu, 27 Dec 2018 14:05:11 -0500

Greetings,

> logical replication without having it for RLS and vice versa.

RLS already is able to be enabled/disabled on a per-table basis.  I
could see how we might want to extend the existing policy system to have
a way to enable/disable individual policies for RLS but that should be
reasonably straight-forward to do, I would think.

> something the current patch is missing as well BTW).

The policy system already has the notion of a 'visible' check and a
'does the new row match this' check (USING vs. WITH CHECK policies).
Perhaps if you could outline the specific use-cases that you're thinking
about, we could discuss them and make sure that they fit within those
mechanisms- or, if not, discuss if such a use-case would make sense for
RLS as well and, if so, figure out a way to support that for both.

> don't want this to depend on RLS being enabled for a table).

I'm fine with this being able to be independently enabled/disabled,
apart from RLS.

> duplicating all the work for every connection.

I agree that supporting this could be done in a later patch, however, I
do feel that when we go to add support for non-owners to create
publications then RLS needs to be supported at that point (and by more
than just 'throw an error').  I can agree with incremental improvements
but I don't want to get to a point where we've got a bunch of
independent things only half of which work with other parts of the
system.

> Yeah that part is definitely worth investigating.

Glad to hear that.

Thanks!

Stephen

---

From: Stephen Frost <[email-masked]>
Date: Thu, 27 Dec 2018 14:19:25 -0500

Greetings,

> and we don't have it and IMHO it's not as easy to have one as you might

We don't have a great solution but we should be able to at least drop
and recreate the publication or subscription, even today, can't we?
Sure, that means having to recopy everything, but that's what you get if
you break your publication/subscription.  If we allow the user to get to
a point where the system can't be fixed then I agree that's a serious
issue, but hopefully that isn't the case.

> dependencies of say plpython function...

Well, we could at leasts depend on the functions explicitly listed at
the top level and I don't believe we even do that today.  I can't think
of any downside off-hand to that, given that we have create-or-replace
function.

> to need bit of creative thinking to do this.

We can't write to things inside the database in a historic snapshot and
we do have to deal with the fact that we're in error processing.  What
about writing somewhere that's outside of the regular database system?
Maybe a pg_logical/failed directory?  There's all the usual
complications from that around dealing with durable writes (if we need
to worry about that and I'm not sure that we do...  if we fail to
persist a write saying "X failed" and we restart.. well, it's gonna fail
again and we write it then), and cleaning things up as needed (but maybe
this is handled as part of the DROP, and we WAL that, so we can re-do
the removal of the failed marker file...), and if we need to think about
what should happen on replicas (is there anything?).

> about that publication for example) and continue replication without

I would think we'd have to fail the whole publication if there's a
failure for any part of it.  Replicating a partial set definitely sounds
wrong to me.  Once we stop replication, yes, we should clean the slot
and mark it failed so that we don't keep back WAL and so that we allow
the catalog xmin to move forward so that the failed publication doesn't
run the server out of disk space.

If we really think there's a use-case for keeping the replication slot
and allowing it to cause WAL to spool on the server and keep the catalog
xmin back then I'd suggest we make this behavior configurable- so that
users can choose on a publication if they want a failure to be
considered a 'soft' fail or a 'hard' fail.  A 'soft' fail would keep the
slot and keep the WAL and keep the catalog xmin, with the expectation
that the user will either drop the slot themselves or somehow fix it,
while a 'hard' fail would clean everything up except the skeleton of the
slot itself which the user would need to drop.

Thanks!

Stephen

---

From: Petr Jelinek <[email-masked]>
Date: Fri, 28 Dec 2018 00:15:10 +0100

Hi,

> reasonably straight-forward to do, I would think.

Sure, I was mostly referring to having ability of enable/disable this
independently of enabling/disabling RLS which you are okay with based on
bellow so no issue there from my side.

> RLS as well and, if so, figure out a way to support that for both.

So we'd use USING for old row images (UPDATE/DELETE) and WITH CHECK for
new ones (UPDATE/INSERT)? I think OLD/NEW is somewhat more natural
naming of this as there is no "SELECT" part of operation here, but as
long as the functionality is there I don't mind syntax that much.

> I'm fine with this being able to be independently enabled/disabled,

Cool.

> independent things only half of which work with other parts of the

Yes, using RLS infrastructure now will make it easier to add support for
publishing without being owner at some later point, just let's please
not make publishing without being owner part of requirements for this.

---

From: Petr Jelinek <[email-masked]>
Date: Fri, 28 Dec 2018 00:36:25 +0100

> and recreate the publication or subscription, even today, can't we?

Well we can drop thing always, yes, not having ability to drop things
when they break would be bad design. I am debating ability to recover
without rebuilding everything a there are cases where you simply can't
rebuild everything (ie we allow filtering out deletes). I don't like
disabling UDFs either as that means that user created types are unusable
in filters, I just wonder if saying "sorry your replica is gone" is any
better.

> you break your publication/subscription.

This is but off-topic here, but I really wonder how are you currently
breaking your publications/subscriptions.

> of any downside off-hand to that, given that we have create-or-replace

I dunno how much is that worth it TBH, the situations where I've seen
this issue (pglogical has this feature for long time and suffers from
the same lack of dependency tracking) is that somebody drops table/type
used in a function that is used as filter.

> what should happen on replicas (is there anything?).

That sounds pretty reasonable. Given that this is corner-case user error
we could perhaps do extra work to ensure things are fsynced even if it's
all not too fast...

> run the server out of disk space.

I agree that continuing replication where some part of publication is
broken seems wrong and that we should stop replication at that point.

> If we really think there's a use-case for keeping the replication slot

It's not so much about use-case as it is about complete change of
behavior - there is no current error where we remove existing slot.
The use case for keeping slot is a) investigation of the issue, b) just
skipping the broken part of stream by advancing origin on subscription
and continuing replication, with some luck that can mean only single
table needs resyncing, which is better than rebuilding everything.

I think some kind of automated slot cleanup is desirable, but likely
separate feature that should be designed based on amount of outstanding
wal or something.

---

From: Andres Freund <[email-masked]>
Date: Sun, 3 Feb 2019 02:14:39 -0800

Hi,

> and improving docs. I also forget to declare pg_publication_rel TOAST

As far as I can tell, the patch has not been refreshed since. So I'm
marking this as returned with feedback for now. Please resubmit once
ready.

Greetings,

Andres Freund

---

From: [email-masked]
Date: Wed, 28 Aug 2019 00:10:47 +0300

Hi Euler,

> marking this as returned with feedback for now. Please resubmit once

Do you have any plans for continuing working on this patch and 
submitting it again on the closest September commitfest? There are only 
a few days left. Anyway, I will be glad to review the patch if you do 
submit it, though I didn't yet dig deeply into the code.

I've rebased recently the entire patch set (attached) and it works fine. 
Your tap test is passed. Also I've added a new test case (see 0009 
attached) with real life example of bidirectional replication (BDR) 
utilising this new WHERE clause. This naive BDR is implemented using 
is_cloud flag, which is set to TRUE/FALSE on cloud/remote nodes 
respectively.

Although almost all new tests are passed, there is a problem with DELETE 
replication, so 1 out of 10 tests is failed. It isn't replicated if the 
record was created with is_cloud=TRUE on cloud, replicated to remote; 
then updated with is_cloud=FALSE on remote, replicated to cloud; then 
deleted on remote.

Regards
--
Alexey Kondratov
Postgres Professional https://www.postgrespro.com
Russian Postgres Company

From ae80e1616fb6374968a09e3c44f0abe59ebf3a87 Mon Sep 17 00:00:00 2001
From: Euler Taveira <[email-masked]>
Date: Fri, 9 Mar 2018 18:39:22 +0000
Subject: [PATCH v2 1/9] Remove unused atttypmod column from initial table
 synchronization

diff --git a/src/backend/replication/logical/tablesync.c b/src/backend/replication/logical/tablesync.c
index 7881079e96..0a565dd837 100644
--- a/src/backend/replication/logical/tablesync.c
+++ b/src/backend/replication/logical/tablesync.c
@@ -647,7 +647,7 @@ fetch_remote_table_info(char *nspname, char *relname,
 	StringInfoData cmd;
 	TupleTableSlot *slot;
 	Oid			tableRow[2] = {OIDOID, CHAROID};
-	Oid			attrRow[4] = {TEXTOID, OIDOID, INT4OID, BOOLOID};
+	Oid			attrRow[3] = {TEXTOID, OIDOID, BOOLOID};
 	bool		isnull;
 	int			natt;
 
@@ -691,7 +691,6 @@ fetch_remote_table_info(char *nspname, char *relname,
 	appendStringInfo(&cmd,
 					 "SELECT a.attname,"
 					 "       a.atttypid,"
-					 "       a.atttypmod,"
 					 "       a.attnum = ANY(i.indkey)"
 					 "  FROM pg_catalog.pg_attribute a"
 					 "  LEFT JOIN pg_catalog.pg_index i"
@@ -703,7 +702,7 @@ fetch_remote_table_info(char *nspname, char *relname,
 					 lrel->remoteid,
 					 (walrcv_server_version(wrconn) >= 120000 ? "AND a.attgenerated = ''" : ""),
 					 lrel->remoteid);
-	res = walrcv_exec(wrconn, cmd.data, 4, attrRow);
+	res = walrcv_exec(wrconn, cmd.data, 3, attrRow);
 
@@ -724,7 +723,7 @@ fetch_remote_table_info(char *nspname, char *relname,

base-commit: 9acda731184c1ebdf99172cbb19d0404b7eebc37
-- 
2.19.1

From 362f2cc97745690ff4739b530f5ba95aea59be09 Mon Sep 17 00:00:00 2001
From: Euler Taveira <[email-masked]>
Date: Fri, 9 Mar 2018 17:37:36 +0000
Subject: [PATCH v2 2/9] Store number of tuples in WalRcvExecResult

 .../replication/libpqwalreceiver/libpqwalreceiver.c      | 9 ++++++---

diff --git a/src/backend/replication/libpqwalreceiver/libpqwalreceiver.c b/src/backend/replication/libpqwalreceiver/libpqwalreceiver.c
index 6eba08a920..846b6f89f1 100644
--- a/src/backend/replication/libpqwalreceiver/libpqwalreceiver.c
+++ b/src/backend/replication/libpqwalreceiver/libpqwalreceiver.c
@@ -878,6 +878,7 @@ libpqrcv_processTuples(PGresult *pgres, WalRcvExecResult *walres,
 				 errdetail("Expected %d fields, got %d fields.",
 						   nRetTypes, nfields)));

 	/* Create tuple descriptor corresponding to expected result. */
@@ -888,7 +889,7 @@ libpqrcv_processTuples(PGresult *pgres, WalRcvExecResult *walres,
 	attinmeta = TupleDescGetAttInMetadata(walres->tupledesc);

 	/* Create temporary context for local allocations. */
@@ -897,15 +898,17 @@ libpqrcv_processTuples(PGresult *pgres, WalRcvExecResult *walres,
 									   ALLOCSET_DEFAULT_SIZES);

 		ProcessWalRcvInterrupts();

+		cstrs = palloc(nfields * sizeof(char *));
+
 		/*
 		 * Fill cstrs with null-terminated strings of column values.
 		 */
diff --git a/src/backend/replication/logical/tablesync.c b/src/backend/replication/logical/tablesync.c
index 0a565dd837..42db4ada9e 100644
--- a/src/backend/replication/logical/tablesync.c
+++ b/src/backend/replication/logical/tablesync.c
@@ -709,9 +709,8 @@ fetch_remote_table_info(char *nspname, char *relname,
 				(errmsg("could not fetch table info for table \"%s.%s\": %s",
 						nspname, relname, res->err)));

 	natt = 0;
diff --git a/src/include/replication/walreceiver.h b/src/include/replication/walreceiver.h
index e12a934966..0d32d598d8 100644
--- a/src/include/replication/walreceiver.h
+++ b/src/include/replication/walreceiver.h
@@ -196,6 +196,7 @@ typedef struct WalRcvExecResult
 	char	   *err;
 	Tuplestorestate *tuplestore;
 	TupleDesc	tupledesc;
+	int			ntuples;
 } WalRcvExecResult;
 
2.19.1

From 2b23421d208c4760e7b3d38adc57ec62685b2d35 Mon Sep 17 00:00:00 2001
From: Euler Taveira <[email-masked]>
Date: Tue, 27 Feb 2018 02:21:03 +0000
Subject: [PATCH v2 3/9] Refactor function create_estate_for_relation

diff --git a/src/backend/replication/logical/worker.c b/src/backend/replication/logical/worker.c
index 11e6331f49..d9952c8b7e 100644
--- a/src/backend/replication/logical/worker.c
+++ b/src/backend/replication/logical/worker.c
@@ -173,7 +173,7 @@ ensure_transaction(void)
  * This is based on similar code in copy.c
  */
 static EState *
-create_estate_for_relation(LogicalRepRelMapEntry *rel)
+create_estate_for_relation(Relation rel)
 {
 	EState	   *estate;
 	ResultRelInfo *resultRelInfo;
@@ -183,13 +183,13 @@ create_estate_for_relation(LogicalRepRelMapEntry *rel)

 
 	estate->es_result_relations = resultRelInfo;
 	estate->es_num_result_relations = 1;
@@ -589,7 +589,7 @@ apply_handle_insert(StringInfo s)
 	}
 
 	/* Initialize the executor state. */
-	estate = create_estate_for_relation(rel);
+	estate = create_estate_for_relation(rel->localrel);
 	remoteslot = ExecInitExtraTupleSlot(estate,
 										RelationGetDescr(rel->localrel),
 										&TTSOpsVirtual);
@@ -696,7 +696,7 @@ apply_handle_update(StringInfo s)
 	check_relation_updatable(rel);
 
 	/* Initialize the executor state. */
-	estate = create_estate_for_relation(rel);
+	estate = create_estate_for_relation(rel->localrel);
 	remoteslot = ExecInitExtraTupleSlot(estate,
 										RelationGetDescr(rel->localrel),
 										&TTSOpsVirtual);
@@ -815,7 +815,7 @@ apply_handle_delete(StringInfo s)
 	check_relation_updatable(rel);
 
 										RelationGetDescr(rel->localrel),
 										&TTSOpsVirtual);
-- 
2.19.1

From b5e55cace866ad8b4de554fe6cd326ad2d11fe81 Mon Sep 17 00:00:00 2001
From: Euler Taveira <[email-masked]>
Date: Wed, 24 Jan 2018 17:01:31 -0200
Subject: [PATCH v2 4/9] Rename a WHERE node

diff --git a/src/backend/parser/gram.y b/src/backend/parser/gram.y
index c97bb367f8..1de8f56794 100644
--- a/src/backend/parser/gram.y
+++ b/src/backend/parser/gram.y
@@ -476,7 +476,7 @@ static Node *makeRecursiveViewSelect(char *relname, List *aliases, Node *query);
 %type <node>	def_arg columnElem where_clause where_or_current_clause
 				a_expr b_expr c_expr AexprConst indirection_el opt_slice_bound
 				columnref in_expr having_clause func_table xmltable array_expr
-				ExclusionWhereClause operator_def_arg
+				OptWhereClause operator_def_arg
 %type <list>	rowsfrom_item rowsfrom_list opt_col_def_list
 %type <boolean> opt_ordinality
 %type <list>	ExclusionConstraintList ExclusionConstraintElem
@@ -3710,7 +3710,7 @@ ConstraintElem:
 					$$ = (Node *)n;
 				}
 			| EXCLUDE access_method_clause '(' ExclusionConstraintList ')'
-				opt_c_include opt_definition OptConsTableSpace  ExclusionWhereClause
+				opt_c_include opt_definition OptConsTableSpace  OptWhereClause
 				ConstraintAttributeSpec
 				{
 					Constraint *n = makeNode(Constraint);
@@ -3812,7 +3812,7 @@ ExclusionConstraintElem: index_elem WITH any_operator
 			}
 		;
 
-ExclusionWhereClause:
+OptWhereClause:
2.19.1

From fcb9a06babe9ce94bff256e93ae1a2f44c4532b6 Mon Sep 17 00:00:00 2001
From: Euler Taveira <[email-masked]>
Date: Tue, 27 Feb 2018 04:03:13 +0000
Subject: [PATCH v2 5/9] Row filtering for logical replication

When you define or modify a publication you optionally can filter rows
to be published using a WHERE condition. This condition is any
expression that evaluates to boolean. Only those rows that
satisfy the WHERE condition will be sent to subscribers.
---
 doc/src/sgml/ref/alter_publication.sgml     |   9 +-
 doc/src/sgml/ref/create_publication.sgml    |  14 ++-
 src/backend/catalog/pg_publication.c        |  46 +++++++-
 src/backend/commands/publicationcmds.c      |  74 ++++++++----
 src/backend/parser/gram.y                   |  26 ++++-
 src/backend/parser/parse_agg.c              |  10 ++
 src/backend/parser/parse_expr.c             |   5 +
 src/backend/parser/parse_func.c             |   2 +
 src/backend/replication/logical/proto.c     |   2 +-
 src/backend/replication/logical/relation.c  |  13 +++
 src/backend/replication/logical/tablesync.c | 119 ++++++++++++++++++--
 src/backend/replication/logical/worker.c    |   2 +-
 src/backend/replication/pgoutput/pgoutput.c |  98 +++++++++++++++-
 src/include/catalog/pg_publication.h        |  10 +-
 src/include/catalog/pg_publication_rel.h    |  10 +-
 src/include/nodes/nodes.h                   |   1 +
 src/include/nodes/parsenodes.h              |  11 +-
 src/include/parser/parse_node.h             |   1 +
 src/include/replication/logicalproto.h      |   2 +
 src/include/replication/logicalrelation.h   |   2 +
 src/test/regress/expected/misc_sanity.out   |   3 +-
 src/test/subscription/t/010_row_filter.pl   |  97 ++++++++++++++++
 22 files changed, 499 insertions(+), 58 deletions(-)
 create mode 100644 src/test/subscription/t/010_row_filter.pl

index 534e598d93..5984915767 100644
 
  <refsynopsisdiv>
 <synopsis>
index 99f87ca393..d5fed304a1 100644

      <para>
@@ -183,6 +186,13 @@ CREATE PUBLICATION mypublication FOR TABLE users, departments;
 </programlisting>
   </para>
 
+  <para>
index fd5da7d5f7..47a793408d 100644
 
@@ -149,18 +153,21 @@ pg_relation_is_publishable(PG_FUNCTION_ARGS)
 
 	rel = table_open(PublicationRelRelationId, RowExclusiveLock);
 
@@ -180,10 +187,27 @@ publication_add_relation(Oid pubid, Relation targetrel,
 		ereport(ERROR,

 	/* Form a tuple. */
 	memset(values, 0, sizeof(values));
@@ -197,6 +221,12 @@ publication_add_relation(Oid pubid, Relation targetrel,
 	values[Anum_pg_publication_rel_prrelid - 1] =
 		ObjectIdGetDatum(relid);

 	/* Insert tuple into catalog. */
@@ -213,11 +243,17 @@ publication_add_relation(Oid pubid, Relation targetrel,
 	ObjectAddressSet(referenced, RelationRelationId, relid);
 	recordDependencyOn(&myself, &referenced, DEPENDENCY_AUTO);
 
+	/* Add dependency on the objects mentioned in the row filter expression */
+	if (whereclause)
+		recordDependencyOnExpr(&myself, whereclause, pstate->p_rtable, DEPENDENCY_NORMAL);
+
+	free_parsestate(pstate);
+
 	/* Close the table. */
 	table_close(rel, RowExclusiveLock);

 	return myself;
 }
diff --git a/src/backend/commands/publicationcmds.c b/src/backend/commands/publicationcmds.c
index f115d4bf80..bc7f9210e9 100644
--- a/src/backend/commands/publicationcmds.c
+++ b/src/backend/commands/publicationcmds.c
@@ -352,6 +352,27 @@ AlterPublicationTables(AlterPublicationStmt *stmt, Relation rel,

+	/*
 
 	if (stmt->tableAction == DEFELEM_ADD)
@@ -373,9 +394,9 @@ AlterPublicationTables(AlterPublicationStmt *stmt, Relation rel,
 
 			foreach(newlc, rels)
 			{
 
-				if (RelationGetRelid(newrel) == oldrelid)
+				if (RelationGetRelid(newrel->relation) == oldrelid)
 				{
 					found = true;
 					break;
@@ -384,7 +405,8 @@ AlterPublicationTables(AlterPublicationStmt *stmt, Relation rel,
 
 			if (!found)
 			{
-				Relation	oldrel = table_open(oldrelid,
+				PublicationRelationQual *oldrel = palloc(sizeof(PublicationRelationQual));
+				oldrel->relation = table_open(oldrelid,
 												ShareUpdateExclusiveLock);
 
 				delrels = lappend(delrels, oldrel);
@@ -510,16 +532,22 @@ OpenTableList(List *tables)
 	List	   *relids = NIL;
 	List	   *rels = NIL;
 	ListCell   *lc;
+	PublicationRelationQual	*relqual;
 
 	/*
 	 * Open, share-lock, and check all the explicitly-specified relations
 	 */
 	foreach(lc, tables)
 	{
-		RangeVar   *rv = castNode(RangeVar, lfirst(lc));
-		bool		recurse = rv->inh;
-		Relation	rel;
-		Oid			myrelid;
+		// RangeVar   *rv = castNode(RangeVar, lfirst(lc));
+		// bool		recurse = rv->inh;
+		// Relation	rel;
+		// Oid			myrelid;
+		PublicationTable	*t = lfirst(lc);
+		RangeVar  			*rv = t->relation;
+		Relation			rel;
+		bool				recurse = rv->inh;
+		Oid					myrelid;
 
 		/* Allow query cancel in case this takes a long time */
 		CHECK_FOR_INTERRUPTS();
@@ -539,8 +567,10 @@ OpenTableList(List *tables)
 			table_close(rel, ShareUpdateExclusiveLock);
 			continue;
 		}
-
-		rels = lappend(rels, rel);
+		relqual = palloc(sizeof(PublicationRelationQual));
+		relqual->relation = rel;
+		relqual->whereClause = t->whereClause;
+		rels = lappend(rels, relqual);
 		relids = lappend_oid(relids, myrelid);
 
 		/* Add children of this rel, if requested */
@@ -568,7 +598,11 @@ OpenTableList(List *tables)
 
 				rel = table_open(childrelid, NoLock);
@@ -589,10 +623,12 @@ CloseTableList(List *rels)
 
 	foreach(lc, rels)
 	{
 
-		table_close(rel, NoLock);
+		table_close(rel->relation, NoLock);
 	}
+
+	list_free_deep(rels);
 }
 
 /*
@@ -608,13 +644,13 @@ PublicationAddTables(Oid pubid, List *rels, bool if_not_exists,
 
 	foreach(lc, rels)
 	{

 		obj = publication_add_relation(pubid, rel, if_not_exists);
 		if (stmt)
@@ -640,8 +676,8 @@ PublicationDropTables(Oid pubid, List *rels, bool missing_ok)
 
 	foreach(lc, rels)
 	{
 
 		prid = GetSysCacheOid2(PUBLICATIONRELMAP, Anum_pg_publication_rel_oid,
 							   ObjectIdGetDatum(relid),
@@ -654,7 +690,7 @@ PublicationDropTables(Oid pubid, List *rels, bool missing_ok)
 			ereport(ERROR,
 					(errcode(ERRCODE_UNDEFINED_OBJECT),
 					 errmsg("relation \"%s\" is not part of the publication",
-							RelationGetRelationName(rel))));
+							RelationGetRelationName(rel->relation))));
 		}
 
index 1de8f56794..bd87e80e1b 100644
@@ -404,13 +404,13 @@ static Node *makeRecursiveViewSelect(char *relname, List *aliases, Node *query);

 %type <list>	opt_fdw_options fdw_options
@@ -9518,7 +9518,7 @@ opt_publication_for_tables:
 		;
 
 publication_for_tables:
@@ -9549,7 +9549,7 @@ AlterPublicationStmt:
@@ -9575,6 +9575,20 @@ AlterPublicationStmt:
 				}
 		;
 
+publication_table_list:
index f418c61545..e317e04695 100644
@@ -536,6 +536,13 @@ check_agglevels_and_constraints(ParseState *pstate, Node *expr)
 
 			break;
 
 		case EXPR_KIND_COPY_WHERE:
 			if (isAgg)
@@ -933,6 +940,9 @@ transformWindowFuncCall(ParseState *pstate, WindowFunc *wfunc,
 		case EXPR_KIND_GENERATED_COLUMN:
 			err = _("window functions are not allowed in column generation expressions");
 			break;
+		case EXPR_KIND_PUBLICATION_WHERE:
+			err = _("window functions are not allowed in publication WHERE expressions");
+			break;
 
 			/*
 			 * There is intentionally no default: case here, so that the
diff --git a/src/backend/parser/parse_expr.c b/src/backend/parser/parse_expr.c
index 76f3dd7076..4942f28f50 100644
--- a/src/backend/parser/parse_expr.c
+++ b/src/backend/parser/parse_expr.c
@@ -571,6 +571,7 @@ transformColumnRef(ParseState *pstate, ColumnRef *cref)
 		case EXPR_KIND_CALL_ARGUMENT:
 		case EXPR_KIND_COPY_WHERE:
 		case EXPR_KIND_GENERATED_COLUMN:
+		case EXPR_KIND_PUBLICATION_WHERE:
 			/* okay */
 			break;
 
@@ -1924,6 +1925,8 @@ transformSubLink(ParseState *pstate, SubLink *sublink)
 			break;
 		case EXPR_KIND_CALL_ARGUMENT:
 			err = _("cannot use subquery in CALL argument");
+		case EXPR_KIND_PUBLICATION_WHERE:
+			err = _("cannot use subquery in publication WHERE expression");
 			break;
 		case EXPR_KIND_COPY_WHERE:
 			err = _("cannot use subquery in COPY FROM WHERE condition");
@@ -3561,6 +3564,8 @@ ParseExprKindName(ParseExprKind exprKind)
 			return "WHERE";
 		case EXPR_KIND_GENERATED_COLUMN:
 			return "GENERATED AS";
+		case EXPR_KIND_PUBLICATION_WHERE:
+			return "publication expression";
 
 			/*
 			 * There is intentionally no default: case here, so that the
diff --git a/src/backend/parser/parse_func.c b/src/backend/parser/parse_func.c
index 8e926539e6..c7ec300d0e 100644
--- a/src/backend/parser/parse_func.c
+++ b/src/backend/parser/parse_func.c
@@ -2509,6 +2509,8 @@ check_srf_call_placement(ParseState *pstate, Node *last_srf, int location)
 			break;
 		case EXPR_KIND_CALL_ARGUMENT:
 			err = _("set-returning functions are not allowed in CALL arguments");
+		case EXPR_KIND_PUBLICATION_WHERE:
+			err = _("set-returning functions are not allowed in publication WHERE expressions");
 			break;
 		case EXPR_KIND_COPY_WHERE:
 			err = _("set-returning functions are not allowed in COPY FROM WHERE conditions");
diff --git a/src/backend/replication/logical/proto.c b/src/backend/replication/logical/proto.c
index e7df47de3e..eb4eaa2a33 100644
--- a/src/backend/replication/logical/proto.c
+++ b/src/backend/replication/logical/proto.c
@@ -378,7 +378,7 @@ logicalrep_write_rel(StringInfo out, Relation rel)
 LogicalRepRelation *
 logicalrep_read_rel(StringInfo in)
 {
-	LogicalRepRelation *rel = palloc(sizeof(LogicalRepRelation));
+	LogicalRepRelation *rel = palloc0(sizeof(LogicalRepRelation));

index 85269c037d..a14986e3ab 100644

index 42db4ada9e..fc37f74e89 100644
@@ -648,8 +648,14 @@ fetch_remote_table_info(char *nspname, char *relname,
 
 	lrel->nspname = nspname;
 	lrel->relname = relname;
@@ -713,20 +719,20 @@ fetch_remote_table_info(char *nspname, char *relname,
 	lrel->atttyps = palloc0(res->ntuples * sizeof(Oid));
 	lrel->attkeys = NULL;
 
-	natt = 0;
+	n = 0;
 	slot = MakeSingleTupleTableSlot(res->tupledesc, &TTSOpsMinimalTuple);

@@ -734,7 +740,54 @@ fetch_remote_table_info(char *nspname, char *relname,
 	}
 	ExecDropSingleTupleTableSlot(slot);
 
-	lrel->natts = natt;
+	lrel->natts = n;
+
+	slot = MakeSingleTupleTableSlot(res->tupledesc, &TTSOpsMinimalTuple);
 
 	walrcv_clear_result(res);
 	pfree(cmd.data);
@@ -767,10 +820,57 @@ copy_table(Relation rel)
 	relmapentry = logicalrep_rel_open(lrel.remoteid, NoLock);
 	Assert(rel == relmapentry->localrel);
 
@@ -785,7 +885,6 @@ copy_table(Relation rel)

 	/* Do the copy */
diff --git a/src/backend/replication/logical/worker.c b/src/backend/replication/logical/worker.c
index d9952c8b7e..cef0c52955 100644
--- a/src/backend/replication/logical/worker.c
+++ b/src/backend/replication/logical/worker.c
@@ -172,7 +172,7 @@ ensure_transaction(void)
  *
  * This is based on similar code in copy.c
  */
-static EState *
+EState *
 create_estate_for_relation(Relation rel)
 {
 	EState	   *estate;
diff --git a/src/backend/replication/pgoutput/pgoutput.c b/src/backend/replication/pgoutput/pgoutput.c
index 9c08757fca..e9646ac483 100644
--- a/src/backend/replication/pgoutput/pgoutput.c
+++ b/src/backend/replication/pgoutput/pgoutput.c
@@ -12,15 +12,26 @@
  */
 #include "postgres.h"
 
+#include "optimizer/optimizer.h"
 
 #include "fmgr.h"

@@ -60,6 +71,7 @@ typedef struct RelationSyncEntry
 
 /* Map used to remember which relation schemas we sent. */
@@ -335,6 +347,63 @@ pgoutput_change(LogicalDecodingContext *ctx, ReorderBufferTXN *txn,
 			Assert(false);
 	}
 
+		ecxt->ecxt_scantuple = ExecInitExtraTupleSlot(estate, tupdesc, &TTSOpsHeapTuple);
 
@@ -570,10 +639,14 @@ get_rel_sync_entry(PGOutputData *data, Oid relid)
 		 */
 		entry->pubactions.pubinsert = entry->pubactions.pubupdate =
 			entry->pubactions.pubdelete = entry->pubactions.pubtruncate = false;
+		entry->row_filter = NIL;

 			if (pub->alltables || list_member_oid(pubids, pub->oid))
 			{
@@ -583,9 +656,23 @@ get_rel_sync_entry(PGOutputData *data, Oid relid)
 				entry->pubactions.pubtruncate |= pub->pubactions.pubtruncate;
 			}

 		list_free(pubids);
@@ -660,5 +747,10 @@ rel_sync_cache_publication_cb(Datum arg, int cacheid, uint32 hashvalue)
 	 */
 	hash_seq_init(&status, RelationSyncCache);
 	while ((entry = (RelationSyncEntry *) hash_seq_search(&status)) != NULL)
+	{
 		entry->replicate_valid = false;
+		if (list_length(entry->row_filter) > 0)
+			list_free(entry->row_filter);
+		entry->row_filter = NIL;
+	}
 }
diff --git a/src/include/catalog/pg_publication.h b/src/include/catalog/pg_publication.h
index 20a2f0ac1b..fe878c6957 100644
--- a/src/include/catalog/pg_publication.h
+++ b/src/include/catalog/pg_publication.h
@@ -78,6 +78,12 @@ typedef struct Publication
 	PublicationActions pubactions;
 } Publication;
 
@@ -86,8 +92,8 @@ extern List *GetAllTablesPublications(void);
 
-											  bool if_not_exists);
+						 					  bool if_not_exists);
 
 extern char *get_publication_name(Oid pubid, bool missing_ok);
index 5f5bc92ab3..70ffef5dfd 100644
@@ -28,9 +28,13 @@
-	Oid			oid;			/* oid */
+	Oid				oid;			/* oid */
 
 /* ----------------
diff --git a/src/include/nodes/nodes.h b/src/include/nodes/nodes.h
index 3cbb08df92..7f83da1ee8 100644
--- a/src/include/nodes/nodes.h
+++ b/src/include/nodes/nodes.h
@@ -476,6 +476,7 @@ typedef enum NodeTag
 	T_PartitionRangeDatum,
 	T_PartitionCmd,
 	T_VacuumRelation,
+	T_PublicationTable,
 
 	/*
 	 * TAGS FOR REPLICATION GRAMMAR PARSE NODES (replnodes.h)
diff --git a/src/include/nodes/parsenodes.h b/src/include/nodes/parsenodes.h
index 94ded3c135..359f773092 100644
--- a/src/include/nodes/parsenodes.h
+++ b/src/include/nodes/parsenodes.h
@@ -3461,12 +3461,19 @@ typedef struct AlterTSConfigurationStmt
 } AlterTSConfigurationStmt;

@@ -3479,7 +3486,7 @@ typedef struct AlterPublicationStmt
 	List	   *options;		/* List of DefElem nodes */
 
 	/* parameters used for ALTER PUBLICATION ... ADD/DROP TABLE */
-	List	   *tables;			/* List of tables to add/drop */
+	List	   *tables;			/* List of PublicationTable to add/drop */
 	bool		for_all_tables; /* Special publication for all tables in db */
 	DefElemAction tableAction;	/* What action to perform with the tables */
 } AlterPublicationStmt;
diff --git a/src/include/parser/parse_node.h b/src/include/parser/parse_node.h
index 7c099e7084..c2e8b9fcb9 100644
--- a/src/include/parser/parse_node.h
+++ b/src/include/parser/parse_node.h
@@ -73,6 +73,7 @@ typedef enum ParseExprKind
 	EXPR_KIND_CALL_ARGUMENT,	/* procedure argument in CALL */
 	EXPR_KIND_COPY_WHERE,		/* WHERE condition in COPY FROM */
 	EXPR_KIND_GENERATED_COLUMN, /* generation expression for a column */
+	EXPR_KIND_PUBLICATION_WHERE	/* WHERE condition for a table in PUBLICATION */
 } ParseExprKind;

index 3fc430af01..8dabc35791 100644
 
 /* Type mapping info */
index 2642a3f94e..5cc307ee0e 100644
 
index 8538173ff8..68bbdf7016 100644
@@ -107,5 +107,6 @@ ORDER BY 1, 2;
 
index 0000000000..6c174fa895
2.19.1

From 19cb78c319e3755f412b3d622e68837ccaf05496 Mon Sep 17 00:00:00 2001
From: Euler Taveira <[email-masked]>
Date: Thu, 17 May 2018 20:52:28 +0000
Subject: [PATCH v2 6/9] Print publication WHERE condition in psql

---

diff --git a/src/bin/psql/describe.c b/src/bin/psql/describe.c
index 774cc764ff..f103747901 100644
--- a/src/bin/psql/describe.c
+++ b/src/bin/psql/describe.c
@@ -5875,7 +5875,8 @@ describePublications(const char *pattern)
 		if (!puballtables)
 		{
 			printfPQExpBuffer(&buf,
-							  "SELECT n.nspname, c.relname\n"
+							  "SELECT n.nspname, c.relname,\n"
+							  "  pg_get_expr(pr.prrowfilter, c.oid)\n"
 							  "FROM pg_catalog.pg_class c,\n"
 							  "     pg_catalog.pg_namespace n,\n"
 							  "     pg_catalog.pg_publication_rel pr\n"
@@ -5905,6 +5906,10 @@ describePublications(const char *pattern)
 								  PQgetvalue(tabres, j, 0),
 								  PQgetvalue(tabres, j, 1));
 
2.19.1

From c773fc66047feed639b50648f86d9a5cb783fc07 Mon Sep 17 00:00:00 2001
From: Euler Taveira <[email-masked]>
Date: Sat, 15 Sep 2018 02:52:00 +0000
Subject: [PATCH v2 7/9] Publication where condition support for pg_dump

---

diff --git a/src/bin/pg_dump/pg_dump.c b/src/bin/pg_dump/pg_dump.c
index 34981401bf..91475bc5f8 100644
--- a/src/bin/pg_dump/pg_dump.c
+++ b/src/bin/pg_dump/pg_dump.c
@@ -3959,6 +3959,7 @@ getPublicationTables(Archive *fout, TableInfo tblinfo[], int numTables)
 	int			i_tableoid;
 	int			i_oid;
 	int			i_pubname;
+	int			i_pubrelqual;
 	int			i,
 				j,
 				ntups;
@@ -3991,7 +3992,8 @@ getPublicationTables(Archive *fout, TableInfo tblinfo[], int numTables)
 
@@ -4012,6 +4014,7 @@ getPublicationTables(Archive *fout, TableInfo tblinfo[], int numTables)

@@ -4027,6 +4030,11 @@ getPublicationTables(Archive *fout, TableInfo tblinfo[], int numTables)
 			pubrinfo[j].pubname = pg_strdup(PQgetvalue(res, j, i_pubname));
 			pubrinfo[j].pubtable = tbinfo;
 
@@ -4055,8 +4063,11 @@ dumpPublicationTable(Archive *fout, PublicationRelInfo *pubrinfo)

 	/*
 	 * There is no point in creating drop query as the drop is done by table
diff --git a/src/bin/pg_dump/pg_dump.h b/src/bin/pg_dump/pg_dump.h
index dfba58ac58..3ed2e1be9c 100644
--- a/src/bin/pg_dump/pg_dump.h
+++ b/src/bin/pg_dump/pg_dump.h
@@ -608,6 +608,7 @@ typedef struct _PublicationRelInfo
 	DumpableObject dobj;
 	TableInfo  *pubtable;
 	char	   *pubname;
+	char	   *pubrelqual;
 } PublicationRelInfo;
 
 /*
-- 
2.19.1

From 61a3e08edd5c3ce3cbcc750a2c074922afd6676d Mon Sep 17 00:00:00 2001
From: Euler Taveira <[email-masked]>
Date: Wed, 14 Mar 2018 00:53:17 +0000
Subject: [PATCH v2 8/9] Debug for row filtering

---
 src/backend/commands/publicationcmds.c      | 11 ++++
 src/backend/replication/logical/tablesync.c |  1 +
 src/backend/replication/pgoutput/pgoutput.c | 66 +++++++++++++++++++++
 3 files changed, 78 insertions(+)

diff --git a/src/backend/commands/publicationcmds.c b/src/backend/commands/publicationcmds.c
index bc7f9210e9..8e107bddfb 100644
--- a/src/backend/commands/publicationcmds.c
+++ b/src/backend/commands/publicationcmds.c
@@ -341,6 +341,7 @@ AlterPublicationTables(AlterPublicationStmt *stmt, Relation rel,
 	List	   *rels = NIL;
 	Form_pg_publication pubform = (Form_pg_publication) GETSTRUCT(tup);
 	Oid			pubid = pubform->oid;
+	ListCell   *xpto;
 
 	/* Check that user is allowed to manipulate the publication tables. */
 	if (pubform->puballtables)
@@ -352,6 +353,16 @@ AlterPublicationTables(AlterPublicationStmt *stmt, Relation rel,

index fc37f74e89..c86affad03 100644
@@ -871,6 +871,7 @@ copy_table(Relation rel)
index e9646ac483..5012cfdde7 100644
@@ -34,6 +34,7 @@
@@ -323,6 +324,10 @@ pgoutput_change(LogicalDecodingContext *ctx, ReorderBufferTXN *txn,

@@ -347,6 +352,17 @@ pgoutput_change(LogicalDecodingContext *ctx, ReorderBufferTXN *txn,
 			Assert(false);
 	}
 
@@ -363,6 +379,42 @@ pgoutput_change(LogicalDecodingContext *ctx, ReorderBufferTXN *txn,
 
+#ifdef	_NOT_USED
+		if (old_tuple)
+		{
+			int i;
+
@@ -378,6 +430,7 @@ pgoutput_change(LogicalDecodingContext *ctx, ReorderBufferTXN *txn,
 			Oid			expr_type;
 			Datum		res;
 			bool		isnull;
+			char		*s = NULL;

@@ -388,14 +441,24 @@ pgoutput_change(LogicalDecodingContext *ctx, ReorderBufferTXN *txn,
 			expr_state = ExecInitExpr(expr, NULL);
 			res = ExecEvalExpr(expr_state, ecxt, &isnull);

@@ -666,9 +729,12 @@ get_rel_sync_entry(PGOutputData *data, Oid relid)
 
2.19.1

From 59d97603c8b0ecc20a7f04acee2e123fb1a26265 Mon Sep 17 00:00:00 2001
From: Alexey Kondratov <[email-masked]>
Date: Sun, 25 Aug 2019 16:21:32 +0300
Subject: [PATCH v2 9/9] Add simple BDR test for row filtering

---
 .../{010_row_filter.pl => 013_row_filter.pl}  |   0
 src/test/subscription/t/014_simple_bdr.pl     | 194 ++++++++++++++++++
 2 files changed, 194 insertions(+)
 rename src/test/subscription/t/{010_row_filter.pl => 013_row_filter.pl} (100%)
 create mode 100644 src/test/subscription/t/014_simple_bdr.pl

diff --git a/src/test/subscription/t/010_row_filter.pl b/src/test/subscription/t/013_row_filter.pl
similarity index 100%
rename from src/test/subscription/t/010_row_filter.pl
rename to src/test/subscription/t/013_row_filter.pl
diff --git a/src/test/subscription/t/014_simple_bdr.pl b/src/test/subscription/t/014_simple_bdr.pl
new file mode 100644
index 0000000000..f33b754d2a
--- /dev/null
+++ b/src/test/subscription/t/014_simple_bdr.pl
@@ -0,0 +1,194 @@
+# Test simple bidirectional logical replication behavior with row filtering
+# ID is meant to be something like uuid (e.g. from pgcrypto), but integer
+# type is used for simplicity.
+use strict;
+use warnings;
+use PostgresNode;
+use TestLib;
+use Test::More tests => 10;
+
+our $node_cloud;
+our $node_remote;
+our $cloud_appname = 'cloud_sub';
+our $remote_appname = 'remote_sub';
+
+sub check_data_consistency
+{
+	my $test_name = shift;
+	my $query = shift;
+	my $true_result = shift;
+	my $result;
+
+	$node_cloud->wait_for_catchup($remote_appname);
+	$node_remote->wait_for_catchup($cloud_appname);
+
+	$result =
+	$node_remote->safe_psql('postgres', $query);
+	is($result, $true_result, $test_name . ' on remote');
+	$result =
+	$node_cloud->safe_psql('postgres', $query);
+	is($result, $true_result, $test_name . ' on cloud');
+
+	return;
+}
+
+# Create cloud node
+$node_cloud = get_new_node('publisher');
+$node_cloud->init(allows_streaming => 'logical');
+$node_cloud->start;
+
+# Create remote node
+$node_remote = get_new_node('subscriber');
+$node_remote->init(allows_streaming => 'logical');
+$node_remote->start;
+
+# Test tables
+my $users_table = "CREATE TABLE users (
+						id integer primary key,
+						name text,
+						is_cloud boolean
+					);";
+my $docs_table = "CREATE TABLE docs (
+						id integer primary key,
+						user_id integer,
+						FOREIGN KEY (user_id) REFERENCES users (id),
+						content text,
+						is_cloud boolean
+					);";
+
+# Setup structure on cloud server
+$node_cloud->safe_psql('postgres', $users_table);
+
+# Setup structure on remote server
+$node_remote->safe_psql('postgres', $users_table);
+
+# Put in initial data
+$node_cloud->safe_psql('postgres',
+	"INSERT INTO users (id, name, is_cloud) VALUES (1, 'user1_on_cloud', TRUE);");
+$node_remote->safe_psql('postgres',
+	"INSERT INTO users (id, name, is_cloud) VALUES (2, 'user2_on_remote', FALSE);");
+$node_remote->safe_psql('postgres',
+	"INSERT INTO users (id, name, is_cloud) VALUES (100, 'user100_local_on_remote', TRUE);");
+
+# Setup logical replication
+$node_cloud->safe_psql('postgres', "CREATE PUBLICATION cloud;");
+$node_cloud->safe_psql('postgres', "ALTER PUBLICATION cloud ADD TABLE users WHERE (is_cloud IS TRUE);");
+
+$node_remote->safe_psql('postgres', "CREATE PUBLICATION remote;");
+$node_remote->safe_psql('postgres', "ALTER PUBLICATION remote ADD TABLE users WHERE (is_cloud IS FALSE);");
+
+my $cloud_connstr = $node_cloud->connstr . ' dbname=postgres';
+$node_remote->safe_psql('postgres',
+	"CREATE SUBSCRIPTION cloud_to_remote CONNECTION '$cloud_connstr application_name=$remote_appname' PUBLICATION cloud"
+);
+
+my $remote_connstr = $node_remote->connstr . ' dbname=postgres';
+$node_cloud->safe_psql('postgres',
+	"CREATE SUBSCRIPTION remote_to_cloud CONNECTION '$remote_connstr application_name=$cloud_appname' PUBLICATION remote"
+);
+
+# Wait for initial table sync to finish
+my $synced_query =
+"SELECT count(1) = 0 FROM pg_subscription_rel WHERE srsubstate NOT IN ('r', 's');";
+$node_remote->poll_query_until('postgres', $synced_query)
+  or die "Timed out while waiting for remote to synchronize data";
+$node_cloud->poll_query_until('postgres', $synced_query)
+  or die "Timed out while waiting for cloud to synchronize data";
+
+$node_cloud->wait_for_catchup($remote_appname);
+$node_remote->wait_for_catchup($cloud_appname);
+
+# Test initial table sync
+my $result =
+  $node_remote->safe_psql('postgres', "SELECT count(*) from users");
+is($result, qq(3), 'check initial table sync on remote');
+$result =
+  $node_cloud->safe_psql('postgres', "SELECT count(*) from users");
+is($result, qq(2), 'check initial table sync on cloud');
+
+# Test BDR
+$node_cloud->safe_psql('postgres',
+	"INSERT INTO users (id, name, is_cloud) VALUES (3, 'user3_on_cloud', TRUE);");
+$node_cloud->safe_psql('postgres',
+	"INSERT INTO users (id, name, is_cloud) VALUES (4, 'user4_on_cloud', TRUE);");
+$node_remote->safe_psql('postgres',
+	"INSERT INTO users (id, name, is_cloud) VALUES (5, 'user5_on_remote', FALSE);");
+
+$node_cloud->wait_for_catchup($remote_appname);
+$node_remote->wait_for_catchup($cloud_appname);
+
+$result =
+  $node_remote->safe_psql('postgres', "SELECT id, name, is_cloud FROM users ORDER BY id;");
+is($result, qq(1|user1_on_cloud|t
+2|user2_on_remote|f
+3|user3_on_cloud|t
+4|user4_on_cloud|t
+5|user5_on_remote|f
+100|user100_local_on_remote|t), 'check users on remote');
+$result =
+  $node_cloud->safe_psql('postgres', "SELECT id, name, is_cloud FROM users ORDER BY id;");
+is($result, qq(1|user1_on_cloud|t
+2|user2_on_remote|f
+3|user3_on_cloud|t
+4|user4_on_cloud|t
+5|user5_on_remote|f), 'check users on cloud');
+
+# Add table to cloud server
+$node_cloud->safe_psql('postgres', $docs_table);
+
+# Add table to remote server
+$node_remote->safe_psql('postgres', $docs_table);
+
+# Put in initial data
+$node_cloud->safe_psql('postgres',
+	"INSERT INTO docs (id, user_id, content, is_cloud) VALUES (1, 3, 'user3__doc1_on_cloud', TRUE);");
+
+# Add table to publication
+$node_cloud->safe_psql('postgres', "ALTER PUBLICATION cloud ADD TABLE docs WHERE (is_cloud IS TRUE);");
+$node_remote->safe_psql('postgres', "ALTER PUBLICATION remote ADD TABLE docs WHERE (is_cloud IS FALSE);");
+
+# Refresh
+$node_cloud->safe_psql('postgres', "ALTER SUBSCRIPTION remote_to_cloud REFRESH PUBLICATION;");
+$node_remote->safe_psql('postgres', "ALTER SUBSCRIPTION cloud_to_remote REFRESH PUBLICATION;");
+
+# Test BDR on new table
+$node_cloud->safe_psql('postgres',
+	"INSERT INTO docs (id, user_id, content, is_cloud) VALUES (2, 3, 'user3__doc2_on_cloud', TRUE);");
+$node_remote->safe_psql('postgres',
+	"INSERT INTO docs (id, user_id, content, is_cloud) VALUES (3, 3, 'user3__doc3_on_remote', FALSE);");
+
+check_data_consistency(
+	'check docs after insert',
+	"SELECT id, user_id, content, is_cloud FROM docs WHERE user_id = 3 ORDER BY id;",
+	qq(1|3|user3__doc1_on_cloud|t
+2|3|user3__doc2_on_cloud|t
+3|3|user3__doc3_on_remote|f)
+);
+
+# Test update of remote doc on cloud and vice versa
+$node_cloud->safe_psql('postgres',
+	"UPDATE docs SET content = 'user3__doc3_on_remote__updated', is_cloud = TRUE WHERE id = 3;");
+$node_remote->safe_psql('postgres',
+	"UPDATE docs SET content = 'user3__doc2_on_cloud__to_be_deleted', is_cloud = FALSE WHERE id = 2;");
+
+check_data_consistency(
+	'check docs after update',
+	"SELECT id, user_id, content, is_cloud FROM docs WHERE user_id = 3 ORDER BY id;",
+	qq(1|3|user3__doc1_on_cloud|t
+2|3|user3__doc2_on_cloud__to_be_deleted|f
+3|3|user3__doc3_on_remote__updated|t)
+);
+
+# Test delete
+$node_remote->safe_psql('postgres',
+	"DELETE FROM docs WHERE id = 2;");
+
+check_data_consistency(
+	'check docs after delete',
+	"SELECT id, user_id, content, is_cloud FROM docs WHERE user_id = 3 ORDER BY id;",
+	qq(1|3|user3__doc1_on_cloud|t
+3|3|user3__doc3_on_remote__updated|t)
+);
+
+$node_remote->stop('fast');
+$node_cloud->stop('fast');
-- 
2.19.1

---

From: Euler Taveira <[email-masked]>
Date: Sat, 31 Aug 2019 21:28:16 -0300

> marking this as returned with feedback for now. Please resubmit once
I fix all of the bugs pointed in this thread. I decide to disallow
UDFs in filters (it is safer for a first version). We can add this
functionality later. However, I'll check if allow "safe" functions
(aka builtin functions) are ok. I add more docs explaining that
expressions are executed with the role used for replication connection
and also that columns used in expressions must be part of PK or
REPLICA IDENTITY. I add regression tests.

Comments?

From 87945236590e9fd37b203d325b74dc5baccee64d Mon Sep 17 00:00:00 2001

 

2.11.0

From 3a5b4c541982357c2231b9882ac01f1f0d0a8e29 Mon Sep 17 00:00:00 2001
From: Euler Taveira <[email-masked]>
Date: Tue, 27 Feb 2018 02:21:03 +0000
Subject: [PATCH 3/8] Refactor function create_estate_for_relation

index 43edfef089..31fc7c5048 100644

 

 
2.11.0

From 7ef5ccffcb7bc71d298427e7b2c3a2cfae8556c6 Mon Sep 17 00:00:00 2001
From: Euler Taveira <[email-masked]>
Date: Wed, 24 Jan 2018 17:01:31 -0200
Subject: [PATCH 4/8] Rename a WHERE node

index 3dc0e8a4fb..61cc59fe7c 100644
 
-ExclusionWhereClause:
+OptWhereClause:
2.11.0

From 4b5ca55f83e8036d6892a458bf73c891329c01f8 Mon Sep 17 00:00:00 2001
From: Euler Taveira <[email-masked]>
Date: Fri, 9 Mar 2018 17:37:36 +0000
Subject: [PATCH 2/8] Store number of tuples in WalRcvExecResult

It seems to be a useful information while allocating memory for queries
that returns more than one row. It reduces memory allocation
for initial table synchronization.
---
 src/backend/replication/libpqwalreceiver/libpqwalreceiver.c | 5 +++--
 src/backend/replication/logical/tablesync.c                 | 5 ++---
 src/include/replication/walreceiver.h                       | 1 +
 3 files changed, 6 insertions(+), 5 deletions(-)

index 765d58d120..e657177c00 100644

 
 
 	/* Create temporary context for local allocations. */
@@ -897,7 +898,7 @@ libpqrcv_processTuples(PGresult *pgres, WalRcvExecResult *walres,
 									   ALLOCSET_DEFAULT_SIZES);
 
 	/* Process returned rows. */
-	for (tupn = 0; tupn < PQntuples(pgres); tupn++)
+	for (tupn = 0; tupn < walres->ntuples; tupn++)
 	{
 		char	   *cstrs[MaxTupleAttributeNumber];

 
 	natt = 0;
index 7f2927cb46..d0fb98df09 100644
 
2.11.0

From fc1090c6922d1a66d3ad03d21441829f8cae0472 Mon Sep 17 00:00:00 2001
From: Euler Taveira <[email-masked]>
Date: Tue, 27 Feb 2018 04:03:13 +0000
Subject: [PATCH 5/8] Row filtering for logical replication

When you define or modify a publication you optionally can filter rows
to be published using a WHERE condition. This condition is any
expression that evaluates to boolean. Only those rows that
satisfy the WHERE condition will be sent to subscribers.
---
 doc/src/sgml/catalogs.sgml                  |   9 +++
 doc/src/sgml/ref/alter_publication.sgml     |  11 ++-
 doc/src/sgml/ref/create_publication.sgml    |  26 +++++-
 src/backend/catalog/pg_publication.c        | 102 ++++++++++++++++++++++--
 src/backend/commands/publicationcmds.c      |  93 +++++++++++++++-------
 src/backend/parser/gram.y                   |  26 ++++--
 src/backend/parser/parse_agg.c              |  10 +++
 src/backend/parser/parse_expr.c             |  14 +++-
 src/backend/parser/parse_func.c             |   3 +
 src/backend/replication/logical/tablesync.c | 119 +++++++++++++++++++++++++---
 src/backend/replication/logical/worker.c    |   2 +-
 src/backend/replication/pgoutput/pgoutput.c | 100 ++++++++++++++++++++++-
 src/include/catalog/pg_publication.h        |   9 ++-
 src/include/catalog/pg_publication_rel.h    |  10 ++-
 src/include/catalog/toasting.h              |   1 +
 src/include/nodes/nodes.h                   |   1 +
 src/include/nodes/parsenodes.h              |  11 ++-
 src/include/parser/parse_node.h             |   1 +
 src/include/replication/logicalrelation.h   |   2 +
 src/test/regress/expected/publication.out   |  29 +++++++
 src/test/regress/sql/publication.sql        |  21 +++++
 src/test/subscription/t/013_row_filter.pl   |  96 ++++++++++++++++++++++
 22 files changed, 629 insertions(+), 67 deletions(-)
 create mode 100644 src/test/subscription/t/013_row_filter.pl

diff --git a/doc/src/sgml/catalogs.sgml b/doc/src/sgml/catalogs.sgml
index 4c7e93892a..88177279c7 100644
--- a/doc/src/sgml/catalogs.sgml
+++ b/doc/src/sgml/catalogs.sgml
@@ -5587,6 +5587,15 @@ SCRAM-SHA-256$<replaceable>&lt;iteration count&gt;</replaceable>:<replaceable>&l
       <entry><literal><link linkend="catalog-pg-class"><structname>pg_class</structname></link>.oid</literal></entry>
       <entry>Reference to relation</entry>
      </row>
+
+     <row>
+      <entry><structfield>prqual</structfield></entry>
+      <entry><type>pg_node_tree</type></entry>
+      <entry></entry>
+      <entry>Expression tree (in the form of a
+      <function>nodeToString()</function> representation) for the relation's
+      qualifying condition</entry>
+     </row>
     </tbody>
    </tgroup>
   </table>
diff --git a/doc/src/sgml/ref/alter_publication.sgml b/doc/src/sgml/ref/alter_publication.sgml
index 534e598d93..9608448207 100644
--- a/doc/src/sgml/ref/alter_publication.sgml
+++ b/doc/src/sgml/ref/alter_publication.sgml
@@ -21,8 +21,8 @@ PostgreSQL documentation
 
  <refsynopsisdiv>
 <synopsis>
@@ -91,7 +91,12 @@ ALTER PUBLICATION <replaceable class="parameter">name</replaceable> RENAME TO <r
+      not be published. Note that parentheses are required around the
+      expression. The <replaceable class="parameter">expression</replaceable>
+      is executed with the role used for the replication connection.
index 99f87ca393..6e99943374 100644

      <para>
@@ -157,6 +160,13 @@ CREATE PUBLICATION <replaceable class="parameter">name</replaceable>
   </para>
 
   <para>
+  Columns used in the <literal>WHERE</literal> clause must be part of the
+  primary key or be covered by <literal>REPLICA IDENTITY</literal> otherwise
+  <command>UPDATE</command> and <command>DELETE</command> operations will not
+  be replicated.
+  </para>
+
+  <para>
    For an <command>INSERT ... ON CONFLICT</command> command, the publication will
    publish the operation that actually results from the command.  So depending
    of the outcome, it may be published as either <command>INSERT</command> or
@@ -171,6 +181,11 @@ CREATE PUBLICATION <replaceable class="parameter">name</replaceable>
   <para>
    <acronym>DDL</acronym> operations are not published.
   </para>
+
+  <para>
+  The <literal>WHERE</literal> clause expression is executed with the role used
+  for the replication connection.
+  </para>
  </refsect1>
 
  <refsect1>
@@ -184,6 +199,13 @@ CREATE PUBLICATION mypublication FOR TABLE users, departments;
   </para>
 
   <para>
index f8475c1aba..ff30fdd9f6 100644

 

 	/* Form a tuple. */
 
+	/* Add qualifications, if available */
+	if (whereclause)
+		values[Anum_pg_publication_rel_prqual - 1] = CStringGetTextDatum(nodeToString(whereclause));
+	else
+		nulls[Anum_pg_publication_rel_prqual - 1] = true;
+
 	tup = heap_form_tuple(RelationGetDescr(rel), values, nulls);

+	/* Add dependency on the objects mentioned in the qualifications */
+	if (whereclause)
+		recordDependencyOnExpr(&myself, whereclause, pstate->p_rtable, DEPENDENCY_NORMAL);
+
+	free_parsestate(pstate);
+
 	/* Close the table. */
 	table_close(rel, RowExclusiveLock);

 	return myself;
 }
@@ -292,6 +328,62 @@ GetPublicationRelations(Oid pubid)
 }
 
 /*
+ * Gets list of PublicationRelationQuals for a publication.
+ */
+List *
+GetPublicationRelationQuals(Oid pubid)
+{
+	List	   *result;
+	Relation	pubrelsrel;
+	ScanKeyData scankey;
+	SysScanDesc scan;
+	HeapTuple	tup;
+
+	/* Find all publications associated with the relation. */
+	pubrelsrel = heap_open(PublicationRelRelationId, AccessShareLock);
+
+	ScanKeyInit(&scankey,
+				Anum_pg_publication_rel_prpubid,
+				BTEqualStrategyNumber, F_OIDEQ,
+				ObjectIdGetDatum(pubid));
+
+	scan = systable_beginscan(pubrelsrel, PublicationRelPrrelidPrpubidIndexId,
+							  true, NULL, 1, &scankey);
+
+	result = NIL;
+	while (HeapTupleIsValid(tup = systable_getnext(scan)))
+	{
+		Form_pg_publication_rel pubrel;
+		PublicationRelationQual	*relqual;
+		Datum	value_datum;
+		char	*qual_value;
+		Node	*qual_expr;
+		bool	isnull;
+
+		pubrel = (Form_pg_publication_rel) GETSTRUCT(tup);
+
+		value_datum = heap_getattr(tup, Anum_pg_publication_rel_prqual, RelationGetDescr(pubrelsrel), &isnull);
+		if (!isnull)
+		{
+			qual_value = TextDatumGetCString(value_datum);
+			qual_expr = (Node *) stringToNode(qual_value);
+		}
+		else
+			qual_expr = NULL;
+
+		relqual = palloc(sizeof(PublicationRelationQual));
+		relqual->relation = table_open(pubrel->prrelid, ShareUpdateExclusiveLock);
+		relqual->whereClause = copyObject(qual_expr);
+		result = lappend(result, relqual);
+	}
+
+	systable_endscan(scan);
+	heap_close(pubrelsrel, AccessShareLock);
+
+	return result;
+}
+
+/*
  * Gets list of publication oids for publications marked as FOR ALL TABLES.
  */
 List *
diff --git a/src/backend/commands/publicationcmds.c b/src/backend/commands/publicationcmds.c
index 4d48be0b92..6d56893c3e 100644
--- a/src/backend/commands/publicationcmds.c
+++ b/src/backend/commands/publicationcmds.c
@@ -344,6 +344,27 @@ AlterPublicationTables(AlterPublicationStmt *stmt, Relation rel,

+	/*
 
 	if (stmt->tableAction == DEFELEM_ADD)
@@ -352,47 +373,55 @@ AlterPublicationTables(AlterPublicationStmt *stmt, Relation rel,
 		PublicationDropTables(pubid, rels, false);
 	else						/* DEFELEM_SET */
 	{
-		List	   *oldrelids = GetPublicationRelations(pubid);
+		List	   *oldrels = GetPublicationRelationQuals(pubid);
 		List	   *delrels = NIL;
 		ListCell   *oldlc;
 
 		/* Calculate which relations to drop. */
-		foreach(oldlc, oldrelids)
+		foreach(oldlc, oldrels)
 		{
-			Oid			oldrelid = lfirst_oid(oldlc);
+			PublicationRelationQual *oldrel = lfirst(oldlc);
+			PublicationRelationQual *newrel;
 			ListCell   *newlc;
 			bool		found = false;
 
 			foreach(newlc, rels)
 			{
-				Relation	newrel = (Relation) lfirst(newlc);
+				newrel = (PublicationRelationQual *) lfirst(newlc);
 
-				if (RelationGetRelid(newrel) == oldrelid)
+				if (RelationGetRelid(newrel->relation) == RelationGetRelid(oldrel->relation))
 				{
 					found = true;
 					break;
 				}
 			}
 
-			if (!found)
+			/*
+			 * Remove publication / relation mapping iif (i) table is not found in
+			 * the new list or (ii) table is found in the new list, however,
+			 * its qual does not match the old one (in this case, a simple
+			 * tuple update is not enough because of the dependencies).
+			 */
+			if (!found || (found && !equal(oldrel->whereClause, newrel->whereClause)))
 			{
-				Relation	oldrel = table_open(oldrelid,
-												ShareUpdateExclusiveLock);
+				PublicationRelationQual *oldrelqual = palloc(sizeof(PublicationRelationQual));
+				oldrelqual->relation = table_open(RelationGetRelid(oldrel->relation),
+											   ShareUpdateExclusiveLock);
 
-				delrels = lappend(delrels, oldrel);
+				delrels = lappend(delrels, oldrelqual);
 			}
 		}
 
 		/* And drop them. */
 		PublicationDropTables(pubid, delrels, true);
+		CloseTableList(oldrels);
+		CloseTableList(delrels);
 
 		/*
 		 * Don't bother calculating the difference for adding, we'll catch and
 		 * skip existing ones when doing catalog update.
 		 */
 		PublicationAddTables(pubid, rels, true, stmt);
-
-		CloseTableList(delrels);
 	}
 
 	CloseTableList(rels);
@@ -502,16 +531,18 @@ OpenTableList(List *tables)
 	List	   *relids = NIL;
 	List	   *rels = NIL;
 	ListCell   *lc;
+	PublicationRelationQual	*relqual;
 
 	/*
+		RangeVar  			*rv = castNode(RangeVar, t->relation);
 
@@ -531,8 +562,10 @@ OpenTableList(List *tables)
 
 		/* Add children of this rel, if requested */
@@ -560,7 +593,11 @@ OpenTableList(List *tables)
 
@@ -581,10 +618,12 @@ CloseTableList(List *rels)
 
 	foreach(lc, rels)
 	{

 /*
@@ -600,13 +639,13 @@ PublicationAddTables(Oid pubid, List *rels, bool if_not_exists,
 
 	foreach(lc, rels)
 	{

 		obj = publication_add_relation(pubid, rel, if_not_exists);
 		if (stmt)
@@ -632,8 +671,8 @@ PublicationDropTables(Oid pubid, List *rels, bool missing_ok)
 
 	foreach(lc, rels)
 	{
 
@@ -646,7 +685,7 @@ PublicationDropTables(Oid pubid, List *rels, bool missing_ok)
 
index 61cc59fe7c..2580da9deb 100644

 
 publication_for_tables:
 
+publication_table_list:
+			publication_table_elem									{ $$ = list_make1($1); }
+			| publication_table_list ',' publication_table_elem		{ $$ = lappend($1, $3); }
+		;
+
+publication_table_elem: relation_expr OptWhereClause
+				{
+					PublicationTable *n = makeNode(PublicationTable);
+					n->relation = $1;
+					n->whereClause = $2;
+					$$ = (Node *) n;
+				}
+		;
+
 /*****************************************************************************
  *
  * CREATE SUBSCRIPTION name ...
diff --git a/src/backend/parser/parse_agg.c b/src/backend/parser/parse_agg.c
index c745fcdd2b..b11d159b54 100644
--- a/src/backend/parser/parse_agg.c
+++ b/src/backend/parser/parse_agg.c
@@ -544,6 +544,13 @@ check_agglevels_and_constraints(ParseState *pstate, Node *expr)
 				err = _("grouping operations are not allowed in COPY FROM WHERE conditions");
 
 			break;
 
 			/*
 
 			/*
 			 * There is intentionally no default: case here, so that the
diff --git a/src/backend/parser/parse_expr.c b/src/backend/parser/parse_expr.c
index 8e136a7981..f82518afc8 100644
--- a/src/backend/parser/parse_expr.c
+++ b/src/backend/parser/parse_expr.c
@@ -170,6 +170,13 @@ transformExprRecurse(ParseState *pstate, Node *expr)
 	/* Guard against stack overflow due to overly complex expressions */
 	check_stack_depth();
 
+	/* Functions are not allowed in publication WHERE clauses */
+	if (pstate->p_expr_kind == EXPR_KIND_PUBLICATION_WHERE && nodeTag(expr) == T_FuncCall)
+		ereport(ERROR,
+				(errcode(ERRCODE_FEATURE_NOT_SUPPORTED),
+				 errmsg("functions are not allowed in WHERE"),
+				 parser_errposition(pstate, exprLocation(expr))));
+
 	switch (nodeTag(expr))
 	{
 		case T_ColumnRef:
@@ -571,6 +578,7 @@ transformColumnRef(ParseState *pstate, ColumnRef *cref)
 		case EXPR_KIND_CALL_ARGUMENT:
 		case EXPR_KIND_COPY_WHERE:
 		case EXPR_KIND_GENERATED_COLUMN:
+		case EXPR_KIND_PUBLICATION_WHERE:
 			/* okay */
 			break;
 
@@ -1924,13 +1932,15 @@ transformSubLink(ParseState *pstate, SubLink *sublink)
 			break;
 		case EXPR_KIND_CALL_ARGUMENT:
 			err = _("cannot use subquery in CALL argument");
-			break;
 		case EXPR_KIND_COPY_WHERE:
 			err = _("cannot use subquery in COPY FROM WHERE condition");
 			break;
 		case EXPR_KIND_GENERATED_COLUMN:
 			err = _("cannot use subquery in column generation expression");
 			break;
+		case EXPR_KIND_PUBLICATION_WHERE:
+			err = _("cannot use subquery in publication WHERE expression");
+			break;
 
 			/*
@@ -3563,6 +3573,8 @@ ParseExprKindName(ParseExprKind exprKind)
 
 			/*
 			 * There is intentionally no default: case here, so that the
diff --git a/src/backend/parser/parse_func.c b/src/backend/parser/parse_func.c
index 752cf1b315..50653a89d8 100644
--- a/src/backend/parser/parse_func.c
+++ b/src/backend/parser/parse_func.c
@@ -2529,6 +2529,9 @@ check_srf_call_placement(ParseState *pstate, Node *last_srf, int location)
 		case EXPR_KIND_GENERATED_COLUMN:
 			err = _("set-returning functions are not allowed in column generation expressions");
 			break;
+		case EXPR_KIND_PUBLICATION_WHERE:
+			err = _("set-returning functions are not allowed in publication WHERE expressions");
+			break;
 
 			/*
 			 * There is intentionally no default: case here, so that the
diff --git a/src/backend/replication/logical/tablesync.c b/src/backend/replication/logical/tablesync.c
index 42db4ada9e..5468b694f6 100644
--- a/src/backend/replication/logical/tablesync.c
+++ b/src/backend/replication/logical/tablesync.c
@@ -637,19 +637,26 @@ copy_read_data(void *outbuf, int minread, int maxread)
 
 /*
  * Get information about remote relation in similar fashion the RELATION
- * message provides during replication.
+ * message provides during replication. This function also returns the relation
+ * qualifications to be used in COPY.
  */
 static void
 fetch_remote_table_info(char *nspname, char *relname,
-						LogicalRepRelation *lrel)
+						LogicalRepRelation *lrel, List **qual)
 {
 	WalRcvExecResult *res;
 	StringInfoData cmd;
 	TupleTableSlot *slot;
 	Oid			tableRow[2] = {OIDOID, CHAROID};
 	Oid			attrRow[3] = {TEXTOID, OIDOID, BOOLOID};
+	Oid			qualRow[1] = {TEXTOID};
 	bool		isnull;
-	int			natt;
+	int			n;
+	ListCell   *lc;
+	bool		first;
+
+	/* Avoid trashing relation map cache */
+	memset(lrel, 0, sizeof(LogicalRepRelation));
 
 	lrel->nspname = nspname;
 	lrel->relname = relname;
@@ -713,20 +720,20 @@ fetch_remote_table_info(char *nspname, char *relname,
 	lrel->atttyps = palloc0(res->ntuples * sizeof(Oid));
 	lrel->attkeys = NULL;
 
-	natt = 0;
+	n = 0;

@@ -734,7 +741,46 @@ fetch_remote_table_info(char *nspname, char *relname,
 	}
 	ExecDropSingleTupleTableSlot(slot);
 
-	lrel->natts = natt;
+	lrel->natts = n;
+
+	walrcv_clear_result(res);
+
+	/* Get relation qual */
+	resetStringInfo(&cmd);
+	appendStringInfo(&cmd, "SELECT pg_get_expr(prqual, prrelid) FROM pg_publication p INNER JOIN pg_publication_rel pr ON (p.oid = pr.prpubid) WHERE pr.prrelid = %u AND p.pubname IN (", lrel->remoteid);
+
+	first = true;
+	foreach(lc, MySubscription->publications)
+	{
+		char	*pubname = strVal(lfirst(lc));
+
+		if (first)
+			first = false;
+		else
+			appendStringInfoString(&cmd, ", ");
+
+		appendStringInfoString(&cmd, quote_literal_cstr(pubname));
+	}
+	appendStringInfoChar(&cmd, ')');
+
+	res = walrcv_exec(wrconn, cmd.data, 1, qualRow);
+
+	if (res->status != WALRCV_OK_TUPLES)
+		ereport(ERROR,
+				(errmsg("could not fetch relation qualifications for table \"%s.%s\" from publisher: %s",
+						nspname, relname, res->err)));
+
+	slot = MakeSingleTupleTableSlot(res->tupledesc, &TTSOpsMinimalTuple);
+	while (tuplestore_gettupleslot(res->tuplestore, true, false, slot))
+	{
+		Datum rf = slot_getattr(slot, 1, &isnull);
+
+		if (!isnull)
+			*qual = lappend(*qual, makeString(TextDatumGetCString(rf)));
+
+		ExecClearTuple(slot);
+	}
+	ExecDropSingleTupleTableSlot(slot);
 
 	walrcv_clear_result(res);
 	pfree(cmd.data);
@@ -750,6 +796,7 @@ copy_table(Relation rel)
 {
 	LogicalRepRelMapEntry *relmapentry;
 	LogicalRepRelation lrel;
+	List *qual = NIL;
 	WalRcvExecResult *res;
 	StringInfoData cmd;
 	CopyState	cstate;
@@ -758,7 +805,7 @@ copy_table(Relation rel)
 
 	/* Get the publisher relation info. */
 	fetch_remote_table_info(get_namespace_name(RelationGetNamespace(rel)),
-							RelationGetRelationName(rel), &lrel);
+							RelationGetRelationName(rel), &lrel, &qual);
 
 	/* Put the relation into relmap. */
 	logicalrep_relmap_update(&lrel);
@@ -767,10 +814,57 @@ copy_table(Relation rel)
 	relmapentry = logicalrep_rel_open(lrel.remoteid, NoLock);
 	Assert(rel == relmapentry->localrel);
 
+	 * filters for COPY. If no row filters are available, use COPY for all
+	if (list_length(qual) > 0)
+		foreach(lc, qual)
+			char	*q = strVal(lfirst(lc));
+			appendStringInfo(&cmd, "%s", q);
+		list_free_deep(qual);
@@ -785,7 +879,6 @@ copy_table(Relation rel)

 	/* Do the copy */
index 31fc7c5048..22b95d52b5 100644
index 63687a97ec..49f533280b 100644
@@ -12,13 +12,24 @@
  */
 #include "postgres.h"

 
@@ -58,6 +69,7 @@ typedef struct RelationSyncEntry
+	List		*qual;
 } RelationSyncEntry;
 
 /* Map used to remember which relation schemas we sent. */
@@ -333,6 +345,65 @@ pgoutput_change(LogicalDecodingContext *ctx, ReorderBufferTXN *txn,
 			Assert(false);
 	}
 
+	/* ... then check row filter */
+	if (list_length(relentry->qual) > 0)
+	{
+		HeapTuple		old_tuple;
+		HeapTuple		new_tuple;
+		TupleDesc		tupdesc;
+		EState			*estate;
+		ExprContext		*ecxt;
+		MemoryContext	oldcxt;
+		ListCell		*lc;
+		bool			matched = true;
+
+		old_tuple = change->data.tp.oldtuple ? &change->data.tp.oldtuple->tuple : NULL;
+		new_tuple = change->data.tp.newtuple ? &change->data.tp.newtuple->tuple : NULL;
+		tupdesc = RelationGetDescr(relation);
+		estate = create_estate_for_relation(relation);
+
+		/* prepare context per tuple */
+		ecxt = GetPerTupleExprContext(estate);
+		oldcxt = MemoryContextSwitchTo(estate->es_query_cxt);
+		ecxt->ecxt_scantuple = ExecInitExtraTupleSlot(estate, tupdesc, &TTSOpsVirtual);
+
+		ExecStoreHeapTuple(new_tuple ? new_tuple : old_tuple, ecxt->ecxt_scantuple, false);
+
+		foreach (lc, relentry->qual)
+		{
+			Node		*qual;
+			ExprState	*expr_state;
+			Expr		*expr;
+			Oid			expr_type;
+			Datum		res;
+			bool		isnull;
+
+			qual = (Node *) lfirst(lc);
+
+			/* evaluates row filter */
+			expr_type = exprType(qual);
+			expr = (Expr *) coerce_to_target_type(NULL, qual, expr_type, BOOLOID, -1, COERCION_ASSIGNMENT, COERCE_IMPLICIT_CAST, -1);
+			expr = expression_planner(expr);
+			expr_state = ExecInitExpr(expr, NULL);
+			res = ExecEvalExpr(expr_state, ecxt, &isnull);
+
+			/* if tuple does not match row filter, bail out */
+			if (!DatumGetBool(res) || isnull)
+			{
+				matched = false;
+				break;
+			}
+		}
+
+		MemoryContextSwitchTo(oldcxt);
+
+		ExecDropSingleTupleTableSlot(ecxt->ecxt_scantuple);
+		FreeExecutorState(estate);
+
+		if (!matched)
+			return;
+	}
+
 	/* Avoid leaking memory by using and resetting our own context */
 	old = MemoryContextSwitchTo(data->context);
 
@@ -568,10 +639,14 @@ get_rel_sync_entry(PGOutputData *data, Oid relid)
 		 */
 		entry->pubactions.pubinsert = entry->pubactions.pubupdate =
 			entry->pubactions.pubdelete = entry->pubactions.pubtruncate = false;
+		entry->qual = NIL;

 			if (pub->alltables || list_member_oid(pubids, pub->oid))
 			{
@@ -581,9 +656,23 @@ get_rel_sync_entry(PGOutputData *data, Oid relid)
 				entry->pubactions.pubtruncate |= pub->pubactions.pubtruncate;
 			}
 
+				rf_datum = SysCacheGetAttr(PUBLICATIONRELMAP, rf_tuple, Anum_pg_publication_rel_prqual, &rf_isnull);
+					entry->qual = lappend(entry->qual, rf_node);
 
 		list_free(pubids);
@@ -658,5 +747,10 @@ rel_sync_cache_publication_cb(Datum arg, int cacheid, uint32 hashvalue)
 	 */
 	hash_seq_init(&status, RelationSyncCache);
 	while ((entry = (RelationSyncEntry *) hash_seq_search(&status)) != NULL)
+	{
 		entry->replicate_valid = false;
+		if (list_length(entry->qual) > 0)
+			list_free_deep(entry->qual);
+		entry->qual = NIL;
+	}
 }
diff --git a/src/include/catalog/pg_publication.h b/src/include/catalog/pg_publication.h
index 2dad24fc9f..74ab2c25d1 100644
--- a/src/include/catalog/pg_publication.h
+++ b/src/include/catalog/pg_publication.h
@@ -78,15 +78,22 @@ typedef struct Publication
 	PublicationActions pubactions;
 } Publication;
 
+typedef struct PublicationRelationQual
+{
+	Relation	relation;
+	Node		*whereClause;
+} PublicationRelationQual;
+
 extern Publication *GetPublication(Oid pubid);
 extern Publication *GetPublicationByName(const char *pubname, bool missing_ok);
 extern List *GetRelationPublications(Oid relid);
 extern List *GetPublicationRelations(Oid pubid);
+extern List *GetPublicationRelationQuals(Oid pubid);
 extern List *GetAllTablesPublications(void);
 extern List *GetAllTablesPublicationRelations(void);

index 5f5bc92ab3..a75b2d5345 100644
+	pg_node_tree	prqual;			/* qualifications */
 
 /* ----------------
diff --git a/src/include/catalog/toasting.h b/src/include/catalog/toasting.h
index 5ee628c837..aedf27b483 100644
--- a/src/include/catalog/toasting.h
+++ b/src/include/catalog/toasting.h
@@ -66,6 +66,7 @@ DECLARE_TOAST(pg_namespace, 4163, 4164);
 DECLARE_TOAST(pg_partitioned_table, 4165, 4166);
 DECLARE_TOAST(pg_policy, 4167, 4168);
 DECLARE_TOAST(pg_proc, 2836, 2837);
+DECLARE_TOAST(pg_publication_rel, 8287, 8288);
 DECLARE_TOAST(pg_rewrite, 2838, 2839);
 DECLARE_TOAST(pg_seclabel, 3598, 3599);
 DECLARE_TOAST(pg_statistic, 2840, 2841);
diff --git a/src/include/nodes/nodes.h b/src/include/nodes/nodes.h
index ffb4cd4bcc..4e624317b0 100644
--- a/src/include/nodes/nodes.h
+++ b/src/include/nodes/nodes.h
@@ -476,6 +476,7 @@ typedef enum NodeTag
 	T_PartitionRangeDatum,
 	T_PartitionCmd,
 	T_VacuumRelation,
+	T_PublicationTable,
 
 	/*
 	 * TAGS FOR REPLICATION GRAMMAR PARSE NODES (replnodes.h)
diff --git a/src/include/nodes/parsenodes.h b/src/include/nodes/parsenodes.h
index 12e9730dd0..91cd750047 100644
--- a/src/include/nodes/parsenodes.h
+++ b/src/include/nodes/parsenodes.h
@@ -3462,12 +3462,19 @@ typedef struct AlterTSConfigurationStmt
 } AlterTSConfigurationStmt;

@@ -3480,7 +3487,7 @@ typedef struct AlterPublicationStmt
 	List	   *options;		/* List of DefElem nodes */
 
index 3d8039aa51..048a445030 100644

index 85e0b6ea62..29af52ce3a 100644
 
+extern EState *create_estate_for_relation(Relation rel);
+
 #endif							/* LOGICALRELATION_H */
diff --git a/src/test/regress/expected/publication.out b/src/test/regress/expected/publication.out
index afbbdd543d..cf67b7b186 100644
--- a/src/test/regress/expected/publication.out
+++ b/src/test/regress/expected/publication.out
@@ -107,6 +107,35 @@ Tables:
 
 DROP TABLE testpub_tbl3, testpub_tbl3a;
 DROP PUBLICATION testpub3, testpub4;
+CREATE TABLE testpub_rf_tbl1 (a integer, b text);
+CREATE TABLE testpub_rf_tbl2 (c text, d integer);
+CREATE TABLE testpub_rf_tbl3 (e integer);
+CREATE TABLE testpub_rf_tbl4 (g text);
+SET client_min_messages = 'ERROR';
+CREATE PUBLICATION testpub5 FOR TABLE testpub_rf_tbl1, testpub_rf_tbl2 WHERE (c <> 'test' AND d < 5);
+RESET client_min_messages;
+ALTER PUBLICATION testpub5 ADD TABLE testpub_rf_tbl3 WHERE (e > 1000 AND e < 2000);
+ALTER PUBLICATION testpub5 DROP TABLE testpub_rf_tbl2;
+-- remove testpub_rf_tbl1 and add testpub_rf_tbl3 again (another WHERE expression)
+ALTER PUBLICATION testpub5 SET TABLE testpub_rf_tbl3 WHERE (e > 300 AND e < 500);
+-- fail - functions disallowed
+ALTER PUBLICATION testpub5 ADD TABLE testpub_rf_tbl4 WHERE (length(g) < 6);
+ERROR:  functions are not allowed in WHERE
+LINE 1: ...ICATION testpub5 ADD TABLE testpub_rf_tbl4 WHERE (length(g) ...
+                                                             ^
+\dRp+ testpub5
+                              Publication testpub5
+          Owner           | All tables | Inserts | Updates | Deletes | Truncates 
+--------------------------+------------+---------+---------+---------+-----------
+ regress_publication_user | f          | t       | t       | t       | t
+Tables:
+    "public.testpub_rf_tbl3"  WHERE ((e > 300) AND (e < 500))
+
+DROP TABLE testpub_rf_tbl1;
+DROP TABLE testpub_rf_tbl2;
+DROP TABLE testpub_rf_tbl3;
+DROP TABLE testpub_rf_tbl4;
+DROP PUBLICATION testpub5;
 -- fail - view
 CREATE PUBLICATION testpub_fortbl FOR TABLE testpub_view;
 ERROR:  "testpub_view" is not a table
diff --git a/src/test/regress/sql/publication.sql b/src/test/regress/sql/publication.sql
index 815410b3c5..20c874eb67 100644
--- a/src/test/regress/sql/publication.sql
+++ b/src/test/regress/sql/publication.sql
@@ -60,6 +60,27 @@ CREATE PUBLICATION testpub4 FOR TABLE ONLY testpub_tbl3;
 DROP TABLE testpub_tbl3, testpub_tbl3a;
 DROP PUBLICATION testpub3, testpub4;
 
+CREATE TABLE testpub_rf_tbl1 (a integer, b text);
+CREATE TABLE testpub_rf_tbl2 (c text, d integer);
+CREATE TABLE testpub_rf_tbl3 (e integer);
+CREATE TABLE testpub_rf_tbl4 (g text);
+SET client_min_messages = 'ERROR';
+CREATE PUBLICATION testpub5 FOR TABLE testpub_rf_tbl1, testpub_rf_tbl2 WHERE (c <> 'test' AND d < 5);
+RESET client_min_messages;
+ALTER PUBLICATION testpub5 ADD TABLE testpub_rf_tbl3 WHERE (e > 1000 AND e < 2000);
+ALTER PUBLICATION testpub5 DROP TABLE testpub_rf_tbl2;
+-- remove testpub_rf_tbl1 and add testpub_rf_tbl3 again (another WHERE expression)
+ALTER PUBLICATION testpub5 SET TABLE testpub_rf_tbl3 WHERE (e > 300 AND e < 500);
+-- fail - functions disallowed
+ALTER PUBLICATION testpub5 ADD TABLE testpub_rf_tbl4 WHERE (length(g) < 6);
+\dRp+ testpub5
+
+DROP TABLE testpub_rf_tbl1;
+DROP TABLE testpub_rf_tbl2;
+DROP TABLE testpub_rf_tbl3;
+DROP TABLE testpub_rf_tbl4;
+DROP PUBLICATION testpub5;
+
 -- fail - view
 CREATE PUBLICATION testpub_fortbl FOR TABLE testpub_view;
 CREATE PUBLICATION testpub_fortbl FOR TABLE testpub_tbl1, pub_test.testpub_nopk;
diff --git a/src/test/subscription/t/013_row_filter.pl b/src/test/subscription/t/013_row_filter.pl
new file mode 100644
index 0000000000..99e6db94d6
--- /dev/null
+++ b/src/test/subscription/t/013_row_filter.pl
@@ -0,0 +1,96 @@
+# Test logical replication behavior with row filtering
+use strict;
+use warnings;
+use PostgresNode;
+use TestLib;
+use Test::More tests => 4;
+
+# create publisher node
+my $node_publisher = get_new_node('publisher');
+$node_publisher->init(allows_streaming => 'logical');
+$node_publisher->start;
+
+# create subscriber node
+my $node_subscriber = get_new_node('subscriber');
+$node_subscriber->init(allows_streaming => 'logical');
+$node_subscriber->start;
+
+# setup structure on publisher
+$node_publisher->safe_psql('postgres',
+	"CREATE TABLE tab_rowfilter_1 (a int primary key, b text)");
+$node_publisher->safe_psql('postgres',
+	"CREATE TABLE tab_rowfilter_2 (c int primary key)");
+$node_publisher->safe_psql('postgres',
+	"CREATE TABLE tab_rowfilter_3 (a int primary key, b boolean)");
+
+# setup structure on subscriber
+$node_subscriber->safe_psql('postgres',
+	"CREATE TABLE tab_rowfilter_1 (a int primary key, b text)");
+$node_subscriber->safe_psql('postgres',
+	"CREATE TABLE tab_rowfilter_2 (c int primary key)");
+$node_subscriber->safe_psql('postgres',
+	"CREATE TABLE tab_rowfilter_3 (a int primary key, b boolean)");
+
+# setup logical replication
+$node_publisher->safe_psql('postgres',
+	"CREATE PUBLICATION tap_pub_1 FOR TABLE tab_rowfilter_1 WHERE (a > 1000 AND b <> 'filtered')");
+
+my $result = $node_publisher->psql('postgres',
+	"ALTER PUBLICATION tap_pub_1 DROP TABLE tab_rowfilter_1 WHERE (a > 1000 AND b <> 'filtered')");
+is($result, 3, "syntax error for ALTER PUBLICATION DROP TABLE");
+
+$node_publisher->safe_psql('postgres',
+	"ALTER PUBLICATION tap_pub_1 ADD TABLE tab_rowfilter_2 WHERE (c % 7 = 0)");
+
+$node_publisher->safe_psql('postgres',
+	"ALTER PUBLICATION tap_pub_1 SET TABLE tab_rowfilter_1 WHERE (a > 1000 AND b <> 'filtered'), tab_rowfilter_2 WHERE (c % 2 = 0), tab_rowfilter_3");
+
+$node_publisher->safe_psql('postgres',
+	"CREATE PUBLICATION tap_pub_2 FOR TABLE tab_rowfilter_2 WHERE (c % 3 = 0)");
+
+# test row filtering
+$node_publisher->safe_psql('postgres',
+	"INSERT INTO tab_rowfilter_1 (a, b) VALUES (1, 'not replicated')");
+$node_publisher->safe_psql('postgres',
+	"INSERT INTO tab_rowfilter_1 (a, b) VALUES (1500, 'filtered')");
+$node_publisher->safe_psql('postgres',
+	"INSERT INTO tab_rowfilter_1 (a, b) VALUES (1980, 'not filtered')");
+$node_publisher->safe_psql('postgres',
+	"INSERT INTO tab_rowfilter_1 (a, b) SELECT x, 'test ' || x FROM generate_series(990,1002) x");
+$node_publisher->safe_psql('postgres',
+	"INSERT INTO tab_rowfilter_2 (c) SELECT generate_series(1, 10)");
+$node_publisher->safe_psql('postgres',
+	"INSERT INTO tab_rowfilter_3 (a, b) SELECT x, (x % 3 = 0) FROM generate_series(1, 10) x");
+
+my $publisher_connstr = $node_publisher->connstr . ' dbname=postgres';
+my $appname = 'tap_sub';
+$node_subscriber->safe_psql('postgres',
+"CREATE SUBSCRIPTION tap_sub CONNECTION '$publisher_connstr application_name=$appname' PUBLICATION tap_pub_1, tap_pub_2"
+);
+
+$node_publisher->wait_for_catchup($appname);
+
+# wait for initial table sync to finish
+my $synced_query =
+"SELECT count(1) = 0 FROM pg_subscription_rel WHERE srsubstate NOT IN ('r', 's');";
+$node_subscriber->poll_query_until('postgres', $synced_query)
+  or die "Timed out while waiting for subscriber to synchronize data";
+
+#$node_publisher->wait_for_catchup($appname);
+
+$result =
+  $node_subscriber->safe_psql('postgres', "SELECT a, b FROM tab_rowfilter_1");
+is($result, qq(1980|not filtered
+1001|test 1001
+1002|test 1002), 'check filtered data was copied to subscriber');
+
+$result =
+  $node_subscriber->safe_psql('postgres', "SELECT count(c), min(c), max(c) FROM tab_rowfilter_2");
+is($result, qq(7|2|10), 'check filtered data was copied to subscriber');
+
+$result =
+  $node_subscriber->safe_psql('postgres', "SELECT count(a) FROM tab_rowfilter_3");
+is($result, qq(10), 'check filtered data was copied to subscriber');
+
+$node_subscriber->stop('fast');
+$node_publisher->stop('fast');
-- 
2.11.0

From 2548f727b6f5d61e87de70ec661335fe82ce6ffa Mon Sep 17 00:00:00 2001
From: Euler Taveira <[email-masked]>
Date: Sat, 15 Sep 2018 02:52:00 +0000
Subject: [PATCH 7/8] Publication where condition support for pg_dump

---

diff --git a/src/bin/pg_dump/pg_dump.c b/src/bin/pg_dump/pg_dump.c
index 8b993d6eae..b41d9fd477 100644
--- a/src/bin/pg_dump/pg_dump.c
+++ b/src/bin/pg_dump/pg_dump.c
@@ -3930,6 +3930,7 @@ getPublicationTables(Archive *fout, TableInfo tblinfo[], int numTables)
 	int			i_tableoid;
 	int			i_oid;
 	int			i_pubname;
+	int			i_pubrelqual;
 	int			i,
 				j,
 				ntups;
@@ -3962,7 +3963,8 @@ getPublicationTables(Archive *fout, TableInfo tblinfo[], int numTables)
 
+						  "pg_catalog.pg_get_expr(pr.prqual, pr.prrelid) AS pubrelqual "
@@ -3983,6 +3985,7 @@ getPublicationTables(Archive *fout, TableInfo tblinfo[], int numTables)

@@ -3998,6 +4001,11 @@ getPublicationTables(Archive *fout, TableInfo tblinfo[], int numTables)
 			pubrinfo[j].pubname = pg_strdup(PQgetvalue(res, j, i_pubname));
 			pubrinfo[j].pubtable = tbinfo;
 
@@ -4026,8 +4034,11 @@ dumpPublicationTable(Archive *fout, PublicationRelInfo *pubrinfo)

 	/*
 	 * There is no point in creating drop query as drop query as the drop is
diff --git a/src/bin/pg_dump/pg_dump.h b/src/bin/pg_dump/pg_dump.h
index 4f9ebb4904..d03eaa1dca 100644
--- a/src/bin/pg_dump/pg_dump.h
+++ b/src/bin/pg_dump/pg_dump.h
@@ -612,6 +612,7 @@ typedef struct _PublicationRelInfo
 	DumpableObject dobj;
 	TableInfo  *pubtable;
 	char	   *pubname;
+	char	   *pubrelqual;
 } PublicationRelInfo;
 
 /*
-- 
2.11.0

From 72cfcf940ae2100f0bf8728983bf7313a2d9ef83 Mon Sep 17 00:00:00 2001
From: Euler Taveira <[email-masked]>
Date: Thu, 17 May 2018 20:52:28 +0000
Subject: [PATCH 6/8] Print publication WHERE condition in psql

---

diff --git a/src/bin/psql/describe.c b/src/bin/psql/describe.c
index ee00c5da08..872a544410 100644
--- a/src/bin/psql/describe.c
+++ b/src/bin/psql/describe.c
@@ -5816,7 +5816,8 @@ describePublications(const char *pattern)
 		if (!puballtables)
 		{
 			printfPQExpBuffer(&buf,
-							  "SELECT n.nspname, c.relname\n"
+							  "SELECT n.nspname, c.relname,\n"
+							  "  pg_get_expr(pr.prqual, c.oid)\n"
 							  "FROM pg_catalog.pg_class c,\n"
 							  "     pg_catalog.pg_namespace n,\n"
 							  "     pg_catalog.pg_publication_rel pr\n"
@@ -5846,6 +5847,10 @@ describePublications(const char *pattern)
 								  PQgetvalue(tabres, j, 0),
 								  PQgetvalue(tabres, j, 1));
 
2.11.0

From c947168de90527d30d850f8c5c4bd6e090521500 Mon Sep 17 00:00:00 2001
From: Euler Taveira <[email-masked]>
Date: Wed, 14 Mar 2018 00:53:17 +0000
Subject: [PATCH 8/8] Debug for row filtering

---

diff --git a/src/backend/commands/publicationcmds.c b/src/backend/commands/publicationcmds.c
index 6d56893c3e..65294f2100 100644
--- a/src/backend/commands/publicationcmds.c
+++ b/src/backend/commands/publicationcmds.c
@@ -333,6 +333,7 @@ AlterPublicationTables(AlterPublicationStmt *stmt, Relation rel,
 	List	   *rels = NIL;
 	Form_pg_publication pubform = (Form_pg_publication) GETSTRUCT(tup);
 	Oid			pubid = pubform->oid;
+	ListCell	*lc;
 
 	/* Check that user is allowed to manipulate the publication tables. */
 	if (pubform->puballtables)
@@ -344,6 +345,16 @@ AlterPublicationTables(AlterPublicationStmt *stmt, Relation rel,

index 5468b694f6..1fc7d5647b 100644
@@ -865,6 +865,7 @@ copy_table(Relation rel)
index 49f533280b..4aff5cb515 100644
@@ -32,6 +32,7 @@
@@ -321,6 +322,10 @@ pgoutput_change(LogicalDecodingContext *ctx, ReorderBufferTXN *txn,

@@ -345,6 +350,17 @@ pgoutput_change(LogicalDecodingContext *ctx, ReorderBufferTXN *txn,
 			Assert(false);
 	}
 
 	if (list_length(relentry->qual) > 0)
@@ -362,6 +378,42 @@ pgoutput_change(LogicalDecodingContext *ctx, ReorderBufferTXN *txn,
 
+#ifdef	_NOT_USED
+		if (old_tuple)
+		{
+			int i;
+
@@ -377,6 +429,7 @@ pgoutput_change(LogicalDecodingContext *ctx, ReorderBufferTXN *txn,
 			Oid			expr_type;
 			Datum		res;
 			bool		isnull;
+			char		*s = NULL;
 
 			qual = (Node *) lfirst(lc);
 
@@ -387,12 +440,22 @@ pgoutput_change(LogicalDecodingContext *ctx, ReorderBufferTXN *txn,
 			expr_state = ExecInitExpr(expr, NULL);
 			res = ExecEvalExpr(expr_state, ecxt, &isnull);
 
+				s = TextDatumGetCString(DirectFunctionCall2(pg_get_expr, CStringGetTextDatum(nodeToString(qual)), ObjectIdGetDatum(relentry->relid)));
 				matched = false;
+			s = TextDatumGetCString(DirectFunctionCall2(pg_get_expr, CStringGetTextDatum(nodeToString(qual)), ObjectIdGetDatum(relentry->relid)));
 
 					entry->qual = lappend(entry->qual, rf_node);
 
2.11.0

---

From: Euler Taveira <[email-masked]>
Date: Sat, 31 Aug 2019 21:31:52 -0300

> submit it, though I didn't yet dig deeply into the code.
Sure. See my last email to this thread. I appreciate if you can review it.

> then updated with is_cloud=FALSE on remote, replicated to cloud; then
That's because you don't include is_cloud in PK or REPLICA IDENTITY. I
add a small note in docs.

---

From: Alexey Zagarin <[email-masked]>
Date: Sun, 1 Sep 2019 12:25:41 +0700

I think that I also have found one shortcoming when using the setup described by Alexey Kondratov. The problem that I face is that if both (cloud and remote) tables already have data the moment I add the subscription, then the whole table is copied in both directions initially. Which leads to duplicated data and broken replication because COPY doesn't take into account the filtering condition. In case there are filters in a publication, the COPY command that is executed when adding a subscription (or altering one to refresh a publication) should also filter the data based on the same condition, e.g. COPY (SELECT * FROM ... WHERE ...) TO ...

The current workaround is to always use WITH copy_data = false when subscribing or refreshing, and then manually copy data with the above statement.

Alexey Zagarin

---

From: Erik Rijkers <[email-masked]>
Date: Sun, 01 Sep 2019 11:09:11 +0200

> 0001-Remove-unused-atttypmod-column-from-initial-table-sy.patch
> 0002-Store-number-of-tuples-in-WalRcvExecResult.patch
> 0003-Refactor-function-create_estate_for_relation.patch
> 0004-Rename-a-WHERE-node.patch
> 0005-Row-filtering-for-logical-replication.patch
> 0006-Print-publication-WHERE-condition-in-psql.patch
> 0007-Publication-where-condition-support-for-pg_dump.patch
> 0008-Debug-for-row-filtering.patch

Hi,

The first 4 of these apply without error, but I can't get 0005 to apply. 
This is what I use:

patch --dry-run -b -l -F 5 -p 1 < 
/home/aardvark/download/pgpatches/0130/logrep_rowfilter/20190901/0005-Row-filtering-for-logical-replication.patch

checking file doc/src/sgml/catalogs.sgml
Hunk #1 succeeded at 5595 (offset 8 lines).
checking file doc/src/sgml/ref/alter_publication.sgml
checking file doc/src/sgml/ref/create_publication.sgml
checking file src/backend/catalog/pg_publication.c
checking file src/backend/commands/publicationcmds.c
Hunk #1 succeeded at 352 (offset 8 lines).
Hunk #2 succeeded at 381 (offset 8 lines).
Hunk #3 succeeded at 539 (offset 8 lines).
Hunk #4 succeeded at 570 (offset 8 lines).
Hunk #5 succeeded at 601 (offset 8 lines).
Hunk #6 succeeded at 626 (offset 8 lines).
Hunk #7 succeeded at 647 (offset 8 lines).
Hunk #8 succeeded at 679 (offset 8 lines).
Hunk #9 succeeded at 693 (offset 8 lines).
checking file src/backend/parser/gram.y
checking file src/backend/parser/parse_agg.c
checking file src/backend/parser/parse_expr.c
Hunk #4 succeeded at 3571 (offset -2 lines).
checking file src/backend/parser/parse_func.c
Hunk #1 succeeded at 2516 (offset -13 lines).
checking file src/backend/replication/logical/tablesync.c
checking file src/backend/replication/logical/worker.c
checking file src/backend/replication/pgoutput/pgoutput.c
Hunk #1 FAILED at 12.
Hunk #2 succeeded at 60 (offset 2 lines).
Hunk #3 succeeded at 336 (offset 2 lines).
Hunk #4 succeeded at 630 (offset 2 lines).
Hunk #5 succeeded at 647 (offset 2 lines).
Hunk #6 succeeded at 738 (offset 2 lines).
1 out of 6 hunks FAILED
checking file src/include/catalog/pg_publication.h
checking file src/include/catalog/pg_publication_rel.h
checking file src/include/catalog/toasting.h
checking file src/include/nodes/nodes.h
checking file src/include/nodes/parsenodes.h
Hunk #1 succeeded at 3461 (offset -1 lines).
Hunk #2 succeeded at 3486 (offset -1 lines).
checking file src/include/parser/parse_node.h
checking file src/include/replication/logicalrelation.h
checking file src/test/regress/expected/publication.out
Hunk #1 succeeded at 116 (offset 9 lines).
checking file src/test/regress/sql/publication.sql
Hunk #1 succeeded at 69 with fuzz 1 (offset 9 lines).
checking file src/test/subscription/t/013_row_filter.pl

perhaps that can be fixed?

thanks,

Erik Rijkers

---

From: Euler Taveira <[email-masked]>
Date: Sun, 1 Sep 2019 20:43:57 -0300

> The first 4 of these apply without error, but I can't get 0005 to apply.
Erik, I generate a new patch set with patience diff algorithm. It
seems it applies cleanly.

From c07af2f00b7a72ba9660e389bb1392fc9e5d2688 Mon Sep 17 00:00:00 2001

 
-ExclusionWhereClause:
+OptWhereClause:

From 367631ac4ba1e41170d59d39693e2eaf7c406621 Mon Sep 17 00:00:00 2001
From: Euler Taveira <[email-masked]>
Date: Tue, 27 Feb 2018 02:21:03 +0000
Subject: [PATCH 3/8] Refactor function create_estate_for_relation

 

 

From e83de1cab559f4ca8f9a75356e220356814cd243 Mon Sep 17 00:00:00 2001
From: Euler Taveira <[email-masked]>
Date: Fri, 9 Mar 2018 18:39:22 +0000
Subject: [PATCH 1/8] Remove unused atttypmod column from initial table
 synchronization

 

From f6795f1b6efab48caa07487d9186f844dd48fc65 Mon Sep 17 00:00:00 2001
From: Euler Taveira <[email-masked]>
Date: Fri, 9 Mar 2018 17:37:36 +0000
Subject: [PATCH 2/8] Store number of tuples in WalRcvExecResult

index 6eba08a920..343550a335 100644

 

 

 	natt = 0;

From 44f401ecb53dd43e3a7a05912e25d644748f936f Mon Sep 17 00:00:00 2001
From: Euler Taveira <[email-masked]>
Date: Tue, 27 Feb 2018 04:03:13 +0000
Subject: [PATCH 5/8] Row filtering for logical replication

index 5e71a2e865..7f11225f65 100644
@@ -5595,6 +5595,15 @@ SCRAM-SHA-256$<replaceable>&lt;iteration count&gt;</replaceable>:<replaceable>&l
 
  <refsynopsisdiv>
 <synopsis>

      <para>
 
   <para>
 
  <refsect1>
 
   <para>
index fd5da7d5f7..c873419a9e 100644

 

 	/* Form a tuple. */

 

 	return myself;
 }
 
 /*
index f115d4bf80..716ed2ec58 100644

+	/*
 
@@ -360,47 +381,55 @@ AlterPublicationTables(AlterPublicationStmt *stmt, Relation rel,

 			foreach(newlc, rels)
 			{

-			if (!found)
+			/*

 		/* And drop them. */
 
 		/*
 
 	CloseTableList(rels);
@@ -510,16 +539,18 @@ OpenTableList(List *tables)
 	List	   *relids = NIL;
 	List	   *rels = NIL;
 	ListCell   *lc;
+	PublicationRelationQual	*relqual;
 
 	/*
 
@@ -539,8 +570,10 @@ OpenTableList(List *tables)
 
 		/* Add children of this rel, if requested */
@@ -568,7 +601,11 @@ OpenTableList(List *tables)
 
@@ -589,10 +626,12 @@ CloseTableList(List *rels)
 
 	foreach(lc, rels)
 	{

 /*
@@ -608,13 +647,13 @@ PublicationAddTables(Oid pubid, List *rels, bool if_not_exists,
 
 	foreach(lc, rels)
 	{

 		obj = publication_add_relation(pubid, rel, if_not_exists);
 		if (stmt)
@@ -640,8 +679,8 @@ PublicationDropTables(Oid pubid, List *rels, bool missing_ok)
 
 	foreach(lc, rels)
 	{
 
@@ -654,7 +693,7 @@ PublicationDropTables(Oid pubid, List *rels, bool missing_ok)

 
 
 publication_for_tables:
 
+publication_table_list:
index f418c61545..dea5aadca7 100644
 
 			break;
 
 			/*
 
 			/*
index 76f3dd7076..6d2c6a28ea 100644

 
 			/*
@@ -3561,6 +3571,8 @@ ParseExprKindName(ParseExprKind exprKind)
 
 			/*
index 8e926539e6..66458d8a48 100644
@@ -2516,6 +2516,9 @@ check_srf_call_placement(ParseState *pstate, Node *last_srf, int location)
 
 			/*
 
 /*
 
 	lrel->nspname = nspname;
 	lrel->relname = relname;
 
-	natt = 0;
+	n = 0;

 
-	lrel->natts = natt;
+	lrel->natts = n;
+

 

 
 	/* Do the copy */
index 9c08757fca..83fad465f8 100644

 #include "fmgr.h"

 
 /* Map used to remember which relation schemas we sent. */
@@ -335,6 +347,65 @@ pgoutput_change(LogicalDecodingContext *ctx, ReorderBufferTXN *txn,
 			Assert(false);
 	}

@@ -570,10 +641,14 @@ get_rel_sync_entry(PGOutputData *data, Oid relid)
 		 */
 		entry->pubactions.pubinsert = entry->pubactions.pubupdate =
 			entry->pubactions.pubdelete = entry->pubactions.pubtruncate = false;
+		entry->qual = NIL;

 			if (pub->alltables || list_member_oid(pubids, pub->oid))
 			{
@@ -583,9 +658,23 @@ get_rel_sync_entry(PGOutputData *data, Oid relid)
 				entry->pubactions.pubtruncate |= pub->pubactions.pubtruncate;
 			}

 		list_free(pubids);
@@ -660,5 +749,10 @@ rel_sync_cache_publication_cb(Datum arg, int cacheid, uint32 hashvalue)
 	 */
index 20a2f0ac1b..ae8d2845a0 100644

 
 
 /* ----------------
index cc5dfed0bf..d57ca82e89 100644
 
 	/*

 

+extern EState *create_estate_for_relation(Relation rel);
+
 #endif							/* LOGICALRELATION_H */
diff --git a/src/test/regress/expected/publication.out b/src/test/regress/expected/publication.out
index feb51e4add..202173c376 100644
--- a/src/test/regress/expected/publication.out
+++ b/src/test/regress/expected/publication.out
@@ -116,6 +116,35 @@ Tables:
 
index 5773a755cf..6f0d088984 100644
@@ -69,6 +69,27 @@ RESET client_min_messages;
 
 SET client_min_messages = 'ERROR';

From 8558d7b9913311255b0a4f349ebc0f60f6a83393 Mon Sep 17 00:00:00 2001
From: Euler Taveira <[email-masked]>
Date: Thu, 17 May 2018 20:52:28 +0000
Subject: [PATCH 6/8] Print publication WHERE condition in psql

---

index 774cc764ff..819b74bf4c 100644

From df95a451055ae5325a7494b355f2ef3383dbf304 Mon Sep 17 00:00:00 2001
From: Euler Taveira <[email-masked]>
Date: Sat, 15 Sep 2018 02:52:00 +0000
Subject: [PATCH 7/8] Publication where condition support for pg_dump

---

index 34981401bf..0507441209 100644

 

 
 	/*
 
 /*
-- 
2.11.0

From 3ce3e2e31511f7a0a2a93b0709b050794bfaf0b9 Mon Sep 17 00:00:00 2001
From: Euler Taveira <[email-masked]>
Date: Wed, 14 Mar 2018 00:53:17 +0000
Subject: [PATCH 8/8] Debug for row filtering

---

index 716ed2ec58..d0406d14d7 100644

 
index 83fad465f8..509124b9cf 100644

 
@@ -364,6 +380,42 @@ pgoutput_change(LogicalDecodingContext *ctx, ReorderBufferTXN *txn,
 
+#ifdef	_NOT_USED
+		if (old_tuple)
+		{
+			int i;
+
@@ -379,6 +431,7 @@ pgoutput_change(LogicalDecodingContext *ctx, ReorderBufferTXN *txn,
 			Oid			expr_type;
 			Datum		res;
 			bool		isnull;
+			char		*s = NULL;

@@ -389,12 +442,22 @@ pgoutput_change(LogicalDecodingContext *ctx, ReorderBufferTXN *txn,
 			expr_state = ExecInitExpr(expr, NULL);
 			res = ExecEvalExpr(expr_state, ecxt, &isnull);

@@ -668,9 +731,12 @@ get_rel_sync_entry(PGOutputData *data, Oid relid)

---

From: Erik Rijkers <[email-masked]>
Date: Tue, 03 Sep 2019 01:08:25 +0200

> seems it applies cleanly.

It did apply cleanly, thanks.

But I can't get it to correctly do the partial replication in the 
attached pgbench-script (similar versions of which script I also used 
for earlier versions of the patch, last year).

There are complaints in the log (both pub and sub) like:
ERROR:  trying to store a heap tuple into wrong type of slot

I have no idea what causes that.

I attach a zip:

$ unzip -l logrep_rowfilter.zip
Archive:  logrep_rowfilter.zip
   Length      Date    Time    Name
---------  ---------- -----   ----
     17942  2019-09-03 00:47   logfile.6525
     10412  2019-09-03 00:47   logfile.6526
      6913  2019-09-03 00:47   logrep_rowfilter_2_nodes.sh
      3371  2019-09-03 00:47   output.txt
---------                     -------
     38638                     4 files

That bash script runs 2 instances (as compiled on my local setup so it 
will not run as-is) and tries for one minute to get a slice of the 
pgbench_accounts table replicated.  One minute is short but I wanted 
short logfiles; I have tried the same up to 20 minutes without the 
replication completing.  I'll try even longer but in the meantime I hope 
you can figure out why these errors occur.

thanks,

Erik Rijkers

---

From: Alexey Zagarin <[email-masked]>
Date: Tue, 3 Sep 2019 10:15:44 +0700

> I have no idea what causes that.

Yeah, I've seen that too. It was fixed by Alexey Kondratov, in line 955 of 0005-Row-filtering-for-logical-replication.patch it should be &TTSOpsHeapTuple instead of &TTSOpsVirtual.

---

From: Euler Taveira <[email-masked]>
Date: Tue, 3 Sep 2019 00:32:12 -0300

> Yeah, I've seen that too. It was fixed by Alexey Kondratov, in line 955 of 0005-Row-filtering-for-logical-replication.patch it should be &TTSOpsHeapTuple instead of &TTSOpsVirtual.
Ops... exact. That was an oversight while poking with different types of slots.

---

From: Erik Rijkers <[email-masked]>
Date: Tue, 03 Sep 2019 07:50:29 +0200

On 2019-09-03 05:32, Euler Taveira wrote:
>> Yeah, I've seen that too. It was fixed by Alexey Kondratov, in line 
>> 955 of 0005-Row-filtering-for-logical-replication.patch it should be 
>> &TTSOpsHeapTuple instead of &TTSOpsVirtual.
>> 

OK, I'll consider Alexey Kondratov's set of patches as the current 
state-of-the-art then.  (They still apply.)

I found a problem where I'm not sure it's a bug:

The attached bash script does a test by setting up pgbench tables on 
both master and replica, and then sets up logical replication for a 
slice of pgbench_accounts. Then it does a short pgbench run, and loops 
until the results become identical(ok) (or breaks out after a certain 
time (NOK=not ok)).

It turns out this did not work until I added a wait state after the 
CREATE SUBSCRIPTION.  It always fails without the wait state, and always 
works with the wait state.

Do you agree this is a bug?

thanks (also to both Alexeys :))

Erik Rijkers

PS
by the way, this script won't run as-is on other machines; it has stuff 
particular to my local setup.

---

From: Alexey Zagarin <[email-masked]>
Date: Wed, 4 Sep 2019 08:21:27 +0700

> state-of-the-art then. (They still apply.)

Alexey's patch is the rebased version of previous Euler's patch set, with slot type mistake fixed, and adapted to current changes in the master branch. It also has testing improvements. On the other hand, the new patches from Euler include more fixes and the implementation of filtering in COPY (as far as I can tell from code) which addresses my particular pain point with BDR. Hope they'll be joined soon. :)

> Do you agree this is a bug?

I'm not sure this is a bug as after the subscription is added (or a new table added to the publication and then the subscription is refreshed), the whole table is synchronized using COPY statement. Depending on size of the table it can take some time. You may want to check srsubstate in pg_subscription_rel instead of just sleep for more reliable implementation.

Alexey

---

From: Euler Taveira <[email-masked]>
Date: Wed, 4 Sep 2019 10:11:14 -0300

> Ops... exact. That was an oversight while poking with different types of slots.
Here is a rebased version including this small fix.

From 92474dd8e15d58e253d5b5aa76348d8973bf6d04 Mon Sep 17 00:00:00 2001

 

 

 
 
 	natt = 0;

From b8a8d98368ba032670788ac4f4b4ef666a4dedd0 Mon Sep 17 00:00:00 2001
From: Euler Taveira <[email-masked]>
Date: Tue, 27 Feb 2018 02:21:03 +0000
Subject: [PATCH 3/8] Refactor function create_estate_for_relation

 

 

From 2b398d46e12b0a9c5cf134585597991e4a2d43dc Mon Sep 17 00:00:00 2001
From: Euler Taveira <[email-masked]>
Date: Fri, 9 Mar 2018 18:39:22 +0000
Subject: [PATCH 1/8] Remove unused atttypmod column from initial table
 synchronization

 

From 32ba5ccea3e329044c14f2b3a82de463a573cb63 Mon Sep 17 00:00:00 2001
From: Euler Taveira <[email-masked]>
Date: Wed, 24 Jan 2018 17:01:31 -0200
Subject: [PATCH 4/8] Rename a WHERE node

 
-ExclusionWhereClause:
+OptWhereClause:

From 7ccf083418bb07b130963b711d4b48e138924731 Mon Sep 17 00:00:00 2001
From: Euler Taveira <[email-masked]>
Date: Tue, 27 Feb 2018 04:03:13 +0000
Subject: [PATCH 5/8] Row filtering for logical replication

 
  <refsynopsisdiv>
 <synopsis>

      <para>
 
   <para>
 
  <refsect1>
 
   <para>

 

 	/* Form a tuple. */

 

 	return myself;
 }
 
 /*

+	/*

 
 			foreach(newlc, rels)
 			{

-			if (!found)
+			/*

 		/* And drop them. */
 
 		/*
 
 	CloseTableList(rels);
 
 	/*

 
 
 	foreach(lc, rels)
 	{

 /*
 
 	foreach(lc, rels)
 	{

 
 	foreach(lc, rels)
 	{

 

 publication_for_tables:
 
+publication_table_list:
 
 			break;
 
 			/*
 
 			/*

 
 			/*
 
 			/*
 
 			/*
 
 /*
 
 	lrel->nspname = nspname;
 	lrel->relname = relname;
 
-	natt = 0;
+	n = 0;

 
-	lrel->natts = natt;
+	lrel->natts = n;
+

 

 
 	/* Do the copy */
index 9c08757fca..89a80a8abc 100644

 #include "fmgr.h"

 

 

 
 		list_free(pubids);

 
 
 /* ----------------
 
 	/*

 

 

From c34e95f676a47219b12e06868b1304fbd91f4b3d Mon Sep 17 00:00:00 2001
From: Euler Taveira <[email-masked]>
Date: Thu, 17 May 2018 20:52:28 +0000
Subject: [PATCH 6/8] Print publication WHERE condition in psql

---

From 537e5b46211fd6b9987b47752a3b674e887e3157 Mon Sep 17 00:00:00 2001
From: Euler Taveira <[email-masked]>
Date: Sat, 15 Sep 2018 02:52:00 +0000
Subject: [PATCH 7/8] Publication where condition support for pg_dump

---

index 7ec0c84540..69153fa28d 100644

 

 
 	/*
 
 /*
-- 
2.11.0

From eac82a07d5a0b7d0c71490469f6ca473950e0333 Mon Sep 17 00:00:00 2001
From: Euler Taveira <[email-masked]>
Date: Wed, 14 Mar 2018 00:53:17 +0000
Subject: [PATCH 8/8] Debug for row filtering

---

 
 
index 89a80a8abc..f94dab88b6 100644

 
 
+#ifdef	_NOT_USED
+		if (old_tuple)
+		{
+			int i;
+

---

From: movead li <[email-masked]>
Date: Mon, 23 Sep 2019 04:59:02 +0000

Hello

I find several problems as below when I test the patches:

1. There be some regression problem after apply 0001.patch~0005.patch
   The regression problem is solved in 0006.patch
2. There be a data wrong after create subscription if the relation contains
     inherits table, for example:
   ##########################
   The Tables:
   CREATE TABLE cities (
       name            text,
       population      float,
       altitude        int
   );
   CREATE TABLE capitals (
       state           char(2)
   ) INHERITS (cities);
   
   Do on publication:
   insert into cities values('aaa',123, 134);
   insert into capitals values('bbb',123, 134);
   create publication pub_tc for table cities where (altitude > 100 and altitude < 200);
   postgres=# select * from cities ;
    name | population | altitude 
   ------+------------+----------
    aaa  |        123 |      134
    bbb  |        123 |      134
   (2 rows)
   
   Do on subscription:
   create subscription sub_tc connection 'host=localhost port=5432 dbname=postgres' publication pub_tc;
   postgres=# select * from cities ;
    name | population | altitude 
   ------+------------+----------
    aaa  |        123 |      134
    bbb  |        123 |      134
    bbb  |        123 |      134
   (3 rows)
   ##########################
   An unexcept row appears.
   
3. I am puzzled when I test the update.
      Use the tables in problem 2 and test as below:
      #########################
      On publication:
      postgres=# insert into cities values('t1',123, 34);
      INSERT 0 1
      postgres=# update cities SET altitude = 134 where altitude = 34;
      UPDATE 1
      postgres=# select * from cities ;
       name | population | altitude 
      ------+------------+----------
       t1   |        123 |      134
      (1 row)
      On subscription:
      postgres=# select * from cities ;
       name | population | altitude 
      ------+------------+----------
      (0 rows)
      
      On publication:
      insert into cities values('t1',1,'135');
      update cities set altitude=300 where altitude=135;
      postgres=# table cities ;
       name | population | altitude 
      ------+------------+----------
       t1   |        123 |      134
       t1   |          1 |      300
      (2 rows)
      
      On subscription:
      ostgres=# table cities ;
       name | population | altitude 
      ------+------------+----------
       t1   |          1 |      135
      (1 row)
      #########################
      Result1:Update a row that is not suitable the publication condition to
      suitable, the subscription change nothing.
      Result2: Update a row that is suitable for the publication condition to
      not suitable, the subscription change nothing.
      If it is a bug? Or there should be an explanation about it?

4. SQL splicing code in fetch_remote_table_info() function is too long

---
Highgo Software (Canada/China/Pakistan) 
URL : www.highgo.ca 
EMAIL: mailto:[email-masked]

The new status of this patch is: Waiting on Author

---

From: Euler Taveira <[email-masked]>
Date: Wed, 25 Sep 2019 08:08:40 -0300

> I find several problems as below when I test the patches:
First of all, thanks for your review.

>    The regression problem is solved in 0006.patch
Which regression?

>      inherits table, for example:
Ouch. Good catch! Forgot about the ONLY in COPY with query. I will add
a test for it.

>       Use the tables in problem 2 and test as below:
INSERT isn't replicated.

>       postgres=# update cities SET altitude = 134 where altitude = 34;
There should be an error because you don't have a PK or REPLICA IDENTITY.

postgres=# update cities SET altitude = 134 where altitude = 34;
ERROR:  cannot update table "cities" because it does not have a
replica identity and publishes updates
HINT:  To enable updating the table, set REPLICA IDENTITY using ALTER TABLE.

Even if you create a PK or REPLICA IDENTITY, it won't turn this UPDATE
into a INSERT and send it to the other node (indeed UPDATE will be
sent however there isn't a tuple to update). Also, filter columns must
be in PK or REPLICA IDENTITY. I explain this in documentation.

> 4. SQL splicing code in fetch_remote_table_info() function is too long
I split it into small pieces. I also run pgindent to improve code style.

I'll send a patchset later today.

Regards,

---

From: Euler Taveira <[email-masked]>
Date: Wed, 25 Sep 2019 20:20:32 -0300

> I'll send a patchset later today.
... and it is attached.

From b5d4d1369dbb4e7ec20182507dc5ae920dd8d2e9 Mon Sep 17 00:00:00 2001

index 7881079..0a565dd 100644

 

From 406b2dbe4df63a94364e548a67d085e255ea2644 Mon Sep 17 00:00:00 2001
From: Euler Taveira <[email-masked]>
Date: Fri, 9 Mar 2018 17:37:36 +0000
Subject: [PATCH 2/8] Store number of tuples in WalRcvExecResult

index 6eba08a..343550a 100644

 

 
index 0a565dd..42db4ad 100644

 	natt = 0;
index e12a934..0d32d59 100644

From 428c6f3959b67627e9f1c92fdf38d71bb66163ef Mon Sep 17 00:00:00 2001
From: Euler Taveira <[email-masked]>
Date: Wed, 24 Jan 2018 17:01:31 -0200
Subject: [PATCH 4/8] Rename a WHERE node

index 3f67aaf..21bef5c 100644
@@ -3711,7 +3711,7 @@ ConstraintElem:
@@ -3813,7 +3813,7 @@ ExclusionConstraintElem: index_elem WITH any_operator
 			}
 		;
 
-ExclusionWhereClause:
+OptWhereClause:

From 45231f2c46b61aabb0fcb4f938589a6c21aad2c5 Mon Sep 17 00:00:00 2001
From: Euler Taveira <[email-masked]>
Date: Tue, 27 Feb 2018 02:21:03 +0000
Subject: [PATCH 3/8] Refactor function create_estate_for_relation

index 11e6331..d9952c8 100644

 

 

From e14f1892427778b728c0a684e68d89ee172a4679 Mon Sep 17 00:00:00 2001
From: Euler Taveira <[email-masked]>
Date: Tue, 27 Feb 2018 04:03:13 +0000
Subject: [PATCH 5/8] Row filtering for logical replication

When you define or modify a publication you optionally can filter rows
to be published using a WHERE condition. This condition is any
expression that evaluates to boolean. Only those rows that
satisfy the WHERE condition will be sent to subscribers.
---
 doc/src/sgml/catalogs.sgml                  |   9 ++
 doc/src/sgml/ref/alter_publication.sgml     |  11 ++-
 doc/src/sgml/ref/create_publication.sgml    |  26 +++++-
 src/backend/catalog/pg_publication.c        | 102 ++++++++++++++++++++--
 src/backend/commands/publicationcmds.c      |  89 +++++++++++++------
 src/backend/parser/gram.y                   |  26 ++++--
 src/backend/parser/parse_agg.c              |  10 +++
 src/backend/parser/parse_expr.c             |  14 ++-
 src/backend/parser/parse_func.c             |   3 +
 src/backend/replication/logical/tablesync.c | 127 +++++++++++++++++++++++++---
 src/backend/replication/logical/worker.c    |   2 +-
 src/backend/replication/pgoutput/pgoutput.c | 101 +++++++++++++++++++++-
 src/include/catalog/pg_publication.h        |   9 +-
 src/include/catalog/pg_publication_rel.h    |   4 +
 src/include/catalog/toasting.h              |   1 +
 src/include/nodes/nodes.h                   |   1 +
 src/include/nodes/parsenodes.h              |  11 ++-
 src/include/parser/parse_node.h             |   1 +
 src/include/replication/logicalrelation.h   |   2 +
 src/test/regress/expected/publication.out   |  29 +++++++
 src/test/regress/sql/publication.sql        |  21 +++++
 src/test/subscription/t/013_row_filter.pl   |  96 +++++++++++++++++++++
 22 files changed, 634 insertions(+), 61 deletions(-)
 create mode 100644 src/test/subscription/t/013_row_filter.pl

index 5e71a2e..7f11225 100644
index 534e598..9608448 100644
 
  <refsynopsisdiv>
 <synopsis>
index 99f87ca..6e99943 100644

      <para>
 
   <para>
 
  <refsect1>
 
   <para>
index fd5da7d..f5462dc 100644

 

 	/* Form a tuple. */

 

 	return myself;
 }
 
 /*
index f115d4b..2606377 100644
@@ -352,6 +352,28 @@ AlterPublicationTables(AlterPublicationStmt *stmt, Relation rel,

+	/*
+	 * ALTER PUBLICATION ... DROP TABLE cannot contain a WHERE clause.  Use
+	 * publication_table_list node (that accepts a WHERE clause) but forbid
+	 * the WHERE clause in it.  The use of relation_expr_list node just for
+	 * the DROP TABLE part does not worth the trouble.
+	 */
+	if (stmt->tableAction == DEFELEM_DROP)
+	{
+		ListCell	*lc;
+
+		foreach(lc, stmt->tables)
+		{
+			PublicationTable *t = lfirst(lc);
+
+			if (t->whereClause)
+				ereport(ERROR,
+						(errcode(ERRCODE_OBJECT_NOT_IN_PREREQUISITE_STATE),
+						 errmsg("cannot use a WHERE clause for removing table from publication \"%s\"",
+								NameStr(pubform->pubname))));
+		}
+	}
+
 	rels = OpenTableList(stmt->tables);
 
@@ -360,47 +382,56 @@ AlterPublicationTables(AlterPublicationStmt *stmt, Relation rel,

 			foreach(newlc, rels)
 			{

-			if (!found)
+			/*
+			 * Remove publication / relation mapping iif (i) table is not
+			 * found in the new list or (ii) table is found in the new list,
+			 * however, its qual does not match the old one (in this case, a
+			 * simple tuple update is not enough because of the dependencies).
+			 */
+			if (!found || (found && !equal(oldrel->whereClause, newrel->whereClause)))
 			{
-				Relation	oldrel = table_open(oldrelid,
-												ShareUpdateExclusiveLock);
+				PublicationRelationQual *oldrelqual = palloc(sizeof(PublicationRelationQual));

 		/* And drop them. */
 
 		/*
 
 	CloseTableList(rels);
@@ -510,13 +541,15 @@ OpenTableList(List *tables)
 	List	   *relids = NIL;
 	List	   *rels = NIL;
 	ListCell   *lc;
+	PublicationRelationQual *relqual;
 
 	/*
 		bool		recurse = rv->inh;
 		Oid			myrelid;
@@ -539,8 +572,10 @@ OpenTableList(List *tables)
 
 		/* Add children of this rel, if requested */
@@ -568,7 +603,11 @@ OpenTableList(List *tables)
 
@@ -589,10 +628,12 @@ CloseTableList(List *rels)
 
 	foreach(lc, rels)
 	{

 /*
@@ -608,13 +649,13 @@ PublicationAddTables(Oid pubid, List *rels, bool if_not_exists,
 
 	foreach(lc, rels)
 	{

 		obj = publication_add_relation(pubid, rel, if_not_exists);
 		if (stmt)
@@ -640,8 +681,8 @@ PublicationDropTables(Oid pubid, List *rels, bool missing_ok)
 
 	foreach(lc, rels)
 	{
 
@@ -654,7 +695,7 @@ PublicationDropTables(Oid pubid, List *rels, bool missing_ok)
 
index 21bef5c..8cad2bc 100644

 %type <list>	opt_fdw_options fdw_options
@@ -9547,7 +9547,7 @@ opt_publication_for_tables:
 		;
 
 publication_for_tables:
-			FOR TABLE relation_expr_list
+			FOR TABLE publication_table_list
 				{
 					$$ = (Node *) $3;
 				}
@@ -9578,7 +9578,7 @@ AlterPublicationStmt:
 					n->options = $5;
 					$$ = (Node *)n;
 				}
-			| ALTER PUBLICATION name ADD_P TABLE relation_expr_list
+			| ALTER PUBLICATION name ADD_P TABLE publication_table_list
 				{
 					AlterPublicationStmt *n = makeNode(AlterPublicationStmt);
 					n->pubname = $3;
@@ -9586,7 +9586,7 @@ AlterPublicationStmt:
 					n->tableAction = DEFELEM_ADD;
 					$$ = (Node *)n;
 				}
-			| ALTER PUBLICATION name SET TABLE relation_expr_list
+			| ALTER PUBLICATION name SET TABLE publication_table_list
 				{
 					AlterPublicationStmt *n = makeNode(AlterPublicationStmt);
 					n->pubname = $3;
@@ -9594,7 +9594,7 @@ AlterPublicationStmt:
 					n->tableAction = DEFELEM_SET;
 					$$ = (Node *)n;
 				}
-			| ALTER PUBLICATION name DROP TABLE relation_expr_list
+			| ALTER PUBLICATION name DROP TABLE publication_table_list
 				{
 					AlterPublicationStmt *n = makeNode(AlterPublicationStmt);
 					n->pubname = $3;
@@ -9604,6 +9604,20 @@ AlterPublicationStmt:
 				}
 		;
 
+publication_table_list:
index f418c61..dea5aad 100644
 
 			break;
 
 			/*
 
 			/*
index 76f3dd7..6d2c6a2 100644

 
 			/*
 
 			/*
index 8e92653..66458d8 100644
 
 			/*
index 42db4ad..d3999b1 100644
 
 /*
 
 	lrel->nspname = nspname;
 	lrel->relname = relname;
 
-	natt = 0;
+	n = 0;

@@ -734,7 +741,52 @@ fetch_remote_table_info(char *nspname, char *relname,
 	}
 	ExecDropSingleTupleTableSlot(slot);
 
-	lrel->natts = natt;
+	lrel->natts = n;
+
+	walrcv_clear_result(res);
+
+	/* Get relation qual */
+	resetStringInfo(&cmd);
+	appendStringInfo(&cmd,
+						"SELECT pg_get_expr(prqual, prrelid) "
+						"  FROM pg_publication p "
+						"  INNER JOIN pg_publication_rel pr "
+						"       ON (p.oid = pr.prpubid) "
+						" WHERE pr.prrelid = %u "
+						"   AND p.pubname IN (", lrel->remoteid);
+
+	first = true;
+	foreach(lc, MySubscription->publications)
+	{
+		char	   *pubname = strVal(lfirst(lc));
+
+		if (first)
+			first = false;
+		else
+			appendStringInfoString(&cmd, ", ");
+
+		appendStringInfoString(&cmd, quote_literal_cstr(pubname));
+	}
+	appendStringInfoChar(&cmd, ')');
+
+	res = walrcv_exec(wrconn, cmd.data, 1, qualRow);
+
+	if (res->status != WALRCV_OK_TUPLES)
+		ereport(ERROR,
+				(errmsg("could not fetch relation qualifications for table \"%s.%s\" from publisher: %s",
+						nspname, relname, res->err)));
+
+	slot = MakeSingleTupleTableSlot(res->tupledesc, &TTSOpsMinimalTuple);
+	while (tuplestore_gettupleslot(res->tuplestore, true, false, slot))
+	{
+		Datum		rf = slot_getattr(slot, 1, &isnull);
+
+		if (!isnull)
+			*qual = lappend(*qual, makeString(TextDatumGetCString(rf)));
+
+		ExecClearTuple(slot);
+	}
+	ExecDropSingleTupleTableSlot(slot);
 
 	walrcv_clear_result(res);
 	pfree(cmd.data);
@@ -750,6 +802,7 @@ copy_table(Relation rel)
 {
 	LogicalRepRelMapEntry *relmapentry;
 	LogicalRepRelation lrel;
+	List	   *qual = NIL;
 	WalRcvExecResult *res;
 	StringInfoData cmd;
 	CopyState	cstate;
@@ -758,7 +811,7 @@ copy_table(Relation rel)

@@ -767,10 +820,59 @@ copy_table(Relation rel)
 
+		appendStringInfo(&cmd, " FROM ONLY %s",
@@ -785,7 +887,6 @@ copy_table(Relation rel)

 	/* Do the copy */
index d9952c8..cef0c52 100644
index 9c08757..63596e2 100644

 #include "fmgr.h"

 
 
+		foreach(lc, relentry->qual)

 
 			if (pub->alltables || list_member_oid(pubids, pub->oid))
 			{
@@ -583,9 +658,24 @@ get_rel_sync_entry(PGOutputData *data, Oid relid)
 				entry->pubactions.pubtruncate |= pub->pubactions.pubtruncate;
 			}

 		list_free(pubids);
@@ -660,5 +750,10 @@ rel_sync_cache_publication_cb(Datum arg, int cacheid, uint32 hashvalue)
 	 */
index 20a2f0a..5261666 100644

 
 extern Oid	get_publication_oid(const char *pubname, bool missing_ok);
diff --git a/src/include/catalog/pg_publication_rel.h b/src/include/catalog/pg_publication_rel.h
index 5f5bc92..7fd5915 100644
--- a/src/include/catalog/pg_publication_rel.h
+++ b/src/include/catalog/pg_publication_rel.h
@@ -31,6 +31,10 @@ CATALOG(pg_publication_rel,6106,PublicationRelRelationId)
 	Oid			oid;			/* oid */
 	Oid			prpubid;		/* Oid of the publication */
 	Oid			prrelid;		/* Oid of the relation */
+
+#ifdef	CATALOG_VARLEN			/* variable-length fields start here */
+	pg_node_tree prqual;		/* qualifications */
+#endif
 } FormData_pg_publication_rel;
 
 /* ----------------
index cc5dfed..d57ca82 100644
index bce2d59..52522d0 100644
@@ -477,6 +477,7 @@ typedef enum NodeTag
 	T_PartitionRangeDatum,
 	T_PartitionCmd,
 	T_VacuumRelation,
+	T_PublicationTable,
 
 	/*
 	 * TAGS FOR REPLICATION GRAMMAR PARSE NODES (replnodes.h)
diff --git a/src/include/nodes/parsenodes.h b/src/include/nodes/parsenodes.h
index d93a79a..ca9920c 100644
--- a/src/include/nodes/parsenodes.h
+++ b/src/include/nodes/parsenodes.h
@@ -3474,12 +3474,19 @@ typedef struct AlterTSConfigurationStmt
 } AlterTSConfigurationStmt;

@@ -3492,7 +3499,7 @@ typedef struct AlterPublicationStmt
 	List	   *options;		/* List of DefElem nodes */
 
index 7c099e7..a5c9109 100644

index 2642a3f..5cc307e 100644
 
index feb51e4..202173c 100644
 
index 5773a75..6f0d088 100644
 
index 0000000..99e6db9

From a5ca3aa97cd50b796f58fe60e8dbc9ed196aac9b Mon Sep 17 00:00:00 2001
From: Euler Taveira <[email-masked]>
Date: Thu, 17 May 2018 20:52:28 +0000
Subject: [PATCH 6/8] Print publication WHERE condition in psql

---

diff --git a/src/bin/psql/describe.c b/src/bin/psql/describe.c
index d7c0fc0..76404e0 100644
--- a/src/bin/psql/describe.c
+++ b/src/bin/psql/describe.c
@@ -5876,7 +5876,8 @@ describePublications(const char *pattern)
 		if (!puballtables)
 		{
 			printfPQExpBuffer(&buf,
-							  "SELECT n.nspname, c.relname\n"
+							  "SELECT n.nspname, c.relname,\n"
+							  "  pg_get_expr(pr.prqual, c.oid)\n"
 							  "FROM pg_catalog.pg_class c,\n"
 							  "     pg_catalog.pg_namespace n,\n"
 							  "     pg_catalog.pg_publication_rel pr\n"
@@ -5906,6 +5907,10 @@ describePublications(const char *pattern)
 								  PQgetvalue(tabres, j, 0),
 								  PQgetvalue(tabres, j, 1));

From 90f046605051e79739372339ffeae982fd45e328 Mon Sep 17 00:00:00 2001
From: Euler Taveira <[email-masked]>
Date: Wed, 14 Mar 2018 00:53:17 +0000
Subject: [PATCH 8/8] Debug for row filtering

---

index 2606377..b2378c6 100644

 
 	 * publication_table_list node (that accepts a WHERE clause) but forbid
index d3999b1..2f586c0 100644
@@ -873,6 +873,7 @@ copy_table(Relation rel)
index 63596e2..6306bc3 100644

 
 
+#ifdef	_NOT_USED
+		if (old_tuple)
+		{
+			int i;
+

 
 
@@ -668,10 +731,13 @@ get_rel_sync_entry(PGOutputData *data, Oid relid)

From 3897f998a328fbd42824fe265a15e76ec1247703 Mon Sep 17 00:00:00 2001
From: Euler Taveira <[email-masked]>
Date: Sat, 15 Sep 2018 02:52:00 +0000
Subject: [PATCH 7/8] Publication where condition support for pg_dump

---

index f01fea5..3c37134 100644

 

 
 	/*
index ec5a924..8d61faa 100644
 
 /*
-- 
2.7.4

---

From: "[email-masked]" <[email-masked]>
Date: Thu, 26 Sep 2019 09:38:39 +0800

>Which regression?
Apply the 0001.patch~0005.patch and then do a 'make check', then there be a
failed item. And when you apply the 0006.patch, the failed item disappeared.

>There should be an error because you don't have a PK or REPLICA IDENTITY.
No. I have done the 'ALTER TABLE cities  REPLICA IDENTITY FULL'.

>be in PK or REPLICA IDENTITY. I explain this in documentation.
You should considered the Result2:
     On publication:
      insert into cities values('t1',1,135);
      update cities set altitude=300 where altitude=135;
      postgres=# table cities ;
       name | population | altitude 
      ------+------------+----------
       t1   |        123 |      134
       t1   |          1 |      300
      (2 rows)
      
      On subscription:
      ostgres=# table cities ;

The tuple ('t1',1,135) appeared in both publication and subscription,
but after an update on publication, the tuple is disappeared on 
publication and change nothing on subscription.

The same with Result1, they puzzled me today and I think they will
puzzle the users in the future. It should have a more wonderful design,
for example, a log to notify users that there be a problem during replication
at least.

---
Highgo Software (Canada/China/Pakistan) 
URL : www.highgo.ca 
EMAIL: mailto:movead(dot)li(at)highgo(dot)ca

---

From: Amit Langote <[email-masked]>
Date: Mon, 25 Nov 2019 11:38:51 +0900

Hi Euler,

Thanks for working on this.  I have reviewed the patches, as I too am
working on a patch related to logical replication [1].

> > I'll send a patchset later today.

Needed to be rebased, which I did, to be able to test them; patches attached.

Some comments:

* 0001: seems a no-brainer

* 0002: seems, um, unnecessary?  The only place ntuples will be used is here:

@@ -702,9 +702,8 @@ fetch_remote_table_info(char *nspname, char *relname,
                 (errmsg("could not fetch table info for table \"%s.%s\": %s",
                         nspname, relname, res->err)));

but you might as well use tuplestore_tuple_count(res->tuplestore).  My
point is that if ntuples that this patch is adding was widely useful
(as would be shown by the number of places that could be refactored to
use it), it would have been worthwhile to add it.

* 0003: seems fine to me.

* 0004: seems fine too, although maybe preproc.y should be updated too?

* 0005: naturally many comments here :)

Minor nitpicking: "in the form of a" seems unnecessary.  Other places
that mention nodeToString() just say "in
<function>nodeToString()</function> representation"

Can you please explain the reasoning behind this restriction.  Sorry
if this is already covered in the up-thread discussion.

 /*
+ * Gets list of PublicationRelationQuals for a publication.
+ */
+List *
+GetPublicationRelationQuals(Oid pubid)
+{
...
+        relqual->relation = table_open(pubrel->prrelid,
ShareUpdateExclusiveLock);

I think it's a bad idea to open the table in one file and rely on
something else in the other file closing it.  I know you're having it
to do it because you're using PublicationRelationQual to return
individual tables, but why not just store the table's OID in it and
only open and close the relation where it's needed.  Keeping the
opening and closing of relation close to each other is better as long
as it doesn't need to be done many times over in many different
functions.  In this case, pg_publication.c: publication_add_relation()
is the only place that needs to look at the open relation, so opening
and closing should both be done there.  Nothing else needs to look at
the open relation.

Actually, OpenTableList() should also not open the relation.  Then we
don't need CloseTableList().  I think it would be better to refactor
things around this and include the patch in this series.

+    /* Find all publications associated with the relation. */
+    pubrelsrel = table_open(PublicationRelRelationId, AccessShareLock);

I guess you meant:

/* Get all relations associated with this publication. */

Is copying really necessary?

+    /*

This comment is not very helpful, as it's not clear what the various
names are referring to.  I'd just just write:

    /*
     * Although ALTER PUBLICATION's grammar allows WHERE clause to be
     * specified for DROP TABLE action, it doesn't makes sense to allow it.
     * We implement that rule here, instead of complicating grammar to enforce
     * it.
     */

+                         errmsg("cannot use a WHERE clause for
removing table from publication \"%s\"",

I think: s/for/when/g

+            /*

Aside from the typo on the 1st line (iif), I suggest writing this as:

            /*-----------
             * Remove the publication-table mapping if:
             *
             * 1) Table is not found the new list of tables
             *
             * 2) Table is being re-added with a different qual expression
             *
             * For (2), simply updating the existing tuple is not enough,
             * because of the qual expression's dependencies.
             */

Maybe:

functions are now allowed in publication WHERE expressions

s/expression/expressions/g

Maybe:

publication WHERE expression
or
publication qual

-    int         natt;
+    int         n;

Are this and other related changes really needed?

Hmm, why wouldn't SELECT * suffice?

+        estate = create_estate_for_relation(relation);
+
+        /* prepare context per tuple */
+        ecxt = GetPerTupleExprContext(estate);
+        oldcxt = MemoryContextSwitchTo(estate->es_query_cxt);
+        ecxt->ecxt_scantuple = ExecInitExtraTupleSlot(estate,
tupdesc, &TTSOpsHeapTuple);
...
+        ExecDropSingleTupleTableSlot(ecxt->ecxt_scantuple);
+        FreeExecutorState(estate);

Creating and destroying the EState (that too with the ResultRelInfo
that is never used) for every tuple seems wasteful.  You could store
the standalone ExprContext in RelationSyncEntry and use it for every
tuple.

+            /* evaluates row filter */
+            expr_type = exprType(qual);
+            expr = (Expr *) coerce_to_target_type(NULL, qual,
expr_type, BOOLOID, -1, COERCION_ASSIGNMENT, COERCE_IMPLICIT_CAST,
-1);
+            expr = expression_planner(expr);
+            expr_state = ExecInitExpr(expr, NULL);

Also, there appears to be no need to repeat this for every tuple?  I
think this should be done only once, that is, RelationSyncEntry.qual
should cache ExprState nodes, not bare Expr nodes.

Given the above comments, the following seems unnecessary:

By the way, make check doesn't pass.  I see the following failure:

-    "public.testpub_rf_tbl3"  WHERE ((e > 300) AND (e < 500))
+    "public.testpub_rf_tbl3"

but I guess applying subsequent patches takes care of that.

* 0006 and 0007: small enough that I think it might be better to merge
them into 0005.

* 0008: no comments as it's not intended to be committed. :)

Thanks,
Amit

[1] https://commitfest.postgresql.org/25/2301/

---

From: Amit Langote <[email-masked]>
Date: Mon, 25 Nov 2019 11:48:29 +0900

> Needed to be rebased, which I did, to be able to test them; patches attached.

Oops, really attached this time.

Thanks,
Amit

---

From: Michael Paquier <[email-masked]>
Date: Thu, 28 Nov 2019 11:32:01 +0900

> Oops, really attached this time.

Euler, this thread is waiting for input from you regarding the latest
comments from Amit.
--
Michael

---

From: Tomas Vondra <[email-masked]>
Date: Thu, 16 Jan 2020 22:57:30 +0100

>Euler, this thread is waiting for input from you regarding the latest

Euler, this patch is still in "waiting on author" since 11/25. Do you
plan to review changes made by Amit in the patches he submitted, or what
are your plans with this patch?

regards

---

From: Euler Taveira <[email-masked]>
Date: Thu, 16 Jan 2020 20:58:24 -0300

> are your plans with this patch?
Yes, I'm working on Amit suggestions. I'll post a new patch as soon as possible.

---

From: Craig Ringer <[email-masked]>
Date: Tue, 21 Jan 2020 15:32:45 +0800

> Yes, I'm working on Amit suggestions. I'll post a new patch as soon as possible.

Great. I think this'd be nice to see.

Were you able to fully address the following points that came up in
the discussion?

* Make sure row filters cannot access non-catalog, non-user-catalog
relations i.e. can only use RelationIsAccessibleInLogicalDecoding rels

* Prevent filters from attempting to access attributes that may not be
WAL-logged in a given change record, or give them a way to test for
this. Unchanged TOASTed atts are not logged. There's also REPLICA
IDENTITY FULL to consider if exposing access to the old tuple in the
filter.

Also, while I'm not sure if it was raised earlier, experience with row
filtering in pglogical has shown that error handling is challenging.
Because row filters are read from a historic snapshot of the catalogs
you cannot change them or any SQL or plpgsql functions they use if a
problem causes an ERROR when executing the filter expression. You can
fix the current snapshot's definition but the decoding session won't
see it and will continue to ERROR. We don't really have a good answer
for that yet in pglogical; right now you have to either intervene with
low level tools or drop the subscription and re-create it. Neither of
which is ideal.

You can't just read the row filter from the current snapshot as the
relation definition (atts etc) may not match. Plus that creates a
variety of issues with which txns get which version of a row filter
applied during decoding, consistency between multiple subscribers,
etc.

One option I've thought about was a GUC that allows users to specify
what should be done for errors in row filter expressions: drop the row
as if the filter rejected it; pass the row as if the filter matched;
propagate the ERROR and end the decoding session (default).

I'd welcome ideas about this one. I don't think it's a showstopper for
accepting the feature either, we just have to document that great care
is required with any operator or function that could raise an error in
a row filter. But there are just so many often non-obvious ways you
can land up with an ERROR being thrown that I think it's a bit of a
user foot-gun.

-- 
 Craig Ringer                   http://www.2ndQuadrant.com/
 2ndQuadrant - PostgreSQL Solutions for the Enterprise

---

From: David Steele <[email-masked]>
Date: Tue, 3 Mar 2020 12:39:03 -0500

Hi Euler,

> Great. I think this'd be nice to see.

The last CF for PG13 has started. Do you have a new patch ready?

Regards,
-- 
-David
[email-masked]

---

From: David Steele <[email-masked]>
Date: Mon, 16 Mar 2020 09:58:27 -0400

> The last CF for PG13 has started. Do you have a new patch ready?

I have marked this patch Returned with Feedback since no new patch has 
been posted.

Please submit to a future CF when a new patch is available.

Regards,
-- 
-David
[email-masked]

---

From: Önder Kalacı <[email-masked]>
Date: Thu, 17 Dec 2020 09:43:30 +0300

Hi all,

I'm also interested in this patch. I rebased the changes to the current
master branch and attached. The rebase had two issues. First, patch-8 was
conflicting, and that seems only helpful for debugging purposes during
development. So, I dropped it for simplicity. Second, the changes have a
conflict with `publish_via_partition_root` changes. I tried to fix the
issues, but ended-up having a limitation for now. The limitation is that
"cannot create publication with WHERE clause on the partitioned table
without publish_via_partition_root is set to true". This restriction can be
lifted, though I left out for the sake of focusing on the some issues that
I observed on this patch.

Please see my review:

+       if (list_length(relentry->qual) > 0)
+       {
+               HeapTuple       old_tuple;
+               HeapTuple       new_tuple;
+               TupleDesc       tupdesc;
+               EState     *estate;
+               ExprContext *ecxt;
+               MemoryContext oldcxt;
+               ListCell   *lc;
+               bool            matched = true;
+
+               old_tuple = change->data.tp.oldtuple ?
&change->data.tp.oldtuple->tuple : NULL;
+               new_tuple = change->data.tp.newtuple ?
&change->data.tp.newtuple->tuple : NULL;
+               tupdesc = RelationGetDescr(relation);
+               estate = create_estate_for_relation(relation);
+
+               /* prepare context per tuple */
+               ecxt = GetPerTupleExprContext(estate);
+               oldcxt = MemoryContextSwitchTo(estate->es_query_cxt);
+               ecxt->ecxt_scantuple = ExecInitExtraTupleSlot(estate,
tupdesc, &TTSOpsHeapTuple);
+
+               ExecStoreHeapTuple(new_tuple ? new_tuple : old_tuple,
ecxt->ecxt_scantuple, false);
+
+               foreach(lc, relentry->qual)
+               {
+                       Node       *qual;
+                       ExprState  *expr_state;
+                       Expr       *expr;
+                       Oid                     expr_type;
+                       Datum           res;
+                       bool            isnull;
+
+                       qual = (Node *) lfirst(lc);
+
+                       /* evaluates row filter */
+                       expr_type = exprType(qual);
+                       expr = (Expr *) coerce_to_target_type(NULL, qual,
expr_type, BOOLOID, -1, COERCION_ASSIGNMENT, COERCE_IMPLICIT_CAST, -1);
+                       expr = expression_planner(expr);
+                       expr_state = ExecInitExpr(expr, NULL);
+                       res = ExecEvalExpr(expr_state, ecxt, &isnull);
+
+                       /* if tuple does not match row filter, bail out */
+                       if (!DatumGetBool(res) || isnull)
+                       {
+                               matched = false;
+                               break;
+                       }
+               }
+
+               MemoryContextSwitchTo(oldcxt);
+

The above part can be considered the core of the logic, executed per tuple.
As far as I can see, it has two downsides.

First, calling `expression_planner()` for every tuple can be quite
expensive. I created a sample table, loaded data and ran a quick benchmark
to see its effect. I attached the very simple script that I used to
reproduce the issue on my laptop. I'm pretty sure you can find nicer ways
of doing similar perf tests, just sharing as a reference.

The idea of the test is to add a WHERE clause to a table, but none of the
tuples are filtered out. They just go through this code-path and send it to
the remote node.

#rows       Patched    | Master
1M  00:00:25.067536    | 00:00:16.633988
10M  00:04:50.770791    | 00:02:40.945358

So, it seems a significant overhead to me. What do you think?

Secondly, probably more importantly, allowing any operator is as dangerous
as allowing any function as users can create/overload operator(s). For
example, assume that users create an operator which modifies the table that
is being filtered out:

```
CREATE OR REPLACE FUNCTION function_that_modifies_table(left_art INTEGER,
right_arg INTEGER)
RETURNS BOOL AS
$$
BEGIN

  INSERT INTO test SELECT * FROM test;

  return left_art > right_arg;
 END;
$$ LANGUAGE PLPGSQL VOLATILE;

CREATE OPERATOR >>= (
  PROCEDURE = function_that_modifies_table,
  LEFTARG = INTEGER,
  RIGHTARG = INTEGER
);

CREATE PUBLICATION pub FOR TABLE test WHERE (key >>= 0);
``

With the above, we seem to be in trouble. Although the above is an extreme
example, it felt useful to share to the extent of the problem. We probably
cannot allow any free-form SQL to be on the filters.

To overcome these issues, one approach could be to rely on known safe
operators and functions. I believe the btree and hash operators should
provide a pretty strong coverage across many use cases. As far as I can
see, the procs that the following query returns can be our baseline:

```
select   DISTINCT amproc.amproc::regproc AS opfamily_procedure
from     pg_am am,
         pg_opfamily opf,
         pg_amproc amproc
where    opf.opfmethod = am.oid
and      amproc.amprocfamily = opf.oid
order by
         opfamily_procedure;
```

With that, we aim to prevent users easily shooting themselves by the foot.

The other problematic area was the performance, as calling
`expression_planner()` for every tuple can be very expensive. To avoid
that, it might be considered to ask users to provide a function instead of
a free form WHERE clause, such that if the function returns true, the tuple
is sent. The allowed functions need to be immutable SQL functions with bool
return type. As we can parse the  SQL functions, we should be able to allow
only functions that rely on the above mentioned procs. We can apply as many
restrictions (such as no modification query) as possible. For example, see
below:
```

CREATE OR REPLACE function filter_tuples_for_test(int) returns bool as
$body$
    select $1 > 100;
$body$
language sql immutable;

CREATE PUBLICATION pub FOR TABLE test FILTER = filter_tuples_for_tes(key);
```

In  terms of performance, calling the function should avoid calling the
`expression_planner()` and yield better performance. Though, this needs to
be verified.

If such an approach makes sense, I'd be happy to work on the patch. Please
provide me feedback.

Thanks,
Onder KALACI
Software Engineer at Microsoft &
Developing the Citus database extension for PostgreSQL

David Steele <[email-masked]>, 16 Ara 2020 Çar, 21:43 tarihinde şunu

---

From: Masahiko Sawada <[email-masked]>
Date: Tue, 29 Dec 2020 17:08:11 +0900

Hi Önder,

On Thu, Dec 17, 2020 at 3:43 PM Önder Kalacı <[email-masked]> wrote:
>
> Hi all,
>
> I'm also interested in this patch. I rebased the changes to the current master branch and attached. The rebase had two issues. First, patch-8 was conflicting, and that seems only helpful for debugging purposes during development. So, I dropped it for simplicity. Second, the changes have a conflict with `publish_via_partition_root` changes. I tried to fix the issues, but ended-up having a limitation for now. The limitation is that "cannot create publication with WHERE clause on the partitioned table without publish_via_partition_root is set to true". This restriction can be lifted, though I left out for the sake of focusing on the some issues that I observed on this patch.
>
> Please see my review:
>
> The above part can be considered the core of the logic, executed per tuple. As far as I can see, it has two downsides.
>
> First, calling `expression_planner()` for every tuple can be quite expensive. I created a sample table, loaded data and ran a quick benchmark to see its effect. I attached the very simple script that I used to reproduce the issue on my laptop. I'm pretty sure you can find nicer ways of doing similar perf tests, just sharing as a reference.
>
> The idea of the test is to add a WHERE clause to a table, but none of the tuples are filtered out. They just go through this code-path and send it to the remote node.
>
> #rows       Patched    | Master
> Secondly, probably more importantly, allowing any operator is as dangerous as allowing any function as users can create/overload operator(s). For example, assume that users create an operator which modifies the table that is being filtered out:
>
> ```
> CREATE OR REPLACE FUNCTION function_that_modifies_table(left_art INTEGER, right_arg INTEGER)
> RETURNS BOOL AS
> $$
> BEGIN
>
> With the above, we seem to be in trouble. Although the above is an extreme example, it felt useful to share to the extent of the problem. We probably cannot allow any free-form SQL to be on the filters.
>
> To overcome these issues, one approach could be to rely on known safe operators and functions. I believe the btree and hash operators should provide a pretty strong coverage across many use cases. As far as I can see, the procs that the following query returns can be our baseline:
>
> ```
> The other problematic area was the performance, as calling `expression_planner()` for every tuple can be very expensive. To avoid that, it might be considered to ask users to provide a function instead of a free form WHERE clause, such that if the function returns true, the tuple is sent. The allowed functions need to be immutable SQL functions with bool return type. As we can parse the  SQL functions, we should be able to allow only functions that rely on the above mentioned procs. We can apply as many restrictions (such as no modification query) as possible. For example, see below:
> ```
>
> In  terms of performance, calling the function should avoid calling the `expression_planner()` and yield better performance. Though, this needs to be verified.
>
> If such an approach makes sense, I'd be happy to work on the patch. Please provide me feedback.
>

You sent in your patch to pgsql-hackers on Dec 17, but you did not
post it to the next CommitFest[1] (I found the old entry of this
patch[2] but it's marked as "Returned with feedback"). If this was
intentional, then you need to take no action.  However, if you want
your patch to be reviewed as part of the upcoming CommitFest, then you
need to add it yourself before 2021-01-01 AoE[3]. Thanks for your
contributions.

Regards,

[1] https://commitfest.postgresql.org/31/
[2] https://commitfest.postgresql.org/20/1862/
[2] https://en.wikipedia.org/wiki/Anywhere_on_Earth

--
Masahiko Sawada
EnterpriseDB:  https://www.enterprisedb.com/

---

From: Önder Kalacı <[email-masked]>
Date: Mon, 4 Jan 2021 16:37:39 +0300

Hi Masahiko,

> need to add it yourself before 2021-01-01 AoE[3]. Thanks for your
Thanks for letting me know of this, I added this patch to the next commit
fest before 2021-01-01 AoE[3].

I'm also attaching the updated commits so that the tests pass on the CI.

Thanks,
Onder KALACI

---

From: Andres Freund <[email-masked]>
Date: Wed, 27 Jan 2021 18:20:32 -0800

Hi,

> So, it seems a significant overhead to me. What do you think?

That seems almost prohibitively expensive. I think at the very least
some of this work would need to be done in a cached manner, e.g. via
get_rel_sync_entry().

> as allowing any function as users can create/overload operator(s).

That's not safe, indeed. It's not even just create/overloading
operators, as far as I can tell the expression can contain just plain
function calls.

The issue also isn't primarily that the user can overload functions,
it's that logical decoding is a limited environment, and not everything
is safe to do within. You e.g. only catalog tables can be
accessed. Therefore I don't think we can allow arbitrary expressions.

> restrictions (such as no modification query) as possible. For example, see

I don't think that would get us very far.

---

From: "Euler Taveira" <[email-masked]>
Date: Sun, 31 Jan 2021 21:23:04 -0300

Hi,

This is another version of the row filter patch. Patch summary:

0001: refactor to remove dead code
0002: grammar refactor for row filter
0003: core code, documentation, and tests
0004: psql code
0005: pg_dump support
0006: debug messages (only for test purposes)
0007: measure row filter overhead (only for test purposes)

---

From: japin <[email-masked]>
Date: Mon, 01 Feb 2021 17:11:33 +0800

> 0007: measure row filter overhead (only for test purposes)

Thanks for updating the patch.  Here are some comments:

(1)
+         <para>
+          If this parameter is <literal>false</literal>, it uses the
+          <literal>WHERE</literal> clause from the partition; otherwise,the
+          <literal>WHERE</literal> clause from the partitioned table is used.
          </para>

otherwise,the -> otherwise, the

(2)
+  <para>

IMO we should indent one space here.

(3)
+
+  <para>

Same as (2).

The documentation says:

>  Columns used in the <literal>WHERE</literal> clause must be part of the
>  primary key or be covered by <literal>REPLICA IDENTITY</literal> otherwise
>  <command>UPDATE</command> and <command>DELETE</command> operations will not
>  be replicated.

Why we need this limitation? Am I missing something?

When I tested, I find that the UPDATE can be replicated, while the DELETE
cannot be replicated.  Here is my test-case:

	-- 1. Create tables and publications on publisher
	CREATE TABLE t1 (a int primary key, b int);
        CREATE TABLE t2 (a int primary key, b int);
        INSERT INTO t1 VALUES (1, 11);
        INSERT INTO t2 VALUES (1, 11);
	CREATE PUBLICATION mypub1 FOR TABLE t1;
        CREATE PUBLICATION mypub2 FOR TABLE t2 WHERE (b > 10);

	-- 2. Create tables and subscriptions on subscriber
        CREATE TABLE t1 (a int primary key, b int);
        CREATE TABLE t2 (a int primary key, b int);
        CREATE SUBSCRIPTION mysub1 CONNECTION 'host=localhost port=8765 dbname=postgres' PUBLICATION mypub1;
        CREATE SUBSCRIPTION mysub2 CONNECTION 'host=localhost port=8765 dbname=postgres' PUBLICATION mypub2;

	-- 3. Check publications on publisher
        postgres=# \dRp+
	                           Publication mypub1
	 Owner | All tables | Inserts | Updates | Deletes | Truncates | Via root
	-------+------------+---------+---------+---------+-----------+----------
	 japin | f          | t       | t       | t       | t         | f
	Tables:
	    "public.t1"
	
	                           Publication mypub2
	 Owner | All tables | Inserts | Updates | Deletes | Truncates | Via root
	-------+------------+---------+---------+---------+-----------+----------
	 japin | f          | t       | t       | t       | t         | f
	Tables:
	    "public.t2"  WHERE (b > 10)

	-- 4. Check initialization data on subscriber
	postgres=# table t1;
	 a | b
	---+----
	 1 | 11
	(1 row)
	
	postgres=# table t2;
	 a | b
	---+----
	 1 | 11
	(1 row)

	-- 5. The update on publisher
	postgres=# update t1 set b = 111 where b = 11;
	UPDATE 1
	postgres=# table t1;
	 a |  b
	---+-----
	 1 | 111
	(1 row)

	postgres=# update t2 set b = 111 where b = 11;
	UPDATE 1
	postgres=# table t2;
	 a |  b
	---+-----
	 1 | 111
	(1 row)

	-- 6. check the updated records on subscriber
	postgres=# table  t1;
	 a |  b
	---+-----
	 1 | 111
	(1 row)
	
	postgres=# table  t2;
	 a |  b
	---+-----
	 1 | 111
	(1 row)

	-- 7. Delete records on publisher
	postgres=# delete from t1 where b = 111;
	DELETE 1
	postgres=# table t1;
	 a | b
	---+---
	(0 rows)
	
	postgres=# delete from t2 where b = 111;
	DELETE 1
	postgres=# table t2;
	 a | b
	---+---
	(0 rows)

	-- 8. Check the deleted records on subscriber
	postgres=# table t1;
	 a | b
	---+---
	(0 rows)
	
	postgres=# table t2;
	 a |  b
	---+-----
	 1 | 111
	(1 row)

I do a simple debug, and find that the pgoutput_row_filter() return false when I
execute "delete from t2 where b = 111;".

Does the publication only load the REPLICA IDENTITY columns into oldtuple when we
execute DELETE? So the pgoutput_row_filter() cannot find non REPLICA IDENTITY
columns, which cause it return false, right?  If that's right, the UPDATE might
not be limitation by REPLICA IDENTITY, because all columns are in newtuple,
isn't it?

-- 
Regrads,
Japin Li.
ChengDu WenWu Information Technology Co.,Ltd.

---

From: "Euler Taveira" <[email-masked]>
Date: Mon, 01 Feb 2021 16:11:50 -0300

> Thanks for updating the patch.  Here are some comments:
Thanks for your review. I updated the documentation accordingly.

> >  <command>UPDATE</command> and <command>DELETE</command> operations will not
The UPDATE is an oversight from a previous version.

> not be limitation by REPLICA IDENTITY, because all columns are in newtuple,
No. oldtuple could possibly be available for UPDATE and DELETE. However, row
filter consider only one tuple for filtering. INSERT has only newtuple; row
filter uses it.  UPDATE has newtuple and optionally oldtuple (if it has PK or
REPLICA IDENTITY); row filter uses newtuple. DELETE optionally has only
oldtuple; row filter uses it (if available). Keep in mind, if the expression
evaluates to NULL, it returns false and the row won't be replicated.

After the commit 3696a600e2, the last patch does not apply cleanly. I'm
attaching another version to address the documentation issues.

--
Euler Taveira
EnterpriseDB:  https://www.enterprisedb.com/
From fb45140efacea0cfc9478f2c3747d9a4e5cecbd0 Mon Sep 17 00:00:00 2001
From: Euler Taveira <[email-masked]>
Date: Mon, 18 Jan 2021 11:13:23 -0300
Subject: [PATCH 1/7] Remove unused column from initial table synchronization

Column atttypmod was added in the commit 7c4f52409a, but it is not used.
The removal is safe because COPY from publisher does not need such
information.
---
 src/backend/replication/logical/tablesync.c | 5 ++---
 1 file changed, 2 insertions(+), 3 deletions(-)

diff --git a/src/backend/replication/logical/tablesync.c b/src/backend/replication/logical/tablesync.c
index 863d196fd7..a18f847ade 100644
--- a/src/backend/replication/logical/tablesync.c
+++ b/src/backend/replication/logical/tablesync.c
@@ -640,7 +640,7 @@ fetch_remote_table_info(char *nspname, char *relname,
 	StringInfoData cmd;
 	TupleTableSlot *slot;
 	Oid			tableRow[] = {OIDOID, CHAROID, CHAROID};
-	Oid			attrRow[] = {TEXTOID, OIDOID, INT4OID, BOOLOID};
+	Oid			attrRow[] = {TEXTOID, OIDOID, BOOLOID};
 	bool		isnull;
 	int			natt;
 
@@ -685,7 +685,6 @@ fetch_remote_table_info(char *nspname, char *relname,
 	appendStringInfo(&cmd,
 					 "SELECT a.attname,"
 					 "       a.atttypid,"
-					 "       a.atttypmod,"
 					 "       a.attnum = ANY(i.indkey)"
 					 "  FROM pg_catalog.pg_attribute a"
 					 "  LEFT JOIN pg_catalog.pg_index i"
@@ -718,7 +717,7 @@ fetch_remote_table_info(char *nspname, char *relname,
 		Assert(!isnull);
 		lrel->atttyps[natt] = DatumGetObjectId(slot_getattr(slot, 2, &isnull));
 		Assert(!isnull);
-		if (DatumGetBool(slot_getattr(slot, 4, &isnull)))
+		if (DatumGetBool(slot_getattr(slot, 3, &isnull)))
 			lrel->attkeys = bms_add_member(lrel->attkeys, natt);
 
2.20.1

From 4902323fca0efa8024de600853670b2a1f0e3ca0 Mon Sep 17 00:00:00 2001
From: Euler Taveira <[email-masked]>
Date: Mon, 18 Jan 2021 11:53:34 -0300
Subject: [PATCH 2/7] Rename a WHERE node

diff --git a/src/backend/parser/gram.y b/src/backend/parser/gram.y
index dd72a9fc3c..ecfd98ba5b 100644
--- a/src/backend/parser/gram.y
+++ b/src/backend/parser/gram.y
@@ -485,7 +485,7 @@ static Node *makeRecursiveViewSelect(char *relname, List *aliases, Node *query);
 %type <node>	def_arg columnElem where_clause where_or_current_clause
 				a_expr b_expr c_expr AexprConst indirection_el opt_slice_bound
 				columnref in_expr having_clause func_table xmltable array_expr
-				ExclusionWhereClause operator_def_arg
+				OptWhereClause operator_def_arg
 %type <list>	rowsfrom_item rowsfrom_list opt_col_def_list
 %type <boolean> opt_ordinality
 %type <list>	ExclusionConstraintList ExclusionConstraintElem
@@ -3806,7 +3806,7 @@ ConstraintElem:
 					$$ = (Node *)n;
 				}
 			| EXCLUDE access_method_clause '(' ExclusionConstraintList ')'
-				opt_c_include opt_definition OptConsTableSpace  ExclusionWhereClause
+				opt_c_include opt_definition OptConsTableSpace  OptWhereClause
 				ConstraintAttributeSpec
 				{
 					Constraint *n = makeNode(Constraint);
@@ -3908,7 +3908,7 @@ ExclusionConstraintElem: index_elem WITH any_operator
 			}
 		;
 
-ExclusionWhereClause:
+OptWhereClause:
2.20.1

From a3693dc370e80166c5fb8ba08c765a1559aaf89c Mon Sep 17 00:00:00 2001
From: Euler Taveira <[email-masked]>
Date: Mon, 18 Jan 2021 12:07:51 -0300
Subject: [PATCH 3/7] Row filter for logical replication

This feature adds row filter for publication tables. When you define or
modify a publication you can optionally filter rows that does not
satisfy a WHERE condition. It allows you to partially replicate a
database or set of tables. The row filter is per table which means that
you can define different row filters for different tables. A new row
filter can be added simply by informing the WHERE clause after the table
name. The WHERE expression must be enclosed by parentheses.

The WHERE clause should contain only columns that are part of the
primary key or that are covered by REPLICA IDENTITY. Otherwise, UPDATEs
and DELETEs won't be replicated. For simplicity, functions are not
allowed; it could possibly be addressed in another patch.

If you choose to do the initial table synchronization, only data that
satisfies the row filters is sent. If the subscription has several
publications in which a table has been published with different WHERE
clauses, rows must satisfy all expressions to be copied.

If your publication contains partitioned table, the parameter
publish_via_partition_root determines if it uses the partition row
filter (if the parameter is false -- default) or the partitioned table
row filter.
---
 doc/src/sgml/catalogs.sgml                  |   8 +
 doc/src/sgml/ref/alter_publication.sgml     |  11 +-
 doc/src/sgml/ref/create_publication.sgml    |  36 +++-
 doc/src/sgml/ref/create_subscription.sgml   |   8 +-
 src/backend/catalog/pg_publication.c        | 110 +++++++++++-
 src/backend/commands/publicationcmds.c      |  96 ++++++----
 src/backend/parser/gram.y                   |  28 ++-
 src/backend/parser/parse_agg.c              |  10 ++
 src/backend/parser/parse_expr.c             |  13 ++
 src/backend/parser/parse_func.c             |   3 +
 src/backend/replication/logical/tablesync.c |  93 +++++++++-
 src/backend/replication/logical/worker.c    |  14 +-
 src/backend/replication/pgoutput/pgoutput.c | 179 ++++++++++++++++--
 src/include/catalog/pg_publication.h        |  10 +-
 src/include/catalog/pg_publication_rel.h    |   6 +
 src/include/nodes/nodes.h                   |   1 +
 src/include/nodes/parsenodes.h              |  11 +-
 src/include/parser/parse_node.h             |   1 +
 src/include/replication/logicalrelation.h   |   2 +
 src/test/regress/expected/publication.out   |  32 ++++
 src/test/regress/sql/publication.sql        |  23 +++
 src/test/subscription/t/020_row_filter.pl   | 190 ++++++++++++++++++++
 22 files changed, 800 insertions(+), 85 deletions(-)
 create mode 100644 src/test/subscription/t/020_row_filter.pl

diff --git a/doc/src/sgml/catalogs.sgml b/doc/src/sgml/catalogs.sgml
index 865e826fb0..1ea4076219 100644
--- a/doc/src/sgml/catalogs.sgml
+++ b/doc/src/sgml/catalogs.sgml
@@ -5493,6 +5493,14 @@ SCRAM-SHA-256$<replaceable>&lt;iteration count&gt;</replaceable>:<replaceable>&l
        are simple references.
       </para></entry>
      </row>
+
+     <row>
+      <entry><structfield>prqual</structfield></entry>
+      <entry><type>pg_node_tree</type></entry>
+      <entry></entry>
+      <entry>Expression tree (in <function>nodeToString()</function>
+      representation) for the relation's qualifying condition</entry>
+     </row>
     </tbody>
    </tgroup>
   </table>
diff --git a/doc/src/sgml/ref/alter_publication.sgml b/doc/src/sgml/ref/alter_publication.sgml
index faa114b2c6..ca091aae33 100644
--- a/doc/src/sgml/ref/alter_publication.sgml
+++ b/doc/src/sgml/ref/alter_publication.sgml
@@ -21,8 +21,8 @@ PostgreSQL documentation
 
  <refsynopsisdiv>
 <synopsis>
-ALTER PUBLICATION <replaceable class="parameter">name</replaceable> ADD TABLE [ ONLY ] <replaceable class="parameter">table_name</replaceable> [ * ] [, ...]
-ALTER PUBLICATION <replaceable class="parameter">name</replaceable> SET TABLE [ ONLY ] <replaceable class="parameter">table_name</replaceable> [ * ] [, ...]
+ALTER PUBLICATION <replaceable class="parameter">name</replaceable> ADD TABLE [ ONLY ] <replaceable class="parameter">table_name</replaceable> [ * ] [ WHERE ( <replaceable class="parameter">expression</replaceable> ) ] [, ...]
+ALTER PUBLICATION <replaceable class="parameter">name</replaceable> SET TABLE [ ONLY ] <replaceable class="parameter">table_name</replaceable> [ * ] [ WHERE ( <replaceable class="parameter">expression</replaceable> ) ] [, ...]
 ALTER PUBLICATION <replaceable class="parameter">name</replaceable> DROP TABLE [ ONLY ] <replaceable class="parameter">table_name</replaceable> [ * ] [, ...]
 ALTER PUBLICATION <replaceable class="parameter">name</replaceable> SET ( <replaceable class="parameter">publication_parameter</replaceable> [= <replaceable class="parameter">value</replaceable>] [, ... ] )
 ALTER PUBLICATION <replaceable class="parameter">name</replaceable> OWNER TO { <replaceable>new_owner</replaceable> | CURRENT_ROLE | CURRENT_USER | SESSION_USER }
@@ -92,7 +92,12 @@ ALTER PUBLICATION <replaceable class="parameter">name</replaceable> RENAME TO <r
       table name, only that table is affected.  If <literal>ONLY</literal> is not
       specified, the table and all its descendant tables (if any) are
       affected.  Optionally, <literal>*</literal> can be specified after the table
-      name to explicitly indicate that descendant tables are included.
+      name to explicitly indicate that descendant tables are included. If the
+      optional <literal>WHERE</literal> clause is specified, rows that do not
+      satisfy the <replaceable class="parameter">expression</replaceable> will
+      not be published. Note that parentheses are required around the
+      expression. The <replaceable class="parameter">expression</replaceable>
+      is executed with the role used for the replication connection.
      </para>
     </listitem>
    </varlistentry>
diff --git a/doc/src/sgml/ref/create_publication.sgml b/doc/src/sgml/ref/create_publication.sgml
index ff82fbca55..2136545b04 100644
--- a/doc/src/sgml/ref/create_publication.sgml
+++ b/doc/src/sgml/ref/create_publication.sgml
@@ -22,7 +22,7 @@ PostgreSQL documentation
  <refsynopsisdiv>
 <synopsis>
 CREATE PUBLICATION <replaceable class="parameter">name</replaceable>
-    [ FOR TABLE [ ONLY ] <replaceable class="parameter">table_name</replaceable> [ * ] [, ...]
+    [ FOR TABLE [ ONLY ] <replaceable class="parameter">table_name</replaceable> [ * ] [ WHERE ( <replaceable class="parameter">expression</replaceable> ) ] [, ...]
       | FOR ALL TABLES ]
     [ WITH ( <replaceable class="parameter">publication_parameter</replaceable> [= <replaceable class="parameter">value</replaceable>] [, ... ] ) ]
 </synopsis>
@@ -71,6 +71,10 @@ CREATE PUBLICATION <replaceable class="parameter">name</replaceable>
       This does not apply to a partitioned table, however.  The partitions of
       a partitioned table are always implicitly considered part of the
       publication, so they are never explicitly added to the publication.
+      If the optional <literal>WHERE</literal> clause is specified, rows that do 
+      not satisfy the <replaceable class="parameter">expression</replaceable> 
+      will not be published. Note that parentheses are required around the 
+      expression.
      </para>
 
      <para>
@@ -131,9 +135,15 @@ CREATE PUBLICATION <replaceable class="parameter">name</replaceable>
           on its partitions) contained in the publication will be published
           using the identity and schema of the partitioned table rather than
           that of the individual partitions that are actually changed; the
-          latter is the default.  Enabling this allows the changes to be
-          replicated into a non-partitioned table or a partitioned table
-          consisting of a different set of partitions.
+          latter is the default (<literal>false</literal>).  Enabling this
+          allows the changes to be replicated into a non-partitioned table or a
+          partitioned table consisting of a different set of partitions.
+         </para>
+
+         <para>
+          If this parameter is <literal>false</literal>, it uses the
+          <literal>WHERE</literal> clause from the partition; otherwise, the
+          <literal>WHERE</literal> clause from the partitioned table is used.
          </para>
 
          <para>
@@ -182,6 +192,12 @@ CREATE PUBLICATION <replaceable class="parameter">name</replaceable>
    disallowed on those tables.
   </para>
 
+  <para>
+   Columns used in the <literal>WHERE</literal> clause must be part of the
+   primary key or be covered by <literal>REPLICA IDENTITY</literal> otherwise
+   <command>DELETE</command> operations will not be replicated.
+  </para>
+
   <para>
    For an <command>INSERT ... ON CONFLICT</command> command, the publication will
    publish the operation that actually results from the command.  So depending
@@ -197,6 +213,11 @@ CREATE PUBLICATION <replaceable class="parameter">name</replaceable>
   <para>
    <acronym>DDL</acronym> operations are not published.
   </para>
+
+  <para>
+   The <literal>WHERE</literal> clause expression is executed with the role used
+   for the replication connection.
+  </para>
  </refsect1>
 
  <refsect1>
@@ -209,6 +230,13 @@ CREATE PUBLICATION mypublication FOR TABLE users, departments;
 </programlisting>
   </para>
 
+  <para>
+   Create a publication that publishes all changes from active departments:
+<programlisting>
+CREATE PUBLICATION active_departments FOR TABLE departments WHERE (active IS TRUE);
+</programlisting>
+  </para>
+
   <para>
    Create a publication that publishes all changes in all tables:
 <programlisting>
diff --git a/doc/src/sgml/ref/create_subscription.sgml b/doc/src/sgml/ref/create_subscription.sgml
index e812beee37..b8f4ea5603 100644
--- a/doc/src/sgml/ref/create_subscription.sgml
+++ b/doc/src/sgml/ref/create_subscription.sgml
@@ -102,7 +102,13 @@ CREATE SUBSCRIPTION <replaceable class="parameter">subscription_name</replaceabl
          <para>
           Specifies whether the existing data in the publications that are
           being subscribed to should be copied once the replication starts.
-          The default is <literal>true</literal>.
+          The default is <literal>true</literal>. If any table in the
+          publications has a <literal>WHERE</literal> clause, rows that do not
+          satisfy the <replaceable class="parameter">expression</replaceable>
+          will not be copied. If the subscription has several publications in
+          which a table has been published with different
+          <literal>WHERE</literal> clauses, rows must satisfy all expressions
+          to be copied.
          </para>
         </listitem>
        </varlistentry>
diff --git a/src/backend/catalog/pg_publication.c b/src/backend/catalog/pg_publication.c
index 84d2efcfd2..fd9549a630 100644
--- a/src/backend/catalog/pg_publication.c
+++ b/src/backend/catalog/pg_publication.c
@@ -33,6 +33,11 @@
 #include "catalog/pg_type.h"
 #include "funcapi.h"
 #include "miscadmin.h"
+
+#include "parser/parse_clause.h"
+#include "parser/parse_collate.h"
+#include "parser/parse_relation.h"
+
 #include "utils/array.h"
 #include "utils/builtins.h"
 #include "utils/catcache.h"
@@ -141,18 +146,20 @@ pg_relation_is_publishable(PG_FUNCTION_ARGS)
  * Insert new publication / relation mapping.
  */
 ObjectAddress
-publication_add_relation(Oid pubid, Relation targetrel,
+publication_add_relation(Oid pubid, PublicationRelationQual *targetrel,
 						 bool if_not_exists)
 {
 	Relation	rel;
 	HeapTuple	tup;
 	Datum		values[Natts_pg_publication_rel];
 	bool		nulls[Natts_pg_publication_rel];
-	Oid			relid = RelationGetRelid(targetrel);
 	Oid			prrelid;
 	Publication *pub = GetPublication(pubid);
 	ObjectAddress myself,
 				referenced;
+	ParseState *pstate;
+	ParseNamespaceItem *nsitem;
+	Node	   *whereclause;

@@ -161,7 +168,7 @@ publication_add_relation(Oid pubid, Relation targetrel,
 	 * duplicates, it's here just to provide nicer error message in common
 	 * case. The real protection is the unique key on the catalog.
 	 */
-	if (SearchSysCacheExists2(PUBLICATIONRELMAP, ObjectIdGetDatum(relid),
+	if (SearchSysCacheExists2(PUBLICATIONRELMAP, ObjectIdGetDatum(targetrel->relid),
 							  ObjectIdGetDatum(pubid)))
 	{
 		table_close(rel, RowExclusiveLock);
@@ -172,10 +179,27 @@ publication_add_relation(Oid pubid, Relation targetrel,
 		ereport(ERROR,
 				(errcode(ERRCODE_DUPLICATE_OBJECT),
 				 errmsg("relation \"%s\" is already member of publication \"%s\"",
-						RelationGetRelationName(targetrel), pub->name)));
+						RelationGetRelationName(targetrel->relation), pub->name)));
 	}
 
+	nsitem = addRangeTableEntryForRelation(pstate, targetrel->relation,
+	addNSItemToQuery(pstate, nsitem, false, true, true);
 
 	/* Form a tuple. */
 	memset(values, 0, sizeof(values));
@@ -187,7 +211,13 @@ publication_add_relation(Oid pubid, Relation targetrel,
 	values[Anum_pg_publication_rel_prpubid - 1] =
 		ObjectIdGetDatum(pubid);
 	values[Anum_pg_publication_rel_prrelid - 1] =
-		ObjectIdGetDatum(relid);
+		ObjectIdGetDatum(targetrel->relid);
+
+	/* Add qualifications, if available */
+	if (whereclause)
+		values[Anum_pg_publication_rel_prqual - 1] = CStringGetTextDatum(nodeToString(whereclause));
+	else
+		nulls[Anum_pg_publication_rel_prqual - 1] = true;

@@ -202,14 +232,20 @@ publication_add_relation(Oid pubid, Relation targetrel,
 	recordDependencyOn(&myself, &referenced, DEPENDENCY_AUTO);
 
 	/* Add dependency on the relation */
-	ObjectAddressSet(referenced, RelationRelationId, relid);
+	ObjectAddressSet(referenced, RelationRelationId, targetrel->relid);
 	recordDependencyOn(&myself, &referenced, DEPENDENCY_AUTO);

 
 	return myself;
 }
@@ -304,6 +340,64 @@ GetPublicationRelations(Oid pubid, PublicationPartOpt pub_partopt)
 	return result;
 }
 
+/*
+ * Gets list of PublicationRelationQuals for a publication.
+ */
+List *
+GetPublicationRelationQuals(Oid pubid)
+{
+	List	   *result;
+	Relation	pubrelsrel;
+	ScanKeyData scankey;
+	SysScanDesc scan;
+	HeapTuple	tup;
+
+	/* Find all publications associated with the relation. */
+	pubrelsrel = table_open(PublicationRelRelationId, AccessShareLock);
+
+	ScanKeyInit(&scankey,
+				Anum_pg_publication_rel_prpubid,
+				BTEqualStrategyNumber, F_OIDEQ,
+				ObjectIdGetDatum(pubid));
+
+	scan = systable_beginscan(pubrelsrel, PublicationRelPrrelidPrpubidIndexId,
+							  true, NULL, 1, &scankey);
+
+	result = NIL;
+	while (HeapTupleIsValid(tup = systable_getnext(scan)))
+	{
+		Form_pg_publication_rel pubrel;
+		PublicationRelationQual *prq;
+		Datum		value_datum;
+		char	   *qual_value;
+		Node	   *qual_expr;
+		bool		isnull;
+
+		pubrel = (Form_pg_publication_rel) GETSTRUCT(tup);
+
+		value_datum = heap_getattr(tup, Anum_pg_publication_rel_prqual, RelationGetDescr(pubrelsrel), &isnull);
+		if (!isnull)
+		{
+			qual_value = TextDatumGetCString(value_datum);
+			qual_expr = (Node *) stringToNode(qual_value);
+		}
+		else
+			qual_expr = NULL;
+
+		prq = palloc(sizeof(PublicationRelationQual));
+		prq->relid = pubrel->prrelid;
+		/* table will be opened in AlterPublicationTables */
+		prq->relation = NULL;
+		prq->whereClause = qual_expr;
+		result = lappend(result, prq);
+	}
+
+	systable_endscan(scan);
+	table_close(pubrelsrel, AccessShareLock);
+
+	return result;
+}
+
 /*
  * Gets list of publication oids for publications marked as FOR ALL TABLES.
  */
diff --git a/src/backend/commands/publicationcmds.c b/src/backend/commands/publicationcmds.c
index 95c253c8e0..a5eccbbfb5 100644
--- a/src/backend/commands/publicationcmds.c
+++ b/src/backend/commands/publicationcmds.c
@@ -372,6 +372,28 @@ AlterPublicationTables(AlterPublicationStmt *stmt, Relation rel,

+	/*
+	 * Although ALTER PUBLICATION grammar allows WHERE clause to be specified
+	 * for DROP TABLE action, it doesn't make sense to allow it. We implement
+	 * this restriction here, instead of complicating the grammar to enforce
+	 * it.
+	 */
+	if (stmt->tableAction == DEFELEM_DROP)
+	{
+		ListCell   *lc;
+
+		foreach(lc, stmt->tables)
+		{
+			PublicationTable *t = lfirst(lc);
+
+			if (t->whereClause)
+				ereport(ERROR,
+						(errcode(ERRCODE_OBJECT_NOT_IN_PREREQUISITE_STATE),
+						 errmsg("cannot use a WHERE clause when removing table from publication \"%s\"",
+								NameStr(pubform->pubname))));
+		}
+	}
+
 	rels = OpenTableList(stmt->tables);
 
 	if (stmt->tableAction == DEFELEM_ADD)
@@ -389,27 +411,22 @@ AlterPublicationTables(AlterPublicationStmt *stmt, Relation rel,
 		foreach(oldlc, oldrelids)
 		{
 			Oid			oldrelid = lfirst_oid(oldlc);
-			ListCell   *newlc;
-			bool		found = false;
+			PublicationRelationQual *oldrel;
 
-			foreach(newlc, rels)
-			{
-				Relation	newrel = (Relation) lfirst(newlc);
+			/*
+			 * Remove all publication-table mappings.  We could possibly
+			 * remove (i) tables that are not found in the new table list and
+			 * (ii) tables that are being re-added with a different qual
+			 * expression. For (ii), simply updating the existing tuple is not
+			 * enough, because of qual expression dependencies.
+			 */
+			oldrel = palloc(sizeof(PublicationRelationQual));
+			oldrel->relid = oldrelid;
+			oldrel->whereClause = NULL;
+			oldrel->relation = table_open(oldrel->relid,
+										  ShareUpdateExclusiveLock);
 
-				{
-					found = true;
-				}
-			}
-			{
-			}
+			delrels = lappend(delrels, oldrel);
 		}
 
 		/* And drop them. */
@@ -509,13 +526,15 @@ OpenTableList(List *tables)
 	List	   *relids = NIL;
 	List	   *rels = NIL;
 	ListCell   *lc;
+	PublicationRelationQual *prq;
 
 	/*
 	 * Open, share-lock, and check all the explicitly-specified relations
 	 */
 	foreach(lc, tables)
 	{
-		RangeVar   *rv = castNode(RangeVar, lfirst(lc));
+		PublicationTable *t = lfirst(lc);
+		RangeVar   *rv = castNode(RangeVar, t->relation);
 		bool		recurse = rv->inh;
 		Relation	rel;
 		Oid			myrelid;
@@ -538,8 +557,11 @@ OpenTableList(List *tables)
 			table_close(rel, ShareUpdateExclusiveLock);
 			continue;
 		}
-
-		rels = lappend(rels, rel);
+		prq = palloc(sizeof(PublicationRelationQual));
+		prq->relid = myrelid;
+		prq->relation = rel;
+		prq->whereClause = t->whereClause;
+		rels = lappend(rels, prq);
 		relids = lappend_oid(relids, myrelid);
 
 		/*
@@ -572,7 +594,12 @@ OpenTableList(List *tables)
 
 				/* find_all_inheritors already got lock */
 				rel = table_open(childrelid, NoLock);
-				rels = lappend(rels, rel);
+				prq = palloc(sizeof(PublicationRelationQual));
+				prq->relid = childrelid;
+				prq->relation = rel;
+				/* child inherits WHERE clause from parent */
+				prq->whereClause = t->whereClause;
+				rels = lappend(rels, prq);
 				relids = lappend_oid(relids, childrelid);
 			}
 		}
@@ -593,10 +620,12 @@ CloseTableList(List *rels)
 
 	foreach(lc, rels)
 	{
-		Relation	rel = (Relation) lfirst(lc);
+		PublicationRelationQual *prq = (PublicationRelationQual *) lfirst(lc);
 
-		table_close(rel, NoLock);
+		table_close(prq->relation, NoLock);
 	}
+
+	list_free_deep(rels);
 }
 
 /*
@@ -612,15 +641,15 @@ PublicationAddTables(Oid pubid, List *rels, bool if_not_exists,
 
 	foreach(lc, rels)
 	{
-		Relation	rel = (Relation) lfirst(lc);
+		PublicationRelationQual *prq = (PublicationRelationQual *) lfirst(lc);
 		ObjectAddress obj;
 
 		/* Must be owner of the table or superuser. */
-		if (!pg_class_ownercheck(RelationGetRelid(rel), GetUserId()))
-			aclcheck_error(ACLCHECK_NOT_OWNER, get_relkind_objtype(rel->rd_rel->relkind),
-						   RelationGetRelationName(rel));
+		if (!pg_class_ownercheck(prq->relid, GetUserId()))
+			aclcheck_error(ACLCHECK_NOT_OWNER, get_relkind_objtype(prq->relation->rd_rel->relkind),
+						   RelationGetRelationName(prq->relation));
 
-		obj = publication_add_relation(pubid, rel, if_not_exists);
+		obj = publication_add_relation(pubid, prq, if_not_exists);
 		if (stmt)
 		{
 			EventTriggerCollectSimpleCommand(obj, InvalidObjectAddress,
@@ -644,11 +673,10 @@ PublicationDropTables(Oid pubid, List *rels, bool missing_ok)
 
 	foreach(lc, rels)
 	{
-		Relation	rel = (Relation) lfirst(lc);
-		Oid			relid = RelationGetRelid(rel);
+		PublicationRelationQual *prq = (PublicationRelationQual *) lfirst(lc);
 
 		prid = GetSysCacheOid2(PUBLICATIONRELMAP, Anum_pg_publication_rel_oid,
-							   ObjectIdGetDatum(relid),
+							   ObjectIdGetDatum(prq->relid),
 							   ObjectIdGetDatum(pubid));
 		if (!OidIsValid(prid))
 		{
@@ -658,7 +686,7 @@ PublicationDropTables(Oid pubid, List *rels, bool missing_ok)
 			ereport(ERROR,
 					(errcode(ERRCODE_UNDEFINED_OBJECT),
 					 errmsg("relation \"%s\" is not part of the publication",
-							RelationGetRelationName(rel))));
+							RelationGetRelationName(prq->relation))));
 		}
 
 		ObjectAddressSet(obj, PublicationRelRelationId, prid);
diff --git a/src/backend/parser/gram.y b/src/backend/parser/gram.y
index ecfd98ba5b..54d8f26a64 100644
--- a/src/backend/parser/gram.y
+++ b/src/backend/parser/gram.y
@@ -416,12 +416,12 @@ static Node *makeRecursiveViewSelect(char *relname, List *aliases, Node *query);
 				transform_element_list transform_type_list
 				TriggerTransitions TriggerReferencing
 				vacuum_relation_list opt_vacuum_relation_list
-				drop_option_list
+				drop_option_list publication_table_list

 %type <list>	opt_fdw_options fdw_options
 %type <defelt>	fdw_option
@@ -3806,7 +3806,7 @@ ConstraintElem:
 					$$ = (Node *)n;
 				}
 			| EXCLUDE access_method_clause '(' ExclusionConstraintList ')'
-				opt_c_include opt_definition OptConsTableSpace  OptWhereClause
+				opt_c_include opt_definition OptConsTableSpace OptWhereClause
 				ConstraintAttributeSpec
 				{
 					Constraint *n = makeNode(Constraint);
@@ -9499,7 +9499,7 @@ opt_publication_for_tables:
 		;
 
 publication_for_tables:
-			FOR TABLE relation_expr_list
+			FOR TABLE publication_table_list
 				{
 					$$ = (Node *) $3;
 				}
@@ -9530,7 +9530,7 @@ AlterPublicationStmt:
 					n->options = $5;
 					$$ = (Node *)n;
 				}
-			| ALTER PUBLICATION name ADD_P TABLE relation_expr_list
+			| ALTER PUBLICATION name ADD_P TABLE publication_table_list
 				{
 					AlterPublicationStmt *n = makeNode(AlterPublicationStmt);
 					n->pubname = $3;
@@ -9538,7 +9538,7 @@ AlterPublicationStmt:
 					n->tableAction = DEFELEM_ADD;
 					$$ = (Node *)n;
 				}
-			| ALTER PUBLICATION name SET TABLE relation_expr_list
+			| ALTER PUBLICATION name SET TABLE publication_table_list
 				{
 					AlterPublicationStmt *n = makeNode(AlterPublicationStmt);
 					n->pubname = $3;
@@ -9546,7 +9546,7 @@ AlterPublicationStmt:
 					n->tableAction = DEFELEM_SET;
 					$$ = (Node *)n;
 				}
-			| ALTER PUBLICATION name DROP TABLE relation_expr_list
+			| ALTER PUBLICATION name DROP TABLE publication_table_list
 				{
 					AlterPublicationStmt *n = makeNode(AlterPublicationStmt);
 					n->pubname = $3;
@@ -9556,6 +9556,20 @@ AlterPublicationStmt:
 				}
 		;
 
+publication_table_list:
index fd08b9eeff..06b184f404 100644
 
 			break;
 
 		case EXPR_KIND_CYCLE_MARK:
 			errkind = true;
@@ -940,6 +947,9 @@ transformWindowFuncCall(ParseState *pstate, WindowFunc *wfunc,
 		case EXPR_KIND_CYCLE_MARK:
 			errkind = true;
 			break;
+		case EXPR_KIND_PUBLICATION_WHERE:
+			err = _("window functions are not allowed in publication WHERE expressions");
+			break;
 
 			/*
 			 * There is intentionally no default: case here, so that the
diff --git a/src/backend/parser/parse_expr.c b/src/backend/parser/parse_expr.c
index 6c87783b2c..4b4fb0dc4d 100644
--- a/src/backend/parser/parse_expr.c
+++ b/src/backend/parser/parse_expr.c
@@ -119,6 +119,13 @@ transformExprRecurse(ParseState *pstate, Node *expr)
 	/* Guard against stack overflow due to overly complex expressions */
 	check_stack_depth();
 
+	/* Functions are not allowed in publication WHERE clauses */
+	if (pstate->p_expr_kind == EXPR_KIND_PUBLICATION_WHERE && nodeTag(expr) == T_FuncCall)
+		ereport(ERROR,
+				(errcode(ERRCODE_FEATURE_NOT_SUPPORTED),
+				 errmsg("functions are not allowed in publication WHERE expressions"),
+				 parser_errposition(pstate, exprLocation(expr))));
+
 	switch (nodeTag(expr))
 	{
 		case T_ColumnRef:
@@ -508,6 +515,7 @@ transformColumnRef(ParseState *pstate, ColumnRef *cref)
 		case EXPR_KIND_COPY_WHERE:
 		case EXPR_KIND_GENERATED_COLUMN:
 		case EXPR_KIND_CYCLE_MARK:
+		case EXPR_KIND_PUBLICATION_WHERE:
 			/* okay */
 			break;
 
@@ -1765,6 +1773,9 @@ transformSubLink(ParseState *pstate, SubLink *sublink)
 
 			/*
 			 * There is intentionally no default: case here, so that the
@@ -3048,6 +3059,8 @@ ParseExprKindName(ParseExprKind exprKind)
 			return "GENERATED AS";
 		case EXPR_KIND_CYCLE_MARK:
 			return "CYCLE";
+		case EXPR_KIND_PUBLICATION_WHERE:
+			return "publication expression";
 
 			/*
 			 * There is intentionally no default: case here, so that the
diff --git a/src/backend/parser/parse_func.c b/src/backend/parser/parse_func.c
index 37cebc7d82..bf909d6ed5 100644
--- a/src/backend/parser/parse_func.c
+++ b/src/backend/parser/parse_func.c
@@ -2530,6 +2530,9 @@ check_srf_call_placement(ParseState *pstate, Node *last_srf, int location)
 		case EXPR_KIND_CYCLE_MARK:
 			errkind = true;
 			break;
+		case EXPR_KIND_PUBLICATION_WHERE:
+			err = _("set-returning functions are not allowed in publication WHERE expressions");
+			break;
 
 			/*
 			 * There is intentionally no default: case here, so that the
diff --git a/src/backend/replication/logical/tablesync.c b/src/backend/replication/logical/tablesync.c
index a18f847ade..63d00a75ed 100644
--- a/src/backend/replication/logical/tablesync.c
+++ b/src/backend/replication/logical/tablesync.c
@@ -630,19 +630,25 @@ copy_read_data(void *outbuf, int minread, int maxread)
 
 /*
  * Get information about remote relation in similar fashion the RELATION
- * message provides during replication.
+ * message provides during replication. This function also returns the relation
+ * qualifications to be used in COPY command.
  */
 static void
 fetch_remote_table_info(char *nspname, char *relname,
-						LogicalRepRelation *lrel)
+						LogicalRepRelation *lrel, List **qual)
 {
 	WalRcvExecResult *res;
 	StringInfoData cmd;
 	TupleTableSlot *slot;
 	Oid			tableRow[] = {OIDOID, CHAROID, CHAROID};
 	Oid			attrRow[] = {TEXTOID, OIDOID, BOOLOID};
+	Oid			qualRow[] = {TEXTOID};
 	bool		isnull;
 	int			natt;
+	ListCell   *lc;
+	bool		first;
+
+	memset(lrel, 0, sizeof(LogicalRepRelation));
 
 	lrel->nspname = nspname;
 	lrel->relname = relname;
@@ -731,6 +737,51 @@ fetch_remote_table_info(char *nspname, char *relname,
 
 	lrel->natts = natt;
 
@@ -745,6 +796,7 @@ copy_table(Relation rel)
 	CopyFromState cstate;
@@ -753,7 +805,7 @@ copy_table(Relation rel)

@@ -762,16 +814,23 @@ copy_table(Relation rel)
 
+	/* List of columns for COPY */
+	attnamelist = make_copy_attnamelist(relmapentry);
+
 	/* Start copy on the publisher. */
 	initStringInfo(&cmd);
-	if (lrel.relkind == RELKIND_RELATION)
+
+	/* Regular table with no row filter */
+	if (lrel.relkind == RELKIND_RELATION && list_length(qual) == 0)
 		appendStringInfo(&cmd, "COPY %s TO STDOUT",
 						 quote_qualified_identifier(lrel.nspname, lrel.relname));
 	else
 	{
 		/*
 		 * For non-tables, we need to do COPY (SELECT ...), but we can't just
-		 * do SELECT * because we need to not copy generated columns.
+		 * do SELECT * because we need to not copy generated columns. For
+		 * tables with any row filters, build a SELECT query with AND'ed row
+		 * filters for COPY.
 		 */
 		appendStringInfoString(&cmd, "COPY (SELECT ");
 		for (int i = 0; i < lrel.natts; i++)
@@ -780,9 +839,31 @@ copy_table(Relation rel)
 			if (i < lrel.natts - 1)
 				appendStringInfoString(&cmd, ", ");
 		}
-		appendStringInfo(&cmd, " FROM %s) TO STDOUT",
+		appendStringInfo(&cmd, " FROM %s",
 						 quote_qualified_identifier(lrel.nspname, lrel.relname));
+		/* list of AND'ed filters */
+		if (list_length(qual) > 0)
+		{
+			ListCell   *lc;
+			bool		first = true;
+
+			appendStringInfoString(&cmd, " WHERE ");
+			foreach(lc, qual)
+			{
+				char	   *q = strVal(lfirst(lc));
+
+				if (first)
+					first = false;
+				else
+					appendStringInfoString(&cmd, " AND ");
+				appendStringInfo(&cmd, "%s", q);
+			}
+			list_free_deep(qual);
+		}
+
+		appendStringInfoString(&cmd, ") TO STDOUT");
 	}
+
 	res = walrcv_exec(wrconn, cmd.data, 0, NULL);
 	pfree(cmd.data);
 	if (res->status != WALRCV_OK_COPY_OUT)
diff --git a/src/backend/replication/logical/worker.c b/src/backend/replication/logical/worker.c
index eb7db89cef..6092ae0df0 100644
--- a/src/backend/replication/logical/worker.c
+++ b/src/backend/replication/logical/worker.c
@@ -340,8 +340,8 @@ handle_streamed_transaction(LogicalRepMsgType action, StringInfo s)
  *
  * This is based on similar code in copy.c
  */
-static EState *
-create_estate_for_relation(LogicalRepRelMapEntry *rel)
+EState *
+create_estate_for_relation(Relation rel)
 {
 	EState	   *estate;
 	RangeTblEntry *rte;
@@ -350,8 +350,8 @@ create_estate_for_relation(LogicalRepRelMapEntry *rel)

@@ -1176,7 +1176,7 @@ apply_handle_insert(StringInfo s)
 	}
 
@@ -1301,7 +1301,7 @@ apply_handle_update(StringInfo s)
 
@@ -1458,7 +1458,7 @@ apply_handle_delete(StringInfo s)
 
 	/* Initialize the executor state. */
-	estate = create_estate_for_relation(rel);
+	estate = create_estate_for_relation(rel->localrel);
 	remoteslot = ExecInitExtraTupleSlot(estate,
 										RelationGetDescr(rel->localrel),
 										&TTSOpsVirtual);
diff --git a/src/backend/replication/pgoutput/pgoutput.c b/src/backend/replication/pgoutput/pgoutput.c
index 79765f9696..dd6f3bda3a 100644
--- a/src/backend/replication/pgoutput/pgoutput.c
+++ b/src/backend/replication/pgoutput/pgoutput.c
@@ -15,12 +15,22 @@
 #include "access/tupconvert.h"
 #include "catalog/partition.h"
 #include "catalog/pg_publication.h"
+#include "catalog/pg_publication_rel.h"
+#include "catalog/pg_type.h"
 #include "commands/defrem.h"
+#include "executor/executor.h"
 #include "fmgr.h"
+#include "nodes/execnodes.h"
+#include "nodes/nodeFuncs.h"
+#include "optimizer/planner.h"
+#include "optimizer/optimizer.h"
+#include "parser/parse_coerce.h"
 #include "replication/logical.h"
 #include "replication/logicalproto.h"
+#include "replication/logicalrelation.h"
 #include "replication/origin.h"
 #include "replication/pgoutput.h"
+#include "utils/builtins.h"
 #include "utils/int8.h"
 #include "utils/inval.h"
 #include "utils/lsyscache.h"
@@ -57,6 +67,8 @@ static void pgoutput_stream_abort(struct LogicalDecodingContext *ctx,
 static void pgoutput_stream_commit(struct LogicalDecodingContext *ctx,
 								   ReorderBufferTXN *txn,
 								   XLogRecPtr commit_lsn);
+static bool pgoutput_row_filter(Relation relation, HeapTuple oldtuple,
+								HeapTuple newtuple, List *rowfilter);
 
 static bool publications_valid;
 static bool in_streaming;
@@ -98,6 +110,7 @@ typedef struct RelationSyncEntry
 
 	bool		replicate_valid;
 
 	/*
 	 * OID of the relation to publish changes as.  For a partition, this may
@@ -121,7 +134,7 @@ static HTAB *RelationSyncCache = NULL;
 
 static void init_rel_sync_cache(MemoryContext decoding_context);
 static void cleanup_rel_sync_cache(TransactionId xid, bool is_commit);
-static RelationSyncEntry *get_rel_sync_entry(PGOutputData *data, Oid relid);
+static RelationSyncEntry *get_rel_sync_entry(PGOutputData *data, Relation rel);
 static void rel_sync_cache_relation_cb(Datum arg, Oid relid);
 static void rel_sync_cache_publication_cb(Datum arg, int cacheid,
 										  uint32 hashvalue);
@@ -489,6 +502,99 @@ send_relation_and_attrs(Relation relation, TransactionId xid,
 	OutputPluginWrite(ctx, false);
 }
 
+/*
+ * Evaluates row filter.
+ *
+ * If the row filter evaluates to NULL, it is taken as false i.e. the change
+ * isn't replicated.
+ */
+static inline bool
+pgoutput_row_filter_exec_expr(ExprState *state, ExprContext *econtext)
+{
+	Datum		ret;
+	bool		isnull;
+
+	Assert(state != NULL);
+
+	ret = ExecEvalExprSwitchContext(state, econtext, &isnull);
+
+	if (isnull)
+		return false;
+
+	return DatumGetBool(ret);
+}
+
+/*
+ * Change is checked against the row filter, if any.
+ *
+ * If it returns true, the change is replicated, otherwise, it is not.
+ */
+static bool
+pgoutput_row_filter(Relation relation, HeapTuple oldtuple, HeapTuple newtuple, List *rowfilter)
+{
+	TupleDesc	tupdesc;
+	EState	   *estate;
+	ExprContext *ecxt;
+	MemoryContext oldcxt;
+	ListCell   *lc;
+	bool		result = true;
+
+	/* Bail out if there is no row filter */
+	if (list_length(rowfilter) == 0)
+		return true;
+
+	tupdesc = RelationGetDescr(relation);
+
+	estate = create_estate_for_relation(relation);
+
+	/* Prepare context per tuple */
+	ecxt = GetPerTupleExprContext(estate);
+	oldcxt = MemoryContextSwitchTo(estate->es_query_cxt);
+	ecxt->ecxt_scantuple = ExecInitExtraTupleSlot(estate, tupdesc, &TTSOpsHeapTuple);
+	MemoryContextSwitchTo(oldcxt);
+
+	ExecStoreHeapTuple(newtuple ? newtuple : oldtuple, ecxt->ecxt_scantuple, false);
+
+	/*
+	 * If the subscription has multiple publications and the same table has a
+	 * different row filter in these publications, all row filters must be
+	 * matched in order to replicate this change.
+	 */
+	foreach(lc, rowfilter)
+	{
+		Node	   *rfnode = (Node *) lfirst(lc);
+		Oid			exprtype;
+		Expr	   *expr;
+		ExprState  *exprstate;
+
+		/* Prepare expression for execution */
+		exprtype = exprType(rfnode);
+		expr = (Expr *) coerce_to_target_type(NULL, rfnode, exprtype, BOOLOID, -1, COERCION_ASSIGNMENT, COERCE_IMPLICIT_CAST, -1);
+
+		if (expr == NULL)
+			ereport(ERROR,
+					(errcode(ERRCODE_CANNOT_COERCE),
+					 errmsg("row filter returns type %s that cannot be coerced to the expected type %s",
+							format_type_be(exprtype),
+							format_type_be(BOOLOID)),
+					 errhint("You will need to rewrite the row filter.")));
+
+		exprstate = ExecPrepareExpr(expr, estate);
+
+		/* Evaluates row filter */
+		result = pgoutput_row_filter_exec_expr(exprstate, ecxt);
+
+		/* If the tuple does not match one of the row filters, bail out */
+		if (!result)
+			break;
+	}
+
+	ResetExprContext(ecxt);
+	FreeExecutorState(estate);
+
+	return result;
+}
+
 /*
  * Sends the decoded DML over wire.
  *
@@ -516,7 +622,7 @@ pgoutput_change(LogicalDecodingContext *ctx, ReorderBufferTXN *txn,
 	if (in_streaming)
 		xid = change->txn->xid;
 
-	relentry = get_rel_sync_entry(data, RelationGetRelid(relation));
+	relentry = get_rel_sync_entry(data, relation);
 
 	/* First check the table filter */
 	switch (change->action)
@@ -560,6 +666,10 @@ pgoutput_change(LogicalDecodingContext *ctx, ReorderBufferTXN *txn,
 						tuple = execute_attr_map_tuple(tuple, relentry->map);
 				}
 
+				/* Check row filter. */
+				if (!pgoutput_row_filter(relation, NULL, tuple, relentry->qual))
+					return;
+
 				OutputPluginPrepareWrite(ctx, true);
 				logicalrep_write_insert(ctx->out, xid, relation, tuple,
 										data->binary);
@@ -586,6 +696,10 @@ pgoutput_change(LogicalDecodingContext *ctx, ReorderBufferTXN *txn,
 					}
 				}
 
+				/* Check row filter. */
+				if (!pgoutput_row_filter(relation, oldtuple, newtuple, relentry->qual))
+					return;
+
 				OutputPluginPrepareWrite(ctx, true);
 				logicalrep_write_update(ctx->out, xid, relation, oldtuple,
 										newtuple, data->binary);
@@ -608,6 +722,10 @@ pgoutput_change(LogicalDecodingContext *ctx, ReorderBufferTXN *txn,
 						oldtuple = execute_attr_map_tuple(oldtuple, relentry->map);
 				}
 
+				/* Check row filter. */
+				if (!pgoutput_row_filter(relation, oldtuple, NULL, relentry->qual))
+					return;
+
 				OutputPluginPrepareWrite(ctx, true);
 				logicalrep_write_delete(ctx->out, xid, relation, oldtuple,
 										data->binary);
@@ -655,12 +773,11 @@ pgoutput_truncate(LogicalDecodingContext *ctx, ReorderBufferTXN *txn,
 	for (i = 0; i < nrelations; i++)
 	{
 		Relation	relation = relations[i];
-		Oid			relid = RelationGetRelid(relation);

-		relentry = get_rel_sync_entry(data, relid);
+		relentry = get_rel_sync_entry(data, relation);
 
 		if (!relentry->pubactions.pubtruncate)
 			continue;
@@ -670,10 +787,10 @@ pgoutput_truncate(LogicalDecodingContext *ctx, ReorderBufferTXN *txn,
 		 * root tables through it.
 		 */
 		if (relation->rd_rel->relispartition &&
-			relentry->publish_as_relid != relid)
+			relentry->publish_as_relid != relentry->relid)
 			continue;
 
-		relids[nrelids++] = relid;
+		relids[nrelids++] = relentry->relid;
 		maybe_send_schema(ctx, txn, change, relation, relentry);
 	}
 
@@ -941,16 +1058,21 @@ set_schema_sent_in_streamed_txn(RelationSyncEntry *entry, TransactionId xid)
  * when publishing.
  */
 static RelationSyncEntry *
-get_rel_sync_entry(PGOutputData *data, Oid relid)
+get_rel_sync_entry(PGOutputData *data, Relation rel)
 {
 	RelationSyncEntry *entry;
-	bool		am_partition = get_rel_relispartition(relid);
-	char		relkind = get_rel_relkind(relid);
+	Oid			relid;
+	bool		am_partition;
+	char		relkind;
 	bool		found;
 	MemoryContext oldctx;
 
 	Assert(RelationSyncCache != NULL);
 
+	relid = RelationGetRelid(rel);
+	am_partition = get_rel_relispartition(relid);
+	relkind = get_rel_relkind(relid);
+
 	/* Find cached relation info, creating if not found */
 	entry = (RelationSyncEntry *) hash_search(RelationSyncCache,
 											  (void *) &relid,
@@ -966,6 +1088,7 @@ get_rel_sync_entry(PGOutputData *data, Oid relid)
 		entry->replicate_valid = false;
 		entry->pubactions.pubinsert = entry->pubactions.pubupdate =
 			entry->pubactions.pubdelete = entry->pubactions.pubtruncate = false;
+		entry->qual = NIL;
 		entry->publish_as_relid = InvalidOid;
 	}
 
@@ -997,6 +1120,9 @@ get_rel_sync_entry(PGOutputData *data, Oid relid)
 		{
 			Publication *pub = lfirst(lc);
 			bool		publish = false;
+			HeapTuple	rftuple;
+			Datum		rfdatum;
+			bool		rfisnull;
 
 			if (pub->alltables)
 			{
@@ -1056,9 +1182,29 @@ get_rel_sync_entry(PGOutputData *data, Oid relid)
 				entry->pubactions.pubtruncate |= pub->pubactions.pubtruncate;
 			}
 
-			if (entry->pubactions.pubinsert && entry->pubactions.pubupdate &&
-				entry->pubactions.pubdelete && entry->pubactions.pubtruncate)
-				break;
+			/*
+			 * Cache row filter, if available. All publication-table mappings
+			 * must be checked. If it is a partition and pubviaroot is true,
+			 * use the row filter of the topmost partitioned table instead of
+			 * the row filter of its own partition.
+			 */
+			rftuple = SearchSysCache2(PUBLICATIONRELMAP, ObjectIdGetDatum(publish_as_relid), ObjectIdGetDatum(pub->oid));
+			if (HeapTupleIsValid(rftuple))
+			{
+				rfdatum = SysCacheGetAttr(PUBLICATIONRELMAP, rftuple, Anum_pg_publication_rel_prqual, &rfisnull);
+
+				if (!rfisnull)
+				{
+					Node	   *rfnode;
+
+					oldctx = MemoryContextSwitchTo(CacheMemoryContext);
+					rfnode = stringToNode(TextDatumGetCString(rfdatum));
+					entry->qual = lappend(entry->qual, rfnode);
+					MemoryContextSwitchTo(oldctx);
+				}
+
+				ReleaseSysCache(rftuple);
+			}
 		}
 
 		list_free(pubids);
@@ -1164,6 +1310,7 @@ rel_sync_cache_publication_cb(Datum arg, int cacheid, uint32 hashvalue)
 {
 	HASH_SEQ_STATUS status;
 	RelationSyncEntry *entry;
+	MemoryContext oldctx;
 
 	/*
 	 * We can get here if the plugin was used in SQL interface as the
@@ -1173,6 +1320,8 @@ rel_sync_cache_publication_cb(Datum arg, int cacheid, uint32 hashvalue)
 	if (RelationSyncCache == NULL)
 		return;
 
+	oldctx = MemoryContextSwitchTo(CacheMemoryContext);
+
 	/*
 	 * There is no way to find which entry in our cache the hash belongs to so
 	 * mark the whole cache as invalid.
@@ -1190,5 +1339,11 @@ rel_sync_cache_publication_cb(Datum arg, int cacheid, uint32 hashvalue)
 		entry->pubactions.pubupdate = false;
 		entry->pubactions.pubdelete = false;
 		entry->pubactions.pubtruncate = false;
+
+		if (list_length(entry->qual) > 0)
+			list_free_deep(entry->qual);
+		entry->qual = NIL;
 	}
+
+	MemoryContextSwitchTo(oldctx);
 }
diff --git a/src/include/catalog/pg_publication.h b/src/include/catalog/pg_publication.h
index 4127611f5a..0263baf72a 100644
--- a/src/include/catalog/pg_publication.h
+++ b/src/include/catalog/pg_publication.h
@@ -85,6 +85,13 @@ typedef struct Publication
 	PublicationActions pubactions;
 } Publication;
 
+	Oid			relid;
@@ -106,11 +113,12 @@ typedef enum PublicationPartOpt
 } PublicationPartOpt;
 
 extern List *GetPublicationRelations(Oid pubid, PublicationPartOpt pub_partopt);
+extern List *GetPublicationRelationQuals(Oid pubid);
 extern List *GetAllTablesPublications(void);
 extern List *GetAllTablesPublicationRelations(bool pubviaroot);

index c79b7fb487..d26673a111 100644
 
 /* ----------------
@@ -40,6 +44,8 @@ CATALOG(pg_publication_rel,6106,PublicationRelRelationId)
  */
 typedef FormData_pg_publication_rel *Form_pg_publication_rel;
 
+DECLARE_TOAST(pg_publication_rel, 8287, 8288);
+
 DECLARE_UNIQUE_INDEX_PKEY(pg_publication_rel_oid_index, 6112, on pg_publication_rel using btree(oid oid_ops));
 #define PublicationRelObjectIndexId 6112
 DECLARE_UNIQUE_INDEX(pg_publication_rel_prrelid_prpubid_index, 6113, on pg_publication_rel using btree(prrelid oid_ops, prpubid oid_ops));
diff --git a/src/include/nodes/nodes.h b/src/include/nodes/nodes.h
index 40ae489c23..cabda6f3c4 100644
--- a/src/include/nodes/nodes.h
+++ b/src/include/nodes/nodes.h
@@ -482,6 +482,7 @@ typedef enum NodeTag
 	T_PartitionRangeDatum,
 	T_PartitionCmd,
 	T_VacuumRelation,
+	T_PublicationTable,
 
 	/*
 	 * TAGS FOR REPLICATION GRAMMAR PARSE NODES (replnodes.h)
diff --git a/src/include/nodes/parsenodes.h b/src/include/nodes/parsenodes.h
index 236832a2ca..65f7a8884f 100644
--- a/src/include/nodes/parsenodes.h
+++ b/src/include/nodes/parsenodes.h
@@ -3543,12 +3543,19 @@ typedef struct AlterTSConfigurationStmt
 } AlterTSConfigurationStmt;

@@ -3561,7 +3568,7 @@ typedef struct AlterPublicationStmt
 	List	   *options;		/* List of DefElem nodes */
 
 	/* parameters used for ALTER PUBLICATION ... ADD/DROP TABLE */
-	List	   *tables;			/* List of tables to add/drop */
+	List	   *tables;			/* List of PublicationTable to add/drop */
 	bool		for_all_tables; /* Special publication for all tables in db */
 	DefElemAction tableAction;	/* What action to perform with the tables */
 } AlterPublicationStmt;
diff --git a/src/include/parser/parse_node.h b/src/include/parser/parse_node.h
index 176b9f37c1..edb80fbe5d 100644
--- a/src/include/parser/parse_node.h
+++ b/src/include/parser/parse_node.h
@@ -79,6 +79,7 @@ typedef enum ParseExprKind
 	EXPR_KIND_COPY_WHERE,		/* WHERE condition in COPY FROM */
 	EXPR_KIND_GENERATED_COLUMN, /* generation expression for a column */
 	EXPR_KIND_CYCLE_MARK,		/* cycle mark value */
+	EXPR_KIND_PUBLICATION_WHERE /* WHERE condition for a table in PUBLICATION */
 } ParseExprKind;

diff --git a/src/include/replication/logicalrelation.h b/src/include/replication/logicalrelation.h
index 3f0b3deefb..a59ad2c9c8 100644
--- a/src/include/replication/logicalrelation.h
+++ b/src/include/replication/logicalrelation.h
@@ -49,4 +49,6 @@ extern void logicalrep_rel_close(LogicalRepRelMapEntry *rel,
 extern void logicalrep_typmap_update(LogicalRepTyp *remotetyp);
 extern char *logicalrep_typmap_gettypname(Oid remoteid);
 
+extern EState *create_estate_for_relation(Relation rel);
+
 #endif							/* LOGICALRELATION_H */
diff --git a/src/test/regress/expected/publication.out b/src/test/regress/expected/publication.out
index 63d6ab7a4e..c8cf1b685e 100644
--- a/src/test/regress/expected/publication.out
+++ b/src/test/regress/expected/publication.out
@@ -156,6 +156,38 @@ Tables:
 
 DROP TABLE testpub_parted1;
 DROP PUBLICATION testpub_forparted, testpub_forparted1;
+CREATE TABLE testpub_rf_tbl1 (a integer, b text);
+CREATE TABLE testpub_rf_tbl2 (c text, d integer);
+CREATE TABLE testpub_rf_tbl3 (e integer);
+CREATE TABLE testpub_rf_tbl4 (g text);
+SET client_min_messages = 'ERROR';
+CREATE PUBLICATION testpub5 FOR TABLE testpub_rf_tbl1, testpub_rf_tbl2 WHERE (c <> 'test' AND d < 5);
+RESET client_min_messages;
+ALTER PUBLICATION testpub5 ADD TABLE testpub_rf_tbl3 WHERE (e > 1000 AND e < 2000);
+ALTER PUBLICATION testpub5 DROP TABLE testpub_rf_tbl2;
+-- remove testpub_rf_tbl1 and add testpub_rf_tbl3 again (another WHERE expression)
+ALTER PUBLICATION testpub5 SET TABLE testpub_rf_tbl3 WHERE (e > 300 AND e < 500);
+-- fail - functions disallowed
+ALTER PUBLICATION testpub5 ADD TABLE testpub_rf_tbl4 WHERE (length(g) < 6);
+ERROR:  functions are not allowed in publication WHERE expressions
+LINE 1: ...ICATION testpub5 ADD TABLE testpub_rf_tbl4 WHERE (length(g) ...
+                                                             ^
+-- fail - WHERE not allowed in DROP
+ALTER PUBLICATION testpub5 DROP TABLE testpub_rf_tbl3 WHERE (e < 27);
+ERROR:  cannot use a WHERE clause when removing table from publication "testpub5"
+\dRp+ testpub5
+                                    Publication testpub5
+          Owner           | All tables | Inserts | Updates | Deletes | Truncates | Via root 
+--------------------------+------------+---------+---------+---------+-----------+----------
+ regress_publication_user | f          | t       | t       | t       | t         | f
+Tables:
+    "public.testpub_rf_tbl3"  WHERE ((e > 300) AND (e < 500))
+
+DROP TABLE testpub_rf_tbl1;
+DROP TABLE testpub_rf_tbl2;
+DROP TABLE testpub_rf_tbl3;
+DROP TABLE testpub_rf_tbl4;
+DROP PUBLICATION testpub5;
 -- fail - view
 CREATE PUBLICATION testpub_fortbl FOR TABLE testpub_view;
 ERROR:  "testpub_view" is not a table
diff --git a/src/test/regress/sql/publication.sql b/src/test/regress/sql/publication.sql
index d844075368..35211c56f6 100644
--- a/src/test/regress/sql/publication.sql
+++ b/src/test/regress/sql/publication.sql
@@ -93,6 +93,29 @@ ALTER PUBLICATION testpub_forparted SET (publish_via_partition_root = true);
 DROP TABLE testpub_parted1;
 DROP PUBLICATION testpub_forparted, testpub_forparted1;
 
+CREATE TABLE testpub_rf_tbl1 (a integer, b text);
+CREATE TABLE testpub_rf_tbl2 (c text, d integer);
+CREATE TABLE testpub_rf_tbl3 (e integer);
+CREATE TABLE testpub_rf_tbl4 (g text);
+SET client_min_messages = 'ERROR';
+CREATE PUBLICATION testpub5 FOR TABLE testpub_rf_tbl1, testpub_rf_tbl2 WHERE (c <> 'test' AND d < 5);
+RESET client_min_messages;
+ALTER PUBLICATION testpub5 ADD TABLE testpub_rf_tbl3 WHERE (e > 1000 AND e < 2000);
+ALTER PUBLICATION testpub5 DROP TABLE testpub_rf_tbl2;
+-- remove testpub_rf_tbl1 and add testpub_rf_tbl3 again (another WHERE expression)
+ALTER PUBLICATION testpub5 SET TABLE testpub_rf_tbl3 WHERE (e > 300 AND e < 500);
+-- fail - functions disallowed
+ALTER PUBLICATION testpub5 ADD TABLE testpub_rf_tbl4 WHERE (length(g) < 6);
+-- fail - WHERE not allowed in DROP
+ALTER PUBLICATION testpub5 DROP TABLE testpub_rf_tbl3 WHERE (e < 27);
+\dRp+ testpub5
+
+DROP TABLE testpub_rf_tbl1;
+DROP TABLE testpub_rf_tbl2;
+DROP TABLE testpub_rf_tbl3;
+DROP TABLE testpub_rf_tbl4;
+DROP PUBLICATION testpub5;
+
 -- fail - view
 CREATE PUBLICATION testpub_fortbl FOR TABLE testpub_view;
 SET client_min_messages = 'ERROR';
diff --git a/src/test/subscription/t/020_row_filter.pl b/src/test/subscription/t/020_row_filter.pl
new file mode 100644
index 0000000000..b8c059d44b
--- /dev/null
+++ b/src/test/subscription/t/020_row_filter.pl
@@ -0,0 +1,190 @@
+# Test logical replication behavior with row filtering
+use strict;
+use warnings;
+use PostgresNode;
+use TestLib;
+use Test::More tests => 6;
+
+# create publisher node
+my $node_publisher = get_new_node('publisher');
+$node_publisher->init(allows_streaming => 'logical');
+$node_publisher->start;
+
+# create subscriber node
+my $node_subscriber = get_new_node('subscriber');
+$node_subscriber->init(allows_streaming => 'logical');
+$node_subscriber->start;
+
+# setup structure on publisher
+$node_publisher->safe_psql('postgres',
+	"CREATE TABLE tab_rowfilter_1 (a int primary key, b text)");
+$node_publisher->safe_psql('postgres',
+	"CREATE TABLE tab_rowfilter_2 (c int primary key)");
+$node_publisher->safe_psql('postgres',
+	"CREATE TABLE tab_rowfilter_3 (a int primary key, b boolean)");
+$node_publisher->safe_psql('postgres',
+	"CREATE TABLE tab_rowfilter_partitioned (a int primary key, b integer) PARTITION BY RANGE(a)"
+);
+$node_publisher->safe_psql('postgres',
+	"CREATE TABLE tab_rowfilter_less_10k (LIKE tab_rowfilter_partitioned)");
+$node_publisher->safe_psql('postgres',
+	"ALTER TABLE tab_rowfilter_partitioned ATTACH PARTITION tab_rowfilter_less_10k FOR VALUES FROM (MINVALUE) TO (10000)"
+);
+$node_publisher->safe_psql('postgres',
+	"CREATE TABLE tab_rowfilter_greater_10k (LIKE tab_rowfilter_partitioned)"
+);
+$node_publisher->safe_psql('postgres',
+	"ALTER TABLE tab_rowfilter_partitioned ATTACH PARTITION tab_rowfilter_greater_10k FOR VALUES FROM (10000) TO (MAXVALUE)"
+);
+
+# setup structure on subscriber
+$node_subscriber->safe_psql('postgres',
+	"CREATE TABLE tab_rowfilter_1 (a int primary key, b text)");
+$node_subscriber->safe_psql('postgres',
+	"CREATE TABLE tab_rowfilter_2 (c int primary key)");
+$node_subscriber->safe_psql('postgres',
+	"CREATE TABLE tab_rowfilter_3 (a int primary key, b boolean)");
+$node_subscriber->safe_psql('postgres',
+	"CREATE TABLE tab_rowfilter_partitioned (a int primary key, b integer) PARTITION BY RANGE(a)"
+);
+$node_subscriber->safe_psql('postgres',
+	"CREATE TABLE tab_rowfilter_less_10k (LIKE tab_rowfilter_partitioned)");
+$node_subscriber->safe_psql('postgres',
+	"ALTER TABLE tab_rowfilter_partitioned ATTACH PARTITION tab_rowfilter_less_10k FOR VALUES FROM (MINVALUE) TO (10000)"
+);
+$node_subscriber->safe_psql('postgres',
+	"CREATE TABLE tab_rowfilter_greater_10k (LIKE tab_rowfilter_partitioned)"
+);
+$node_subscriber->safe_psql('postgres',
+	"ALTER TABLE tab_rowfilter_partitioned ATTACH PARTITION tab_rowfilter_greater_10k FOR VALUES FROM (10000) TO (MAXVALUE)"
+);
+
+# setup logical replication
+$node_publisher->safe_psql('postgres',
+	"CREATE PUBLICATION tap_pub_1 FOR TABLE tab_rowfilter_1 WHERE (a > 1000 AND b <> 'filtered')"
+);
+
+$node_publisher->safe_psql('postgres',
+	"ALTER PUBLICATION tap_pub_1 ADD TABLE tab_rowfilter_2 WHERE (c % 7 = 0)"
+);
+
+$node_publisher->safe_psql('postgres',
+	"ALTER PUBLICATION tap_pub_1 SET TABLE tab_rowfilter_1 WHERE (a > 1000 AND b <> 'filtered'), tab_rowfilter_2 WHERE (c % 2 = 0), tab_rowfilter_3"
+);
+
+$node_publisher->safe_psql('postgres',
+	"CREATE PUBLICATION tap_pub_2 FOR TABLE tab_rowfilter_2 WHERE (c % 3 = 0)"
+);
+
+$node_publisher->safe_psql('postgres',
+	"CREATE PUBLICATION tap_pub_3 FOR TABLE tab_rowfilter_partitioned WHERE (a < 5000)"
+);
+$node_publisher->safe_psql('postgres',
+	"ALTER PUBLICATION tap_pub_3 ADD TABLE tab_rowfilter_less_10k WHERE (a < 6000)"
+);
+
+# test row filtering
+$node_publisher->safe_psql('postgres',
+	"INSERT INTO tab_rowfilter_1 (a, b) VALUES (1, 'not replicated')");
+$node_publisher->safe_psql('postgres',
+	"INSERT INTO tab_rowfilter_1 (a, b) VALUES (1500, 'filtered')");
+$node_publisher->safe_psql('postgres',
+	"INSERT INTO tab_rowfilter_1 (a, b) VALUES (1980, 'not filtered')");
+$node_publisher->safe_psql('postgres',
+	"INSERT INTO tab_rowfilter_1 (a, b) SELECT x, 'test ' || x FROM generate_series(990,1002) x"
+);
+$node_publisher->safe_psql('postgres',
+	"INSERT INTO tab_rowfilter_2 (c) SELECT generate_series(1, 20)");
+$node_publisher->safe_psql('postgres',
+	"INSERT INTO tab_rowfilter_3 (a, b) SELECT x, (x % 3 = 0) FROM generate_series(1, 10) x"
+);
+# use partition row filter:
+# - replicate (1, 100) because 1 < 6000 is true
+# - don't replicate (8000, 101) because 8000 < 6000 is false
+# - replicate (15000, 102) because partition tab_rowfilter_greater_10k doesn't have row filter
+$node_publisher->safe_psql('postgres',
+	"INSERT INTO tab_rowfilter_partitioned (a, b) VALUES(1, 100),(8000, 101),(15000, 102)"
+);
+# insert directly into partition
+# use partition row filter: replicate (2, 200) because 2 < 6000 is true
+$node_publisher->safe_psql('postgres',
+	"INSERT INTO tab_rowfilter_less_10k (a, b) VALUES(2, 200)");
+# use partition row filter: replicate (5500, 300) because 5500 < 6000 is true
+$node_publisher->safe_psql('postgres',
+	"INSERT INTO tab_rowfilter_partitioned (a, b) VALUES(5500, 300)");
+
+my $publisher_connstr = $node_publisher->connstr . ' dbname=postgres';
+my $appname           = 'tap_sub';
+$node_subscriber->safe_psql('postgres',
+	"CREATE SUBSCRIPTION tap_sub CONNECTION '$publisher_connstr application_name=$appname' PUBLICATION tap_pub_1, tap_pub_2, tap_pub_3"
+);
+
+$node_publisher->wait_for_catchup($appname);
+
+# wait for initial table sync to finish
+my $synced_query =
+  "SELECT count(1) = 0 FROM pg_subscription_rel WHERE srsubstate NOT IN ('r', 's');";
+$node_subscriber->poll_query_until('postgres', $synced_query)
+  or die "Timed out while waiting for subscriber to synchronize data";
+
+my $result =
+  $node_subscriber->safe_psql('postgres',
+	"SELECT a, b FROM tab_rowfilter_1 ORDER BY 1, 2");
+is( $result, qq(1001|test 1001
+1002|test 1002
+1980|not filtered), 'check filtered data was copied to subscriber');
+
+$result =
+  $node_subscriber->safe_psql('postgres',
+	"SELECT count(c), min(c), max(c) FROM tab_rowfilter_2");
+is($result, qq(3|6|18), 'check filtered data was copied to subscriber');
+
+$result =
+  $node_subscriber->safe_psql('postgres',
+	"SELECT count(a) FROM tab_rowfilter_3");
+is($result, qq(10), 'check filtered data was copied to subscriber');
+
+$result =
+  $node_subscriber->safe_psql('postgres',
+	"SELECT a, b FROM tab_rowfilter_less_10k ORDER BY 1, 2");
+is( $result, qq(1|100
+2|200
+5500|300), 'check filtered data was copied to subscriber');
+
+$result =
+  $node_subscriber->safe_psql('postgres',
+	"SELECT a, b FROM tab_rowfilter_greater_10k ORDER BY 1, 2");
+is($result, qq(15000|102), 'check filtered data was copied to subscriber');
+
+# publish using partitioned table
+$node_publisher->safe_psql('postgres',
+	"ALTER PUBLICATION tap_pub_3 SET (publish_via_partition_root = true)");
+$node_subscriber->safe_psql('postgres',
+	"TRUNCATE TABLE tab_rowfilter_partitioned");
+$node_subscriber->safe_psql('postgres',
+	"ALTER SUBSCRIPTION tap_sub REFRESH PUBLICATION WITH (copy_data = true)");
+# use partitioned table row filter: replicate, 4000 < 5000 is true
+$node_publisher->safe_psql('postgres',
+	"INSERT INTO tab_rowfilter_partitioned (a, b) VALUES(4000, 400)");
+# use partitioned table row filter: replicate, 4500 < 5000 is true
+$node_publisher->safe_psql('postgres',
+	"INSERT INTO tab_rowfilter_less_10k (a, b) VALUES(4500, 450)");
+# use partitioned table row filter: don't replicate, 5600 < 5000 is false
+$node_publisher->safe_psql('postgres',
+	"INSERT INTO tab_rowfilter_less_10k (a, b) VALUES(5600, 123)");
+# use partitioned table row filter: don't replicate, 16000 < 5000 is false
+$node_publisher->safe_psql('postgres',
+	"INSERT INTO tab_rowfilter_greater_10k (a, b) VALUES(16000, 1950)");
+
+$node_publisher->wait_for_catchup($appname);
+
+$result =
+  $node_subscriber->safe_psql('postgres',
+	"SELECT a, b FROM tab_rowfilter_partitioned ORDER BY 1, 2");
+is( $result, qq(1|100
+2|200
+4000|400
+4500|450), 'check publish_via_partition_root behavior');
+
+$node_subscriber->stop('fast');
+$node_publisher->stop('fast');
-- 
2.20.1

From 96f5c57bccb4bd9cae8546ecefcca454b628e927 Mon Sep 17 00:00:00 2001
From: Euler Taveira <[email-masked]>
Date: Mon, 18 Jan 2021 12:09:19 -0300
Subject: [PATCH 4/7] Print publication WHERE condition in psql

---

diff --git a/src/bin/psql/describe.c b/src/bin/psql/describe.c
index 20af5a92b4..8a7113ce15 100644
--- a/src/bin/psql/describe.c
+++ b/src/bin/psql/describe.c
@@ -6011,7 +6011,8 @@ describePublications(const char *pattern)
 		if (!puballtables)
 		{
 			printfPQExpBuffer(&buf,
-							  "SELECT n.nspname, c.relname\n"
+							  "SELECT n.nspname, c.relname,\n"
+							  "  pg_get_expr(pr.prqual, c.oid)\n"
 							  "FROM pg_catalog.pg_class c,\n"
 							  "     pg_catalog.pg_namespace n,\n"
 							  "     pg_catalog.pg_publication_rel pr\n"
@@ -6041,6 +6042,10 @@ describePublications(const char *pattern)
 								  PQgetvalue(tabres, j, 0),
 								  PQgetvalue(tabres, j, 1));
 
2.20.1

From 35b8cc1e0d9ca8e0e5431073cc10b80f5f16972e Mon Sep 17 00:00:00 2001
From: Euler Taveira <[email-masked]>
Date: Mon, 18 Jan 2021 14:29:59 -0300
Subject: [PATCH 5/7] Publication WHERE condition support for pg_dump

---
 src/bin/pg_dump/pg_dump.c | 14 ++++++++++++--
 src/bin/pg_dump/pg_dump.h |  1 +
 2 files changed, 13 insertions(+), 2 deletions(-)

diff --git a/src/bin/pg_dump/pg_dump.c b/src/bin/pg_dump/pg_dump.c
index 39da742e32..50388ae8ca 100644
--- a/src/bin/pg_dump/pg_dump.c
+++ b/src/bin/pg_dump/pg_dump.c
@@ -4080,6 +4080,7 @@ getPublicationTables(Archive *fout, TableInfo tblinfo[], int numTables)
 	int			i_oid;
 	int			i_prpubid;
 	int			i_prrelid;
+	int			i_prrelqual;
 	int			i,
 				j,
 				ntups;
@@ -4091,7 +4092,8 @@ getPublicationTables(Archive *fout, TableInfo tblinfo[], int numTables)
 
 	/* Collect all publication membership info. */
 	appendPQExpBufferStr(query,
-						 "SELECT tableoid, oid, prpubid, prrelid "
+						 "SELECT tableoid, oid, prpubid, prrelid, "
+						 "pg_catalog.pg_get_expr(prqual, prrelid) AS prrelqual "
 						 "FROM pg_catalog.pg_publication_rel");
 	res = ExecuteSqlQuery(fout, query->data, PGRES_TUPLES_OK);
 
@@ -4101,6 +4103,7 @@ getPublicationTables(Archive *fout, TableInfo tblinfo[], int numTables)
 	i_oid = PQfnumber(res, "oid");
 	i_prpubid = PQfnumber(res, "prpubid");
 	i_prrelid = PQfnumber(res, "prrelid");
+	i_prrelqual = PQfnumber(res, "prrelqual");
 
 	/* this allocation may be more than we need */
 	pubrinfo = pg_malloc(ntups * sizeof(PublicationRelInfo));
@@ -4141,6 +4144,10 @@ getPublicationTables(Archive *fout, TableInfo tblinfo[], int numTables)
 		pubrinfo[j].dobj.name = tbinfo->dobj.name;
 		pubrinfo[j].publication = pubinfo;
 		pubrinfo[j].pubtable = tbinfo;
+		if (PQgetisnull(res, i, i_prrelqual))
+			pubrinfo[j].pubrelqual = NULL;
+		else
+			pubrinfo[j].pubrelqual = pg_strdup(PQgetvalue(res, i, i_prrelqual));
 
 		/* Decide whether we want to dump it */
 		selectDumpablePublicationTable(&(pubrinfo[j].dobj), fout);
@@ -4173,8 +4180,11 @@ dumpPublicationTable(Archive *fout, PublicationRelInfo *pubrinfo)
 
 					  fmtId(pubinfo->dobj.name));
 
 	/*
 	 * There is no point in creating a drop query as the drop is done by table
diff --git a/src/bin/pg_dump/pg_dump.h b/src/bin/pg_dump/pg_dump.h
index 1290f9659b..7927039d1d 100644
--- a/src/bin/pg_dump/pg_dump.h
+++ b/src/bin/pg_dump/pg_dump.h
@@ -625,6 +625,7 @@ typedef struct _PublicationRelInfo
 	DumpableObject dobj;
 	PublicationInfo *publication;
 	TableInfo  *pubtable;
+	char	   *pubrelqual;
 } PublicationRelInfo;
 
 /*
-- 
2.20.1

From 7a855df32c456ead8f0634f122c569c82022ba88 Mon Sep 17 00:00:00 2001
From: Euler Taveira <[email-masked]>
Date: Fri, 29 Jan 2021 23:28:13 -0300
Subject: [PATCH 6/7] Debug messages

---
 src/backend/replication/pgoutput/pgoutput.c | 14 ++++++++++++++
 src/test/subscription/t/020_row_filter.pl   |  1 +
 2 files changed, 15 insertions(+)

diff --git a/src/backend/replication/pgoutput/pgoutput.c b/src/backend/replication/pgoutput/pgoutput.c
index dd6f3bda3a..17a728c0bf 100644
--- a/src/backend/replication/pgoutput/pgoutput.c
+++ b/src/backend/replication/pgoutput/pgoutput.c
@@ -518,6 +518,10 @@ pgoutput_row_filter_exec_expr(ExprState *state, ExprContext *econtext)
 
 	ret = ExecEvalExprSwitchContext(state, econtext, &isnull);
 
+	elog(DEBUG2, "pgoutput_row_filter_exec_expr: ret: %d ; isnull: %d",
+		 DatumGetBool(ret) ? 1 : 0,
+		 isnull ? 1 : 0);
+
 	if (isnull)
 		return false;
 
@@ -543,6 +547,8 @@ pgoutput_row_filter(Relation relation, HeapTuple oldtuple, HeapTuple newtuple, L
 	if (list_length(rowfilter) == 0)
 		return true;
 
+	elog(DEBUG1, "table %s has row filter", get_rel_name(relation->rd_id));
+
 	tupdesc = RelationGetDescr(relation);
 
 	estate = create_estate_for_relation(relation);
@@ -566,6 +572,7 @@ pgoutput_row_filter(Relation relation, HeapTuple oldtuple, HeapTuple newtuple, L
 		Oid			exprtype;
 		Expr	   *expr;
 		ExprState  *exprstate;
+		char	   *s = NULL;
 
 		/* Prepare expression for execution */
 		exprtype = exprType(rfnode);
@@ -585,6 +592,13 @@ pgoutput_row_filter(Relation relation, HeapTuple oldtuple, HeapTuple newtuple, L
 		result = pgoutput_row_filter_exec_expr(exprstate, ecxt);
 
 		/* If the tuple does not match one of the row filters, bail out */
+		s = TextDatumGetCString(DirectFunctionCall2(pg_get_expr, CStringGetTextDatum(nodeToString(rfnode)), ObjectIdGetDatum(relation->rd_id)));
+		if (result)
+			elog(DEBUG2, "pgoutput_row_filter: row filter \"%s\" matched", s);
+		else
+			elog(DEBUG2, "pgoutput_row_filter: row filter \"%s\" not matched", s);
+		pfree(s);
+
 		if (!result)
 			break;
 	}
diff --git a/src/test/subscription/t/020_row_filter.pl b/src/test/subscription/t/020_row_filter.pl
index b8c059d44b..ea1d7c30ae 100644
--- a/src/test/subscription/t/020_row_filter.pl
+++ b/src/test/subscription/t/020_row_filter.pl
@@ -8,6 +8,7 @@ use Test::More tests => 6;
 # create publisher node
 my $node_publisher = get_new_node('publisher');
 $node_publisher->init(allows_streaming => 'logical');
+$node_publisher->append_conf('postgresql.conf', 'log_min_messages = DEBUG2');
 $node_publisher->start;
 
 # create subscriber node
-- 
2.20.1

From a861145175c04c25fc7db662b55dd7a9cfddbccd Mon Sep 17 00:00:00 2001
From: Euler Taveira <[email-masked]>
Date: Sun, 31 Jan 2021 20:48:43 -0300
Subject: [PATCH 7/7] Measure row filter overhead

---
 src/backend/replication/pgoutput/pgoutput.c | 9 +++++++++
 1 file changed, 9 insertions(+)

diff --git a/src/backend/replication/pgoutput/pgoutput.c b/src/backend/replication/pgoutput/pgoutput.c
index 17a728c0bf..ebd45c1c84 100644
--- a/src/backend/replication/pgoutput/pgoutput.c
+++ b/src/backend/replication/pgoutput/pgoutput.c
@@ -542,6 +542,8 @@ pgoutput_row_filter(Relation relation, HeapTuple oldtuple, HeapTuple newtuple, L
 	MemoryContext oldcxt;
 	ListCell   *lc;
 	bool		result = true;
+	instr_time	start_time;
+	instr_time	end_time;
 
 	/* Bail out if there is no row filter */
 	if (list_length(rowfilter) == 0)
@@ -549,6 +551,8 @@ pgoutput_row_filter(Relation relation, HeapTuple oldtuple, HeapTuple newtuple, L
 
 	elog(DEBUG1, "table %s has row filter", get_rel_name(relation->rd_id));
 
+	INSTR_TIME_SET_CURRENT(start_time);
+
 	tupdesc = RelationGetDescr(relation);
 
 	estate = create_estate_for_relation(relation);
@@ -606,6 +610,11 @@ pgoutput_row_filter(Relation relation, HeapTuple oldtuple, HeapTuple newtuple, L
 	ResetExprContext(ecxt);
 	FreeExecutorState(estate);
 
+	INSTR_TIME_SET_CURRENT(end_time);
+	INSTR_TIME_SUBTRACT(end_time, start_time);
+
+	elog(DEBUG2, "row filter time: %0.3f us", INSTR_TIME_GET_DOUBLE(end_time) * 1e6);
+
 	return result;
 }
 
-- 
2.20.1

---

From: Michael Paquier <[email-masked]>
Date: Tue, 2 Feb 2021 14:02:53 +0900

> attaching another version to address the documentation issues.

I have bumped into this thread, and applied 0001.  My guess is that
one of the patches developped originally for logical replication
defined atttypmod in LogicalRepRelation, but has finished by not using
it.  Nice catch.
--
Michael

---

From: japin <[email-masked]>
Date: Tue, 02 Feb 2021 13:37:31 +0800

> evaluates to NULL, it returns false and the row won't be replicated.

Thanks for your clarification.

---

From: japin <[email-masked]>
Date: Tue, 02 Feb 2021 19:16:23 +0800

> defined atttypmod in LogicalRepRelation, but has finished by not using

Since the 0001 patch already be commited (4ad31bb2ef), we can remove it.

---

From: japin <[email-masked]>
Date: Tue, 02 Feb 2021 19:38:12 +0800

> Since the 0001 patch already be commited (4ad31bb2ef), we can remove it.

In 0003 patch, function GetPublicationRelationQuals() has been defined, but it
never used.  So why should we define it?

$ grep 'GetPublicationRelationQuals' -rn src/
src/include/catalog/pg_publication.h:116:extern List *GetPublicationRelationQuals(Oid pubid);
src/backend/catalog/pg_publication.c:347:GetPublicationRelationQuals(Oid pubid)

If we must keep it, here are some comments on it.

(1)
value_datum = heap_getattr(tup, Anum_pg_publication_rel_prqual, RelationGetDescr(pubrelsrel), &isnull);

It looks too long, we can split it into two lines.

(2)
Since qual_value only used in "if (!isnull)" branch, so we can narrow it's scope.

(3)
Should we free the memory for qual_value?

---

From: "Euler Taveira" <[email-masked]>
Date: Tue, 02 Feb 2021 09:34:34 -0300

> never used.  So why should we define it?
Thanks for taking a look again. It is an oversight. It was introduced in an
attempt to refactor ALTER PUBLICATION SET TABLE. In AlterPublicationTables, we
could possibly keep some publication-table mappings that does not change,
however, since commit 3696a600e2, it is required to find the qual for all
inheritors (see GetPublicationRelations). I explain this decision in the
following comment:

            /*
             * Remove all publication-table mappings.  We could possibly
             * remove (i) tables that are not found in the new table list and
             * (ii) tables that are being re-added with a different qual
             * expression. For (ii), simply updating the existing tuple is not
             * enough, because of qual expression dependencies.
             */

I will post a new patch set later.

--
Euler Taveira
EDB   https://www.enterprisedb.com/

---

From: Önder Kalacı <[email-masked]>
Date: Mon, 22 Feb 2021 11:45:11 +0100

Hi,

Thanks for working on this. I did some review and testing, please see my
comments below.

1)

   0008 is only for debug purposes (I'm
>    not sure some of these messages will be part of the final patch).

I think if you are planning to keep the debug patch, there seems to be an
area of improvement in the following code:

 		/* If the tuple does not match one of the row filters, bail out */
+		s = TextDatumGetCString(DirectFunctionCall2(pg_get_expr,
CStringGetTextDatum(nodeToString(rfnode)),
ObjectIdGetDatum(relation->rd_id)));
+		if (result)
+			elog(DEBUG2, "pgoutput_row_filter: row filter \"%s\" matched", s);
+		else
+			elog(DEBUG2, "pgoutput_row_filter: row filter \"%s\" not matched", s);
+		pfree(s);
+

We only need to calculate "s" if the debug level is DEBU2 or higher. So, we
could maybe do something like this:

if (log_min_messages <= DEBUG2 || client_min_messages <= DEBUG2)
{
   /* and the code block is moved to here */
}

2) I have done some tests with some different expressions that don't exist
on the regression tests, just to make sure that we don't have any edge
cases. All seems to work fine for the expressions
like: (column/1.0)::int::bool::text::bool, CASE WHEN column1> 4000 THEN
column2/ 100 > 1 ELSE false END, COALESCE((column/50000)::bool, false),
NULLIF((column/50000)::int::bool, false), column IS DISTINCT FROM 50040,
row(column, 2, 3) > row(2000, 2, 3), (column IS DISTINCT FROM), column IS
NULL, column IS NOT NULL, composite types

3) As another  edge case exploration, I tested with tables/types on
different schemas with escape chars on the schemas/custom types etc. All
looks good.

4) In terms of performance, I also separately verified that the overhead
seems pretty low with the final patch. I used the tests in
commands_to_perf_test.sql file which I shared earlier. The steps in the
file do not intend to measure the time precisely per tuple, but just to see
if there is any noticeable regression while moving lots of data. The
difference between (a) no filter (b) simple filter is between %1-%4, which
could even be considered in the noise level.

5) I guess we can by-pass the function limitation via operators. Do you see
anything problematic with that? I think that should be allowed as it helps
power users to create more complex replications if they need.

CREATE FUNCTION simple_f(int, int) RETURNS bool
>     AS $$ SELECT hashint4($1) > $2  $$
>     LANGUAGE SQL;
> CREATE OPERATOR =*>
> (
> PROCEDURE = simple_f,
> LEFTARG = int,
> RIGHTARG = int
> );
> CREATE PUBLICATION pub_1 FOR TABLE test WHERE (key =*> 1000);

5.1) It might be useful to have a regression test which has an user-defined
operator on the WHERE clause, and DROP without cascade is not allowed so
that we cover recordDependencyOnExpr() calls in the tests.

Thanks,
Onder KALACI
Software Engineer at Microsoft

Euler Taveira <[email-masked]>, 2 Şub 2021 Sal, 13:34 tarihinde şunu

---

From: "Euler Taveira" <[email-masked]>
Date: Mon, 22 Feb 2021 09:28:07 -0300

On Mon, Feb 22, 2021, at 7:45 AM, Önder Kalacı wrote:
> Thanks for working on this. I did some review and testing, please see my comments below.
I appreciate your review. I'm working on a new patch set and expect to post it soon.

> I think if you are planning to keep the debug patch, there seems to be an area of improvement in the following code:
I was not planning to include the debug part, however, it would probably help to  
debug some use cases. In the "row [not] matched" message, it should be DEBUG3  
for a final version because it is too noisy. Since you mentioned I will inspect
and include in the main patch those DEBUG messages that could possibly be      
useful for debug purposes.

> 
> 4) In terms of performance, I also separately verified that the overhead seems pretty low with the final patch. I used the tests in commands_to_perf_test.sql file which I shared earlier. The steps in the file do not intend to measure the time precisely per tuple, but just to see if there is any noticeable regression while moving lots of data. The difference between (a) no filter (b) simple filter is between %1-%4, which could even be considered in the noise level.
I'm concerned about it too, I'm currently experimenting alternatives to reduce this overhead.

> 5) I guess we can by-pass the function limitation via operators. Do you see anything problematic with that? I think that should be allowed as it helps power users to create more complex replications if they need.
Yes, you can. I have to check if this user-defined operator could possibly         
break the replication. I will make sure to include a test covering this case      
too.

---

From: "Euler Taveira" <[email-masked]>
Date: Fri, 26 Feb 2021 22:35:13 -0300

> I appreciate your review. I'm working on a new patch set and expect to post it soon.
I'm attaching a new patch set. This new version improves documentation and commit  
messages and incorporates a few debug messages. I did a couple of tests and    
didn't find issues.

Here are some numbers from my i7 using a simple expression (aid > 0) on table
pgbench_accounts.

$ awk '/row filter time/ {print $9}' postgresql.log | /tmp/stat.pl 99
mean:           33.00 us
stddev:         17.65 us
median:         28.83 us
min-max:        [3.48 .. 6404.84] us
percentile(99): 49.66 us
mode:           41.71 us

I don't expect 0005 and 0006 to be included. I attached them to help with some
tests.

--
Euler Taveira
EDB   https://www.enterprisedb.com/
From 596b278d4be77f67467aa1d61ce26e765b078197 Mon Sep 17 00:00:00 2001
From: Euler Taveira <[email-masked]>
Date: Mon, 18 Jan 2021 11:53:34 -0300
Subject: [PATCH 1/6] Rename a WHERE node

 
-ExclusionWhereClause:
+OptWhereClause:

From 9b1d46d5c146d591022c12c7acec8d61d4e0bfcc Mon Sep 17 00:00:00 2001
From: Euler Taveira <[email-masked]>
Date: Mon, 18 Jan 2021 12:07:51 -0300
Subject: [PATCH 2/6] Row filter for logical replication

This feature adds row filter for publication tables. When you define or modify
a publication you can optionally filter rows that does not satisfy a WHERE
condition. It allows you to partially replicate a database or set of tables.
The row filter is per table which means that you can define different row
filters for different tables. A new row filter can be added simply by
informing the WHERE clause after the table name. The WHERE expression must be
enclosed by parentheses.

The WHERE clause should probably contain only columns that are part of the
primary key or that are covered by REPLICA IDENTITY. Otherwise, and DELETEs
won't be replicated. DELETE uses the old row version (that is limited to
primary key or REPLICA IDENTITY) to evaluate the row filter. INSERT and UPDATE
use the new row version to evaluate the row filter, hence, you can use any
column. If the row filter evaluates to NULL, it returns false. For simplicity,
functions are not allowed; it could possibly be addressed in a future patch.

If you choose to do the initial table synchronization, only data that satisfies
the row filters is sent. If the subscription has several publications in which
a table has been published with different WHERE clauses, rows must satisfy all
expressions to be copied.

If your publication contains partitioned table, the parameter
publish_via_partition_root determines if it uses the partition row filter (if
the parameter is false -- default) or the partitioned table row filter.
---
 doc/src/sgml/catalogs.sgml                  |   8 +
 doc/src/sgml/ref/alter_publication.sgml     |  11 +-
 doc/src/sgml/ref/create_publication.sgml    |  38 +++-
 doc/src/sgml/ref/create_subscription.sgml   |   8 +-
 src/backend/catalog/pg_publication.c        |  52 ++++-
 src/backend/commands/publicationcmds.c      |  96 +++++----
 src/backend/parser/gram.y                   |  28 ++-
 src/backend/parser/parse_agg.c              |  10 +
 src/backend/parser/parse_expr.c             |  13 ++
 src/backend/parser/parse_func.c             |   3 +
 src/backend/replication/logical/tablesync.c |  93 ++++++++-
 src/backend/replication/logical/worker.c    |  14 +-
 src/backend/replication/pgoutput/pgoutput.c | 212 ++++++++++++++++++--
 src/include/catalog/pg_publication.h        |   9 +-
 src/include/catalog/pg_publication_rel.h    |   6 +
 src/include/nodes/nodes.h                   |   1 +
 src/include/nodes/parsenodes.h              |  11 +-
 src/include/parser/parse_node.h             |   1 +
 src/include/replication/logicalrelation.h   |   2 +
 src/test/regress/expected/publication.out   |  32 +++
 src/test/regress/sql/publication.sql        |  23 +++
 src/test/subscription/t/020_row_filter.pl   | 190 ++++++++++++++++++
 22 files changed, 776 insertions(+), 85 deletions(-)
 create mode 100644 src/test/subscription/t/020_row_filter.pl

index db29905e91..7466965ad3 100644
@@ -5495,6 +5495,14 @@ SCRAM-SHA-256$<replaceable>&lt;iteration count&gt;</replaceable>:<replaceable>&l
 
  <refsynopsisdiv>
 <synopsis>
index ff82fbca55..715c37f2bb 100644
 
      <para>
 
          <para>
@@ -182,6 +192,14 @@ CREATE PUBLICATION <replaceable class="parameter">name</replaceable>
    disallowed on those tables.
   </para>
 
+  <para>
+   The <literal>WHERE</literal> clause should probably contain only columns
+   that are part of the primary key or be covered  by <literal>REPLICA
+   IDENTITY</literal> otherwise, <command>DELETE</command> operations will not
+   be replicated. For <command>INSERT</command> and <command>UPDATE</command>
+   operations, any column can be used in the <literal>WHERE</literal> clause.
+  </para>
+
   <para>
    For an <command>INSERT ... ON CONFLICT</command> command, the publication will
    publish the operation that actually results from the command.  So depending
@@ -197,6 +215,11 @@ CREATE PUBLICATION <replaceable class="parameter">name</replaceable>
   <para>
    <acronym>DDL</acronym> operations are not published.
   </para>
+
+  <para>
+   The <literal>WHERE</literal> clause expression is executed with the role used
+   for the replication connection.
+  </para>
  </refsect1>
 
  <refsect1>
@@ -209,6 +232,13 @@ CREATE PUBLICATION mypublication FOR TABLE users, departments;
 </programlisting>
   </para>
 
+  <para>
index 84d2efcfd2..7258f7c9a6 100644

 
 
 	/* Form a tuple. */

 

 
 	return myself;
 }

+	/*

-			foreach(newlc, rels)
-			{

 		/* And drop them. */
 
 	/*
 
 		/*

 	foreach(lc, rels)
 	{

 /*
 
 	foreach(lc, rels)
 	{

 
 	foreach(lc, rels)
 	{

 

 publication_for_tables:
 
+publication_table_list:
 
 			break;

 			/*
index f869e159d6..8eceac241e 100644

 
 			/*
 
 			/*
 
 			/*
 			 * There is intentionally no default: case here, so that the
diff --git a/src/backend/replication/logical/tablesync.c b/src/backend/replication/logical/tablesync.c
index feb634e7ac..02ac567a7b 100644
--- a/src/backend/replication/logical/tablesync.c
+++ b/src/backend/replication/logical/tablesync.c
@@ -688,19 +688,25 @@ copy_read_data(void *outbuf, int minread, int maxread)
 
 /*
 
 	lrel->nspname = nspname;
 	lrel->relname = relname;
@@ -789,6 +795,51 @@ fetch_remote_table_info(char *nspname, char *relname,
 
 	lrel->natts = natt;
 
@@ -803,6 +854,7 @@ copy_table(Relation rel)
@@ -811,7 +863,7 @@ copy_table(Relation rel)

@@ -820,16 +872,23 @@ copy_table(Relation rel)
 
@@ -838,9 +897,31 @@ copy_table(Relation rel)
index 18d05286b6..cfb69aeaa6 100644

@@ -1168,7 +1168,7 @@ apply_handle_insert(StringInfo s)
 	}
 
@@ -1293,7 +1293,7 @@ apply_handle_update(StringInfo s)
 
@@ -1450,7 +1450,7 @@ apply_handle_delete(StringInfo s)
 
index 1b993fb032..ce6da8de19 100644
@@ -57,6 +67,10 @@ static void pgoutput_stream_abort(struct LogicalDecodingContext *ctx,
+static ExprState *pgoutput_row_filter_prepare_expr(Node *rfnode, EState *estate);
+static inline bool pgoutput_row_filter_exec_expr(ExprState *state, ExprContext *econtext);
 
 static bool publications_valid;
 static bool in_streaming;
@@ -98,6 +112,7 @@ typedef struct RelationSyncEntry
 
 	bool		replicate_valid;
 
 	/*
 	 * OID of the relation to publish changes as.  For a partition, this may
@@ -121,7 +136,7 @@ static HTAB *RelationSyncCache = NULL;
 
@@ -491,6 +506,130 @@ send_relation_and_attrs(Relation relation, TransactionId xid,
 
+static ExprState *
+pgoutput_row_filter_prepare_expr(Node *rfnode, EState *estate)
+{
+	ExprState  *exprstate;
+	Oid			exprtype;
+	Expr	   *expr;
+
+	/* Prepare expression for execution */
+	exprtype = exprType(rfnode);
+	expr = (Expr *) coerce_to_target_type(NULL, rfnode, exprtype, BOOLOID, -1, COERCION_ASSIGNMENT, COERCE_IMPLICIT_CAST, -1);
+
+	if (expr == NULL)
+		ereport(ERROR,
+				(errcode(ERRCODE_CANNOT_COERCE),
+				 errmsg("row filter returns type %s that cannot be coerced to the expected type %s",
+						format_type_be(exprtype),
+						format_type_be(BOOLOID)),
+				 errhint("You will need to rewrite the row filter.")));
+
+	exprstate = ExecPrepareExpr(expr, estate);
+
+	return exprstate;
+}
+
+/*
+ * Evaluates row filter.
+ *
+ * If the row filter evaluates to NULL, it is taken as false i.e. the change
+ * isn't replicated.
+ */
+static inline bool
+pgoutput_row_filter_exec_expr(ExprState *state, ExprContext *econtext)
+{
+	Datum		ret;
+	bool		isnull;
+
+	Assert(state != NULL);
+
+	ret = ExecEvalExprSwitchContext(state, econtext, &isnull);
+
+	elog(DEBUG3, "row filter evaluates to %s (isnull: %s)",
+		 DatumGetBool(ret) ? "true" : "false",
+		 isnull ? "true" : "false");
+
+	if (isnull)
+		return false;
+
+	return DatumGetBool(ret);
+}
+
+/*
+ * Change is checked against the row filter, if any.
+ *
+ * If it returns true, the change is replicated, otherwise, it is not.
+ */
+static bool
+pgoutput_row_filter(Relation relation, HeapTuple oldtuple, HeapTuple newtuple, List *rowfilter)
+{
+	TupleDesc	tupdesc;
+	EState	   *estate;
+	ExprContext *ecxt;
+	MemoryContext oldcxt;
+	ListCell   *lc;
+	bool		result = true;
+
+	/* Bail out if there is no row filter */
+	if (list_length(rowfilter) == 0)
+		return true;
+
+	elog(DEBUG3, "table \"%s.%s\" has row filter",
+		 get_namespace_name(get_rel_namespace(RelationGetRelid(relation))),
+		 get_rel_name(relation->rd_id));
+
+	tupdesc = RelationGetDescr(relation);
+
+	estate = create_estate_for_relation(relation);
+
+	/* Prepare context per tuple */
+	ecxt = GetPerTupleExprContext(estate);
+	oldcxt = MemoryContextSwitchTo(estate->es_query_cxt);
+	ecxt->ecxt_scantuple = ExecInitExtraTupleSlot(estate, tupdesc, &TTSOpsHeapTuple);
+	MemoryContextSwitchTo(oldcxt);
+
+	ExecStoreHeapTuple(newtuple ? newtuple : oldtuple, ecxt->ecxt_scantuple, false);
+
+	/*
+	 * If the subscription has multiple publications and the same table has a
+	 * different row filter in these publications, all row filters must be
+	 * matched in order to replicate this change.
+	 */
+	foreach(lc, rowfilter)
+	{
+		Node	   *rfnode = (Node *) lfirst(lc);
+		ExprState  *exprstate;
+
+		/* Prepare for expression execution */
+		exprstate = pgoutput_row_filter_prepare_expr(rfnode, estate);
+
+		/* Evaluates row filter */
+		result = pgoutput_row_filter_exec_expr(exprstate, ecxt);
+
+		if (message_level_is_interesting(DEBUG3))
+		{
+			char	   *s = NULL;
+
+			s = TextDatumGetCString(DirectFunctionCall2(pg_get_expr, CStringGetTextDatum(nodeToString(rfnode)), ObjectIdGetDatum(relation->rd_id)));
+			if (result)
+				elog(DEBUG3, "row filter \"%s\" matched", s);
+			else
+				elog(DEBUG3, "row filter \"%s\" not matched", s);
+			pfree(s);
+		}
+
+		/* If the tuple does not match one of the row filters, bail out */
+		if (!result)
+			break;
+	}
+
+	ResetExprContext(ecxt);
+	FreeExecutorState(estate);
+
+	return result;
+}
+
 /*
  * Sends the decoded DML over wire.
  *
@@ -518,7 +657,7 @@ pgoutput_change(LogicalDecodingContext *ctx, ReorderBufferTXN *txn,
 	if (in_streaming)
 		xid = change->txn->xid;

 	/* First check the table filter */
 	switch (change->action)
@@ -562,6 +701,10 @@ pgoutput_change(LogicalDecodingContext *ctx, ReorderBufferTXN *txn,
 						tuple = execute_attr_map_tuple(tuple, relentry->map);
 				}
 
@@ -588,6 +731,10 @@ pgoutput_change(LogicalDecodingContext *ctx, ReorderBufferTXN *txn,
 					}
 				}
 
@@ -610,6 +757,10 @@ pgoutput_change(LogicalDecodingContext *ctx, ReorderBufferTXN *txn,
 
@@ -657,12 +808,11 @@ pgoutput_truncate(LogicalDecodingContext *ctx, ReorderBufferTXN *txn,

 
@@ -672,10 +822,10 @@ pgoutput_truncate(LogicalDecodingContext *ctx, ReorderBufferTXN *txn,

@@ -944,16 +1094,21 @@ set_schema_sent_in_streamed_txn(RelationSyncEntry *entry, TransactionId xid)
  * when publishing.
  */

@@ -969,6 +1124,7 @@ get_rel_sync_entry(PGOutputData *data, Oid relid)
 
@@ -1000,6 +1156,9 @@ get_rel_sync_entry(PGOutputData *data, Oid relid)
 		{
 			Publication *pub = lfirst(lc);
 			bool		publish = false;
+			HeapTuple	rftuple;
+			Datum		rfdatum;
+			bool		rfisnull;
 
 			if (pub->alltables)
 			{
@@ -1059,9 +1218,29 @@ get_rel_sync_entry(PGOutputData *data, Oid relid)
 				entry->pubactions.pubtruncate |= pub->pubactions.pubtruncate;
 			}

 		list_free(pubids);
@@ -1167,6 +1346,7 @@ rel_sync_cache_publication_cb(Datum arg, int cacheid, uint32 hashvalue)
 {
 	HASH_SEQ_STATUS status;
 	RelationSyncEntry *entry;
+	MemoryContext oldctx;
 
 	/*
 	 * We can get here if the plugin was used in SQL interface as the
@@ -1176,6 +1356,8 @@ rel_sync_cache_publication_cb(Datum arg, int cacheid, uint32 hashvalue)
 	if (RelationSyncCache == NULL)
 		return;
 
@@ -1193,5 +1375,11 @@ rel_sync_cache_publication_cb(Datum arg, int cacheid, uint32 hashvalue)
index 1b31fee9e3..fa29468b18 100644
 
@@ -110,7 +117,7 @@ extern List *GetAllTablesPublications(void);

 extern Oid	get_publication_oid(const char *pubname, bool missing_ok);
diff --git a/src/include/catalog/pg_publication_rel.h b/src/include/catalog/pg_publication_rel.h
index aecf53b3b3..e2becb12eb 100644
--- a/src/include/catalog/pg_publication_rel.h
+++ b/src/include/catalog/pg_publication_rel.h
@@ -31,6 +31,10 @@ CATALOG(pg_publication_rel,6106,PublicationRelRelationId)
 	Oid			oid;			/* oid */
 	Oid			prpubid BKI_LOOKUP(pg_publication); /* Oid of the publication */
 	Oid			prrelid BKI_LOOKUP(pg_class);	/* Oid of the relation */
+
+#ifdef	CATALOG_VARLEN			/* variable-length fields start here */
+	pg_node_tree prqual;		/* qualifications */
+#endif
 } FormData_pg_publication_rel;
 
 /* ----------------

 	/*

 

 

From 44ccb34fb666efc6a1ee48c767185de1d13472a3 Mon Sep 17 00:00:00 2001
From: Euler Taveira <[email-masked]>
Date: Mon, 18 Jan 2021 12:09:19 -0300
Subject: [PATCH 3/6] Print publication WHERE condition in psql

---

From c10c8accbf31d4f9b2e874472fdd2a768af3fcc9 Mon Sep 17 00:00:00 2001
From: Euler Taveira <[email-masked]>
Date: Mon, 18 Jan 2021 14:29:59 -0300
Subject: [PATCH 4/6] Publication WHERE condition support for pg_dump

---

index eb988d7eb4..7b22d64b7c 100644

 
 
 		/* Decide whether we want to dump it */
 		selectDumpablePublicationTable(&(pubrinfo[j].dobj), fout);
@@ -4173,8 +4180,11 @@ dumpPublicationTable(Archive *fout, const PublicationRelInfo *pubrinfo)

 	/*
index 0a2213fb06..b3ee37f395 100644
 
 /*
-- 
2.20.1

From 3ddbe8db82efb198825e1889ff4cb0b9afb056b4 Mon Sep 17 00:00:00 2001
From: Euler Taveira <[email-masked]>
Date: Sun, 31 Jan 2021 20:48:43 -0300
Subject: [PATCH 5/6] Measure row filter overhead

---

diff --git a/src/backend/replication/pgoutput/pgoutput.c b/src/backend/replication/pgoutput/pgoutput.c
index ce6da8de19..e603a15ffd 100644
--- a/src/backend/replication/pgoutput/pgoutput.c
+++ b/src/backend/replication/pgoutput/pgoutput.c
@@ -570,6 +570,8 @@ pgoutput_row_filter(Relation relation, HeapTuple oldtuple, HeapTuple newtuple, L
 	MemoryContext oldcxt;
 	ListCell   *lc;
 	bool		result = true;
+	instr_time	start_time;
+	instr_time	end_time;
 
 	/* Bail out if there is no row filter */
 	if (list_length(rowfilter) == 0)
@@ -579,6 +581,8 @@ pgoutput_row_filter(Relation relation, HeapTuple oldtuple, HeapTuple newtuple, L
 		 get_namespace_name(get_rel_namespace(RelationGetRelid(relation))),
 		 get_rel_name(relation->rd_id));

 	estate = create_estate_for_relation(relation);
@@ -627,6 +631,11 @@ pgoutput_row_filter(Relation relation, HeapTuple oldtuple, HeapTuple newtuple, L
 	ResetExprContext(ecxt);
 	FreeExecutorState(estate);

-- 
2.20.1

From e97203ac7340ed24fffa8e779fff0f5a35d92ba0 Mon Sep 17 00:00:00 2001
From: Euler Taveira <[email-masked]>
Date: Fri, 26 Feb 2021 21:08:10 -0300
Subject: [PATCH 6/6] Debug messages

---
 src/backend/parser/parse_expr.c           | 6 ++++++
 src/test/subscription/t/020_row_filter.pl | 1 +
 2 files changed, 7 insertions(+)

diff --git a/src/backend/parser/parse_expr.c b/src/backend/parser/parse_expr.c
index 8eceac241e..52024d78d1 100644
--- a/src/backend/parser/parse_expr.c
+++ b/src/backend/parser/parse_expr.c
@@ -116,6 +116,12 @@ transformExprRecurse(ParseState *pstate, Node *expr)
 	if (expr == NULL)
 		return NULL;
 
+	/*
+	 * T_FuncCall: 349
+	 * EXPR_KIND_PUBLICATION_WHERE: 42
+	 */
+	elog(DEBUG3, "nodeTag(expr): %d ; pstate->p_expr_kind: %d", nodeTag(expr), pstate->p_expr_kind);
+
 	/* Guard against stack overflow due to overly complex expressions */
 	check_stack_depth();
 
diff --git a/src/test/subscription/t/020_row_filter.pl b/src/test/subscription/t/020_row_filter.pl
index b8c059d44b..41775c554a 100644
--- a/src/test/subscription/t/020_row_filter.pl
+++ b/src/test/subscription/t/020_row_filter.pl
@@ -8,6 +8,7 @@ use Test::More tests => 6;
 # create publisher node
 my $node_publisher = get_new_node('publisher');
 $node_publisher->init(allows_streaming => 'logical');
+$node_publisher->append_conf('postgresql.conf', 'log_min_messages = DEBUG3');
 $node_publisher->start;
 
 # create subscriber node
-- 
2.20.1

---

From: Rahila Syed <[email-masked]>
Date: Tue, 9 Mar 2021 20:35:21 +0530

Hi Euler,

Please find some comments below:

1. If the where clause contains non-replica identity columns, the delete
performed on a replicated row
 using DELETE from pub_tab where repl_ident_col = n;
is not being replicated, as logical replication does not have any info
whether the column has
to be filtered or not.
Shouldn't a warning be thrown in this case to notify the user that the
delete is not replicated.

2. Same for update, even if I update a row to match the quals on publisher,
it is still not being replicated to
the subscriber. (if the quals contain non-replica identity columns). I
think for UPDATE at least, the new value
of the non-replicate identity column is available which can be used to
filter and replicate the update.

3. 0001.patch,
Why is the name of the existing ExclusionWhereClause node being changed, if
the exact same definition is being used?

For 0002.patch,
4.   +

Is this needed, apart from the above, patch does not use or update lrel at
all in that function.

5.  PublicationRelationQual and PublicationTable have similar fields, can
PublicationTable
be used in place of PublicationRelationQual instead of defining a new
struct?

Thank you,
Rahila Syed

---

From: Rahila Syed <[email-masked]>
Date: Thu, 18 Mar 2021 16:21:55 +0530

Hi Euler,

Please find below some review comments,

1.
   +
   +     <row>
   +      <entry><structfield>prqual</structfield></entry>
   +      <entry><type>pg_node_tree</type></entry>
   +      <entry></entry>
   +      <entry>Expression tree (in <function>nodeToString()</function>
   +      representation) for the relation's qualifying condition</entry>
   +     </row>
I think the docs are being incorrectly updated to add a column to
pg_partitioned_table
instead of pg_publication_rel.

2.   +typedef struct PublicationRelationQual
 +{
+       Oid                     relid;
+       Relation        relation;
+       Node       *whereClause;
+} PublicationRelationQual;

Can this be given a more generic name like PublicationRelationInfo, so that
the same struct
can be used to store additional relation information in future, for ex.
column names, if column filtering is introduced.

3. Also, in the above structure, it seems that we can do with storing just
relid and derive relation information from it
using table_open when needed. Am I missing something?

4.  Currently in logical replication, I noticed that an UPDATE is being
applied on the subscriber even if the column values
 are unchanged. Can row-filtering feature be used to change it such that,
when all the OLD.columns = NEW.columns, filter out
the row from being sent to the subscriber. I understand this would need
REPLICA IDENTITY FULL to work, but would be an
improvement from the existing state.

On subscriber:

postgres=# select xmin, * from tab_rowfilter_1;
 xmin | a |      b
------+---+-------------
  555 | 1 | unfiltered
(1 row)

On publisher:
postgres=# ALTER TABLE tab_rowfilter_1 REPLICA IDENTITY FULL;
ALTER TABLE
postgres=# update tab_rowfilter_1 SET b = 'unfiltered' where a = 1;
UPDATE 1

On Subscriber:  The xmin has changed indicating the update from the
publisher was applied
even though nothing changed.

postgres=# select xmin, * from tab_rowfilter_1;
 xmin | a |      b
------+---+-------------
  556 | 1 | unfiltered
(1 row)

5. Currently, any existing rows that were not replicated, when updated to
match the publication quals
using UPDATE tab SET pub_qual_column = 'not_filtered' where a = 1; won't be
applied, as row
does not exist on the subscriber.  It would be good if ALTER SUBSCRIBER
REFRESH PUBLICATION
would help fetch such existing rows from publishers that match the qual
now(either because the row changed
or the qual changed)

Thank you,
Rahila Syed

On Tue, Mar 9, 2021 at 8:35 PM Rahila Syed <[email-masked]> wrote:

> 2. Same for update, even if I update a row to match the quals on
> publisher, it is still not being replicated to
> Why is the name of the existing ExclusionWhereClause node being changed,
> if the exact same definition is being used?
>
> For 0002.patch,
> 4.   +

---

From: "Euler Taveira" <[email-masked]>
Date: Sun, 21 Mar 2021 22:58:14 -0300

Thanks for your review.

> 1. If the where clause contains non-replica identity columns, the delete performed on a replicated row
> is not being replicated, as logical replication does not have any info whether the column has 
> to be filtered or not. 
> Shouldn't a warning be thrown in this case to notify the user that the delete is not replicated.
Isn't documentation enough? If you add a WARNING, it should be printed per row,
hence, a huge DELETE will flood the client with WARNING messages by default. If
you are thinking about LOG messages, it is a different story. However, we
should limit those messages to one per transaction. Even if we add such an aid,
it would impose a performance penalty while checking the DELETE is not
replicating because the row filter contains a column that is not part of the PK
or REPLICA IDENTITY. If I were to add any message, it would be to warn at the
creation time (CREATE PUBLICATION or ALTER PUBLICATION ... [ADD|SET] TABLE).

> 2. Same for update, even if I update a row to match the quals on publisher, it is still not being replicated to 
> the subscriber. (if the quals contain non-replica identity columns). I think for UPDATE at least, the new value
> of the non-replicate identity column is available which can be used to filter and replicate the update.
Indeed, the row filter for UPDATE uses the new tuple. Maybe your non-replica
identity column contains NULL that evaluates the expression to false.

> 3. 0001.patch, 
> Why is the name of the existing ExclusionWhereClause node being changed, if the exact same definition is being used?
Because this node ExclusionWhereClause is used for exclusion constraint. This
patch renames the node to made it clear it is a generic node that could be used
for other filtering features in the future.

> Is this needed, apart from the above, patch does not use or update lrel at all in that function.
Good catch. It is a leftover from a previous patch. It will be fixed in the
next patch set.

> 5.  PublicationRelationQual and PublicationTable have similar fields, can PublicationTable
> be used in place of PublicationRelationQual instead of defining a new struct?
I don't think it is a good idea to have additional fields in a parse node. The
DDL commands use Relation (PublicationTableQual) and parse code uses RangeVar
(PublicationTable). publicationcmds.c uses Relation everywhere so I decided to
create a new struct to store Relation and qual as a list item. It also minimizes the places
you have to modify.

---

From: "Euler Taveira" <[email-masked]>
Date: Sun, 21 Mar 2021 23:15:10 -0300

On Thu, Mar 18, 2021, at 7:51 AM, Rahila Syed wrote:
> 1. 
> I think the docs are being incorrectly updated to add a column to pg_partitioned_table
Good catch.

> Can this be given a more generic name like PublicationRelationInfo, so that the same struct 
> can be used to store additional relation information in future, for ex. column names, if column filtering is introduced.
Good idea. I rename it and it'll be in this next patch set.

> 3. Also, in the above structure, it seems that we can do with storing just relid and derive relation information from it
We need the Relation. See OpenTableList(). The way this code is organized, it
opens all publication tables and append each Relation to a list. This list is
used in PublicationAddTables() to update the catalog. I tried to minimize the
number of refactors while introducing this feature. We could probably revise
this code in the future (someone said in a previous discussion that it is weird
to open relations in one source code file -- publicationcmds.c -- and use it
into another one -- pg_publication.c).

> 4.  Currently in logical replication, I noticed that an UPDATE is being applied on the subscriber even if the column values
>  are unchanged. Can row-filtering feature be used to change it such that, when all the OLD.columns = NEW.columns, filter out 
> the row from being sent to the subscriber. I understand this would need REPLICA IDENTITY FULL to work, but would be an
This is how Postgres works.

postgres=# create table foo (a integer, b integer);
CREATE TABLE
postgres=# insert into foo values(1, 100);
INSERT 0 1
postgres=# select ctid, xmin, xmax, a, b from foo;
ctid  |  xmin  | xmax | a |  b 
-------+--------+------+---+-----
(0,1) | 488920 |    0 | 1 | 100
(1 row)

postgres=# update foo set b = 101 where a = 1;
UPDATE 1
postgres=# select ctid, xmin, xmax, a, b from foo;
ctid  |  xmin  | xmax | a |  b 
-------+--------+------+---+-----
(0,2) | 488921 |    0 | 1 | 101
(1 row)

postgres=# update foo set b = 101 where a = 1;
UPDATE 1
postgres=# select ctid, xmin, xmax, a, b from foo;
ctid  |  xmin  | xmax | a |  b 
-------+--------+------+---+-----
(0,3) | 488922 |    0 | 1 | 101
(1 row)

You could probably abuse this feature and skip some UPDATEs when old tuple is
identical to new tuple. The question is: why would someone issue the same
command multiple times? A broken application? I would say: don't do it. Besides
that, this feature could impose an overhead into a code path that already
consume substantial CPU time. I've seen some tables with RIF and dozens of
columns that would certainly contribute to increase the replication lag.

> 5. Currently, any existing rows that were not replicated, when updated to match the publication quals
> using UPDATE tab SET pub_qual_column = 'not_filtered' where a = 1; won't be applied, as row 
> does not exist on the subscriber.  It would be good if ALTER SUBSCRIBER REFRESH PUBLICATION
> would help fetch such existing rows from publishers that match the qual now(either because the row changed
> or the qual changed)
I see. This should be addressed by a resynchronize feature. Such option is
useful when you have to change the row filter. It should certainly be implement
as an ALTER SUBSCRIPTION subcommand.

I attached a new patch set that addresses:

* fix documentation;
* rename PublicationRelationQual to PublicationRelationInfo;
* remove the memset that was leftover from a previous patch set;
* add new tests to improve coverage (INSERT/UPDATE/DELETE to exercise the row
  filter code).

--
Euler Taveira
EDB   https://www.enterprisedb.com/
From a6d893be0091bc8cd8569ac380e6f628263d31c0 Mon Sep 17 00:00:00 2001
From: Euler Taveira <[email-masked]>
Date: Mon, 18 Jan 2021 11:53:34 -0300
Subject: [PATCH v12 1/5] Rename a WHERE node

diff --git a/src/backend/parser/gram.y b/src/backend/parser/gram.y
index bc43641ffe..22bbb27041 100644
--- a/src/backend/parser/gram.y
+++ b/src/backend/parser/gram.y
@@ -495,7 +495,7 @@ static Node *makeRecursiveViewSelect(char *relname, List *aliases, Node *query);
 %type <node>	def_arg columnElem where_clause where_or_current_clause
 				a_expr b_expr c_expr AexprConst indirection_el opt_slice_bound
 				columnref in_expr having_clause func_table xmltable array_expr
-				ExclusionWhereClause operator_def_arg
+				OptWhereClause operator_def_arg
 %type <list>	rowsfrom_item rowsfrom_list opt_col_def_list
 %type <boolean> opt_ordinality
 %type <list>	ExclusionConstraintList ExclusionConstraintElem
@@ -3837,7 +3837,7 @@ ConstraintElem:
 					$$ = (Node *)n;
 				}
 			| EXCLUDE access_method_clause '(' ExclusionConstraintList ')'
-				opt_c_include opt_definition OptConsTableSpace  ExclusionWhereClause
+				opt_c_include opt_definition OptConsTableSpace  OptWhereClause
 				ConstraintAttributeSpec
 				{
 					Constraint *n = makeNode(Constraint);
@@ -3939,7 +3939,7 @@ ExclusionConstraintElem: index_elem WITH any_operator
 			}
 		;
 
-ExclusionWhereClause:
+OptWhereClause:

From 2b50e8ef4ce7362426f6fd305fc3ab95a6d7e448 Mon Sep 17 00:00:00 2001
From: Euler Taveira <[email-masked]>
Date: Mon, 18 Jan 2021 12:07:51 -0300
Subject: [PATCH v12 2/5] Row filter for logical replication

If your publication contains partitioned table, the parameter
publish_via_partition_root determines if it uses the partition row filter (if
the parameter is false -- default) or the partitioned table row filter.
---
 doc/src/sgml/catalogs.sgml                  |   8 +
 doc/src/sgml/ref/alter_publication.sgml     |  11 +-
 doc/src/sgml/ref/create_publication.sgml    |  38 +++-
 doc/src/sgml/ref/create_subscription.sgml   |   8 +-
 src/backend/catalog/pg_publication.c        |  52 ++++-
 src/backend/commands/publicationcmds.c      |  98 +++++----
 src/backend/parser/gram.y                   |  28 ++-
 src/backend/parser/parse_agg.c              |  10 +
 src/backend/parser/parse_expr.c             |  13 ++
 src/backend/parser/parse_func.c             |   3 +
 src/backend/replication/logical/tablesync.c |  91 +++++++-
 src/backend/replication/logical/worker.c    |  14 +-
 src/backend/replication/pgoutput/pgoutput.c | 212 +++++++++++++++++--
 src/include/catalog/pg_publication.h        |   9 +-
 src/include/catalog/pg_publication_rel.h    |   6 +
 src/include/nodes/nodes.h                   |   1 +
 src/include/nodes/parsenodes.h              |  11 +-
 src/include/parser/parse_node.h             |   1 +
 src/include/replication/logicalrelation.h   |   2 +
 src/test/regress/expected/publication.out   |  32 +++
 src/test/regress/sql/publication.sql        |  23 ++
 src/test/subscription/t/020_row_filter.pl   | 221 ++++++++++++++++++++
 22 files changed, 805 insertions(+), 87 deletions(-)
 create mode 100644 src/test/subscription/t/020_row_filter.pl

diff --git a/doc/src/sgml/catalogs.sgml b/doc/src/sgml/catalogs.sgml
index 68d1960698..83c8d33186 100644
--- a/doc/src/sgml/catalogs.sgml
+++ b/doc/src/sgml/catalogs.sgml
@@ -6203,6 +6203,14 @@ SCRAM-SHA-256$<replaceable>&lt;iteration count&gt;</replaceable>:<replaceable>&l
        Reference to relation
       </para></entry>
      </row>
+
+     <row>
+      <entry role="catalog_table_entry"><para role="column_definition">
+      <structfield>prqual</structfield> <type>pg_node_tree</type>
+      </para>
+      <para>Expression tree (in <function>nodeToString()</function>
+      representation) for the relation's qualifying condition</para></entry>
+     </row>
     </tbody>
    </tgroup>
   </table>
diff --git a/doc/src/sgml/ref/alter_publication.sgml b/doc/src/sgml/ref/alter_publication.sgml
index faa114b2c6..ca091aae33 100644
--- a/doc/src/sgml/ref/alter_publication.sgml
+++ b/doc/src/sgml/ref/alter_publication.sgml
@@ -21,8 +21,8 @@ PostgreSQL documentation
 
  <refsynopsisdiv>
 <synopsis>
 
      <para>
 
          <para>
 
+  <para>
 
  <refsect1>
 
+  <para>
index 84d2efcfd2..abc3fb6411 100644
+publication_add_relation(Oid pubid, PublicationRelationInfo *targetrel,

 
 
 	/* Form a tuple. */

 

 
 	return myself;
 }
index 95c253c8e0..e352c66d9c 100644

+	/*
 
 	if (stmt->tableAction == DEFELEM_ADD)
@@ -385,31 +407,24 @@ AlterPublicationTables(AlterPublicationStmt *stmt, Relation rel,
 		List	   *delrels = NIL;
 		ListCell   *oldlc;
 
-		/* Calculate which relations to drop. */
+		/*
+		 * Remove all publication-table mappings.  We could possibly remove (i)
+		 * tables that are not found in the new table list and (ii) tables that
+		 * are being re-added with a different qual expression. For (ii),
+		 * simply updating the existing tuple is not enough, because of qual
+		 * expression dependencies.
+		 */
 		foreach(oldlc, oldrelids)
 		{
 			Oid			oldrelid = lfirst_oid(oldlc);
-			ListCell   *newlc;
-			bool		found = false;
+			PublicationRelationInfo *oldrel;
 
-			foreach(newlc, rels)
-			{
+			oldrel = palloc(sizeof(PublicationRelationInfo));
 
 		/* And drop them. */
@@ -509,13 +524,15 @@ OpenTableList(List *tables)
 	List	   *relids = NIL;
 	List	   *rels = NIL;
 	ListCell   *lc;
+	PublicationRelationInfo *pri;
 
 	/*
 	 * Open, share-lock, and check all the explicitly-specified relations
 	 */
 	foreach(lc, tables)
 	{
-		RangeVar   *rv = castNode(RangeVar, lfirst(lc));
+		PublicationTable *t = lfirst(lc);
+		RangeVar   *rv = castNode(RangeVar, t->relation);
 		bool		recurse = rv->inh;
 		Relation	rel;
 		Oid			myrelid;
@@ -538,8 +555,11 @@ OpenTableList(List *tables)
 			table_close(rel, ShareUpdateExclusiveLock);
 			continue;
 		}
-
-		rels = lappend(rels, rel);
+		pri = palloc(sizeof(PublicationRelationInfo));
+		pri->relid = myrelid;
+		pri->relation = rel;
+		pri->whereClause = t->whereClause;
+		rels = lappend(rels, pri);
 		relids = lappend_oid(relids, myrelid);
 
 		/*
@@ -572,7 +592,12 @@ OpenTableList(List *tables)
 
 				/* find_all_inheritors already got lock */
 				rel = table_open(childrelid, NoLock);
-				rels = lappend(rels, rel);
+				pri = palloc(sizeof(PublicationRelationInfo));
+				pri->relid = childrelid;
+				pri->relation = rel;
+				/* child inherits WHERE clause from parent */
+				pri->whereClause = t->whereClause;
+				rels = lappend(rels, pri);
 				relids = lappend_oid(relids, childrelid);
 			}
 		}
@@ -593,10 +618,12 @@ CloseTableList(List *rels)
 
 	foreach(lc, rels)
 	{
-		Relation	rel = (Relation) lfirst(lc);
+		PublicationRelationInfo *pri = (PublicationRelationInfo *) lfirst(lc);
 
-		table_close(rel, NoLock);
+		table_close(pri->relation, NoLock);
 	}
+
+	list_free_deep(rels);
 }
 
 /*
@@ -612,15 +639,15 @@ PublicationAddTables(Oid pubid, List *rels, bool if_not_exists,
 
 	foreach(lc, rels)
 	{
-		Relation	rel = (Relation) lfirst(lc);
+		PublicationRelationInfo *pri = (PublicationRelationInfo *) lfirst(lc);
 		ObjectAddress obj;
 
 		/* Must be owner of the table or superuser. */
-		if (!pg_class_ownercheck(RelationGetRelid(rel), GetUserId()))
-			aclcheck_error(ACLCHECK_NOT_OWNER, get_relkind_objtype(rel->rd_rel->relkind),
-						   RelationGetRelationName(rel));
+		if (!pg_class_ownercheck(pri->relid, GetUserId()))
+			aclcheck_error(ACLCHECK_NOT_OWNER, get_relkind_objtype(pri->relation->rd_rel->relkind),
+						   RelationGetRelationName(pri->relation));
 
-		obj = publication_add_relation(pubid, rel, if_not_exists);
+		obj = publication_add_relation(pubid, pri, if_not_exists);
 		if (stmt)
 		{
 			EventTriggerCollectSimpleCommand(obj, InvalidObjectAddress,
@@ -644,11 +671,10 @@ PublicationDropTables(Oid pubid, List *rels, bool missing_ok)
 
 	foreach(lc, rels)
 	{
-		Relation	rel = (Relation) lfirst(lc);
-		Oid			relid = RelationGetRelid(rel);
+		PublicationRelationInfo *pri = (PublicationRelationInfo *) lfirst(lc);
 
 		prid = GetSysCacheOid2(PUBLICATIONRELMAP, Anum_pg_publication_rel_oid,
-							   ObjectIdGetDatum(relid),
+							   ObjectIdGetDatum(pri->relid),
 							   ObjectIdGetDatum(pubid));
 		if (!OidIsValid(prid))
 		{
@@ -658,7 +684,7 @@ PublicationDropTables(Oid pubid, List *rels, bool missing_ok)
 			ereport(ERROR,
 					(errcode(ERRCODE_UNDEFINED_OBJECT),
 					 errmsg("relation \"%s\" is not part of the publication",
-							RelationGetRelationName(rel))));
+							RelationGetRelationName(pri->relation))));
 		}
 
index 22bbb27041..2a931efb41 100644
@@ -425,13 +425,13 @@ static Node *makeRecursiveViewSelect(char *relname, List *aliases, Node *query);
 
 %type <groupclause> group_clause
 
 %type <list>	opt_fdw_options fdw_options
 %type <defelt>	fdw_option
@@ -3837,7 +3837,7 @@ ConstraintElem:
 					$$ = (Node *)n;
 				}
 			| EXCLUDE access_method_clause '(' ExclusionConstraintList ')'
-				opt_c_include opt_definition OptConsTableSpace  OptWhereClause
+				opt_c_include opt_definition OptConsTableSpace OptWhereClause
 				ConstraintAttributeSpec
 				{
 					Constraint *n = makeNode(Constraint);
@@ -9530,7 +9530,7 @@ opt_publication_for_tables:
 		;
 
 publication_for_tables:
-			FOR TABLE relation_expr_list
+			FOR TABLE publication_table_list
 				{
 					$$ = (Node *) $3;
 				}
@@ -9561,7 +9561,7 @@ AlterPublicationStmt:
 					n->options = $5;
 					$$ = (Node *)n;
 				}
-			| ALTER PUBLICATION name ADD_P TABLE relation_expr_list
+			| ALTER PUBLICATION name ADD_P TABLE publication_table_list
 				{
 					AlterPublicationStmt *n = makeNode(AlterPublicationStmt);
 					n->pubname = $3;
@@ -9569,7 +9569,7 @@ AlterPublicationStmt:
 					n->tableAction = DEFELEM_ADD;
 					$$ = (Node *)n;
 				}
-			| ALTER PUBLICATION name SET TABLE relation_expr_list
+			| ALTER PUBLICATION name SET TABLE publication_table_list
 				{
 					AlterPublicationStmt *n = makeNode(AlterPublicationStmt);
 					n->pubname = $3;
@@ -9577,7 +9577,7 @@ AlterPublicationStmt:
 					n->tableAction = DEFELEM_SET;
 					$$ = (Node *)n;
 				}
-			| ALTER PUBLICATION name DROP TABLE relation_expr_list
+			| ALTER PUBLICATION name DROP TABLE publication_table_list
 				{
 					AlterPublicationStmt *n = makeNode(AlterPublicationStmt);
 					n->pubname = $3;
@@ -9587,6 +9587,20 @@ AlterPublicationStmt:
 				}
 		;
 
+publication_table_list:
index 7c3e01aa22..544ff55a63 100644
 
 			break;

 			/*

 
 			/*
 
 			/*
 
 			/*
 			 * There is intentionally no default: case here, so that the
diff --git a/src/backend/replication/logical/tablesync.c b/src/backend/replication/logical/tablesync.c
index 6ed31812ab..d47e04f35d 100644
--- a/src/backend/replication/logical/tablesync.c
+++ b/src/backend/replication/logical/tablesync.c
@@ -688,19 +688,23 @@ copy_read_data(void *outbuf, int minread, int maxread)
 
 /*
 
 	lrel->nspname = nspname;
 	lrel->relname = relname;
@@ -789,6 +793,51 @@ fetch_remote_table_info(char *nspname, char *relname,
 
 	lrel->natts = natt;
 
@@ -803,6 +852,7 @@ copy_table(Relation rel)
@@ -811,7 +861,7 @@ copy_table(Relation rel)

@@ -820,16 +870,23 @@ copy_table(Relation rel)
 
@@ -838,9 +895,31 @@ copy_table(Relation rel)
index 21d304a64c..004d0d46c3 100644

 

 
 
 	bool		replicate_valid;
 
 	/*

+static ExprState *

 

 

 

 

 			if (pub->alltables)
 			{

 		list_free(pubids);
 
 	/*
 
index 1b31fee9e3..9a60211259 100644
 
+typedef struct PublicationRelationInfo
+{
+	Oid			relid;
+	Relation	relation;
+	Node	   *whereClause;
+} PublicationRelationInfo;
+
 extern Publication *GetPublication(Oid pubid);
 extern Publication *GetPublicationByName(const char *pubname, bool missing_ok);
 extern List *GetRelationPublications(Oid relid);
@@ -110,7 +117,7 @@ extern List *GetAllTablesPublications(void);
 extern List *GetAllTablesPublicationRelations(bool pubviaroot);
 
 extern bool is_publishable_relation(Relation rel);
-extern ObjectAddress publication_add_relation(Oid pubid, Relation targetrel,
+extern ObjectAddress publication_add_relation(Oid pubid, PublicationRelationInfo *targetrel,
 											  bool if_not_exists);

 /* ----------------
 
+DECLARE_TOAST(pg_publication_rel, 8287, 8288);
+
 DECLARE_UNIQUE_INDEX_PKEY(pg_publication_rel_oid_index, 6112, on pg_publication_rel using btree(oid oid_ops));
 #define PublicationRelObjectIndexId 6112
 DECLARE_UNIQUE_INDEX(pg_publication_rel_prrelid_prpubid_index, 6113, on pg_publication_rel using btree(prrelid oid_ops, prpubid oid_ops));
diff --git a/src/include/nodes/nodes.h b/src/include/nodes/nodes.h
index e22df890ef..1c9f4fe765 100644
--- a/src/include/nodes/nodes.h
+++ b/src/include/nodes/nodes.h
@@ -485,6 +485,7 @@ typedef enum NodeTag
 	T_PartitionRangeDatum,
 	T_PartitionCmd,
 	T_VacuumRelation,
+	T_PublicationTable,
 
 	/*
 	 * TAGS FOR REPLICATION GRAMMAR PARSE NODES (replnodes.h)
diff --git a/src/include/nodes/parsenodes.h b/src/include/nodes/parsenodes.h
index 68425eb2c0..7570c3bc19 100644
--- a/src/include/nodes/parsenodes.h
+++ b/src/include/nodes/parsenodes.h
@@ -3556,12 +3556,19 @@ typedef struct AlterTSConfigurationStmt
 } AlterTSConfigurationStmt;

@@ -3574,7 +3581,7 @@ typedef struct AlterPublicationStmt
 	List	   *options;		/* List of DefElem nodes */

 

index 0000000000..35a41741d3
@@ -0,0 +1,221 @@
+use Test::More tests => 7;
+is($result, qq(1|100
+# test row filter (INSERT, UPDATE, DELETE)
+	"INSERT INTO tab_rowfilter_1 (a, b) VALUES (800, 'test 800')");
+	"INSERT INTO tab_rowfilter_1 (a, b) VALUES (1600, 'test 1600')");
+	"INSERT INTO tab_rowfilter_1 (a, b) VALUES (1601, 'test 1601')");
+	"INSERT INTO tab_rowfilter_1 (a, b) VALUES (1700, 'test 1700')");
+# UPDATE is not replicated ; row filter evaluates to false when b = NULL
+	"UPDATE tab_rowfilter_1 SET b = NULL WHERE a = 1600");
+	"UPDATE tab_rowfilter_1 SET b = 'test 1601 updated' WHERE a = 1601");
+# DELETE is not replicated ; b is not part of the PK or replica identity and
+# old tuple contains b = NULL, hence, row filter evaluates to false
+	"DELETE FROM tab_rowfilter_1 WHERE a = 1700");
+is($result, qq(1001|test 1001
+1600|test 1600
+1601|test 1601 updated
+1700|test 1700

From c0c1b69b4924d64ce5df4bc7c8a229cc73a8b6c5 Mon Sep 17 00:00:00 2001
From: Euler Taveira <[email-masked]>
Date: Mon, 18 Jan 2021 12:09:19 -0300
Subject: [PATCH v12 3/5] Print publication WHERE condition in psql

---

diff --git a/src/bin/psql/describe.c b/src/bin/psql/describe.c
index eeac0efc4f..983ba512f7 100644
--- a/src/bin/psql/describe.c
+++ b/src/bin/psql/describe.c
@@ -6038,7 +6038,8 @@ describePublications(const char *pattern)
 		if (!puballtables)
 		{
 			printfPQExpBuffer(&buf,
-							  "SELECT n.nspname, c.relname\n"
+							  "SELECT n.nspname, c.relname,\n"
+							  "  pg_get_expr(pr.prqual, c.oid)\n"
 							  "FROM pg_catalog.pg_class c,\n"
 							  "     pg_catalog.pg_namespace n,\n"
 							  "     pg_catalog.pg_publication_rel pr\n"
@@ -6068,6 +6069,10 @@ describePublications(const char *pattern)
 								  PQgetvalue(tabres, j, 0),
 								  PQgetvalue(tabres, j, 1));

From 8c11843fa47126d37121fc16a52ffb82d9101443 Mon Sep 17 00:00:00 2001
From: Euler Taveira <[email-masked]>
Date: Mon, 18 Jan 2021 14:29:59 -0300
Subject: [PATCH v12 4/5] Publication WHERE condition support for pg_dump

---

diff --git a/src/bin/pg_dump/pg_dump.c b/src/bin/pg_dump/pg_dump.c
index da6cc054b0..72d87f21c8 100644
--- a/src/bin/pg_dump/pg_dump.c
+++ b/src/bin/pg_dump/pg_dump.c
@@ -4130,6 +4130,7 @@ getPublicationTables(Archive *fout, TableInfo tblinfo[], int numTables)
 	int			i_oid;
 	int			i_prpubid;
 	int			i_prrelid;
+	int			i_prrelqual;
 	int			i,
 				j,
 				ntups;
@@ -4141,7 +4142,8 @@ getPublicationTables(Archive *fout, TableInfo tblinfo[], int numTables)

@@ -4151,6 +4153,7 @@ getPublicationTables(Archive *fout, TableInfo tblinfo[], int numTables)
 
@@ -4191,6 +4194,10 @@ getPublicationTables(Archive *fout, TableInfo tblinfo[], int numTables)
 
 		/* Decide whether we want to dump it */
 		selectDumpablePublicationTable(&(pubrinfo[j].dobj), fout);
@@ -4223,8 +4230,11 @@ dumpPublicationTable(Archive *fout, const PublicationRelInfo *pubrinfo)

 	/*
 	 * There is no point in creating a drop query as the drop is done by table
diff --git a/src/bin/pg_dump/pg_dump.h b/src/bin/pg_dump/pg_dump.h
index 5340843081..155fc2ebb5 100644
--- a/src/bin/pg_dump/pg_dump.h
+++ b/src/bin/pg_dump/pg_dump.h
@@ -626,6 +626,7 @@ typedef struct _PublicationRelInfo
 	DumpableObject dobj;
 	PublicationInfo *publication;
 	TableInfo  *pubtable;
+	char	   *pubrelqual;
 } PublicationRelInfo;
 
 /*
-- 
2.20.1

From 2a1d560036b777c7823805052c094704f51b4324 Mon Sep 17 00:00:00 2001
From: Euler Taveira <[email-masked]>
Date: Sun, 31 Jan 2021 20:48:43 -0300
Subject: [PATCH v12 5/5] Measure row filter overhead

---

 

 
-- 
2.20.1

---

From: Peter Eisentraut <[email-masked]>
Date: Thu, 25 Mar 2021 12:15:56 +0100

> * add new tests to improve coverage (INSERT/UPDATE/DELETE to exercise 

I have committed the 0001 patch.

Attached are a few fixup patches that I recommend you integrate into 
your patch set.  They address backward compatibility with PG13, and a 
few more stylistic issues.

I suggest you combine your 0002, 0003, and 0004 patches into one.  They 
can't be used separately, and for example the psql changes in patch 0003 
already appear as regression test output changes in 0002, so this 
arrangement isn't useful.  (0005 can be kept separately, since it's 
mostly for debugging right now.)
From c29459505db88c86dd7aa61019fa406202c30b0a Mon Sep 17 00:00:00 2001
From: Peter Eisentraut <[email-masked]>
Date: Thu, 25 Mar 2021 11:57:48 +0100
Subject: [PATCH 1/6] fixup! Row filter for logical replication

Remove unused header files, clean up whitespace.
---
 src/backend/catalog/pg_publication.c        | 2 --
 src/backend/replication/pgoutput/pgoutput.c | 4 ----
 2 files changed, 6 deletions(-)

diff --git a/src/backend/catalog/pg_publication.c b/src/backend/catalog/pg_publication.c
index f31ae28de2..78f5780fb7 100644
--- a/src/backend/catalog/pg_publication.c
+++ b/src/backend/catalog/pg_publication.c
@@ -33,11 +33,9 @@
 #include "catalog/pg_type.h"
 #include "funcapi.h"
 #include "miscadmin.h"
-
 #include "parser/parse_clause.h"
 #include "parser/parse_collate.h"
 #include "parser/parse_relation.h"
-
 #include "utils/array.h"
 #include "utils/builtins.h"
 #include "utils/catcache.h"
diff --git a/src/backend/replication/pgoutput/pgoutput.c b/src/backend/replication/pgoutput/pgoutput.c
index ce6da8de19..6151f34925 100644
--- a/src/backend/replication/pgoutput/pgoutput.c
+++ b/src/backend/replication/pgoutput/pgoutput.c
@@ -16,14 +16,10 @@
 #include "catalog/partition.h"
 #include "catalog/pg_publication.h"
 #include "catalog/pg_publication_rel.h"
-#include "catalog/pg_type.h"
 #include "commands/defrem.h"
 #include "executor/executor.h"
 #include "fmgr.h"
-#include "nodes/execnodes.h"
 #include "nodes/nodeFuncs.h"
-#include "optimizer/planner.h"
-#include "optimizer/optimizer.h"
 #include "parser/parse_coerce.h"
 #include "replication/logical.h"
 #include "replication/logicalproto.h"
-- 
2.30.2

From 91c20ecb45a8627a9252ed589fe6b85927be8b45 Mon Sep 17 00:00:00 2001
From: Peter Eisentraut <[email-masked]>
Date: Thu, 25 Mar 2021 11:59:13 +0100
Subject: [PATCH 2/6] fixup! Row filter for logical replication

Allow replication from older PostgreSQL versions without prqual.
---
 src/backend/replication/logical/tablesync.c | 70 +++++++++++----------
 1 file changed, 37 insertions(+), 33 deletions(-)

diff --git a/src/backend/replication/logical/tablesync.c b/src/backend/replication/logical/tablesync.c
index 40d84dadb5..246510c82e 100644
--- a/src/backend/replication/logical/tablesync.c
+++ b/src/backend/replication/logical/tablesync.c
@@ -796,49 +796,53 @@ fetch_remote_table_info(char *nspname, char *relname,
 	walrcv_clear_result(res);
 
 	/* Get relation qual */
-	resetStringInfo(&cmd);
-	appendStringInfo(&cmd,
-					 "SELECT pg_get_expr(prqual, prrelid) "
-					 "  FROM pg_publication p "
-					 "  INNER JOIN pg_publication_rel pr "
-					 "       ON (p.oid = pr.prpubid) "
-					 " WHERE pr.prrelid = %u "
-					 "   AND p.pubname IN (", lrel->remoteid);
-
-	first = true;
-	foreach(lc, MySubscription->publications)
+	if (walrcv_server_version(wrconn) >= 140000)
 	{
-		char	   *pubname = strVal(lfirst(lc));
+		resetStringInfo(&cmd);
+		appendStringInfo(&cmd,
+						 "SELECT pg_get_expr(prqual, prrelid) "
+						 "  FROM pg_publication p "
+						 "  INNER JOIN pg_publication_rel pr "
+						 "       ON (p.oid = pr.prpubid) "
+						 " WHERE pr.prrelid = %u "
+						 "   AND p.pubname IN (", lrel->remoteid);
+
+		first = true;
+		foreach(lc, MySubscription->publications)
+		{
+			char	   *pubname = strVal(lfirst(lc));
 
-		if (first)
-			first = false;
-		else
-			appendStringInfoString(&cmd, ", ");
+			if (first)
+				first = false;
+			else
+				appendStringInfoString(&cmd, ", ");
 
-		appendStringInfoString(&cmd, quote_literal_cstr(pubname));
-	}
-	appendStringInfoChar(&cmd, ')');
+			appendStringInfoString(&cmd, quote_literal_cstr(pubname));
+		}
+		appendStringInfoChar(&cmd, ')');
 
-	res = walrcv_exec(wrconn, cmd.data, 1, qualRow);
+		res = walrcv_exec(wrconn, cmd.data, 1, qualRow);
 
-	if (res->status != WALRCV_OK_TUPLES)
-		ereport(ERROR,
-				(errmsg("could not fetch relation qualifications for table \"%s.%s\" from publisher: %s",
-						nspname, relname, res->err)));
+		if (res->status != WALRCV_OK_TUPLES)
+			ereport(ERROR,
+					(errmsg("could not fetch relation qualifications for table \"%s.%s\" from publisher: %s",
+							nspname, relname, res->err)));
 
-	slot = MakeSingleTupleTableSlot(res->tupledesc, &TTSOpsMinimalTuple);
-	while (tuplestore_gettupleslot(res->tuplestore, true, false, slot))
-	{
-		Datum		rf = slot_getattr(slot, 1, &isnull);
+		slot = MakeSingleTupleTableSlot(res->tupledesc, &TTSOpsMinimalTuple);
+		while (tuplestore_gettupleslot(res->tuplestore, true, false, slot))
+		{
+			Datum		rf = slot_getattr(slot, 1, &isnull);
 
-		if (!isnull)
-			*qual = lappend(*qual, makeString(TextDatumGetCString(rf)));
+			if (!isnull)
+				*qual = lappend(*qual, makeString(TextDatumGetCString(rf)));
 
-		ExecClearTuple(slot);
+			ExecClearTuple(slot);
+		}
+		ExecDropSingleTupleTableSlot(slot);
+
+		walrcv_clear_result(res);
 	}
-	ExecDropSingleTupleTableSlot(slot);
 
-	walrcv_clear_result(res);
 	pfree(cmd.data);
 }
 
-- 
2.30.2

From d40b463def29cdded18579d6e584cb9f5f4764d6 Mon Sep 17 00:00:00 2001
From: Peter Eisentraut <[email-masked]>
Date: Thu, 25 Mar 2021 12:00:35 +0100
Subject: [PATCH 3/6] fixup! Row filter for logical replication

Use more idiomatic style for checking for empty or nonempty lists.
---
 src/backend/replication/logical/tablesync.c | 4 ++--
 src/backend/replication/pgoutput/pgoutput.c | 4 ++--
 2 files changed, 4 insertions(+), 4 deletions(-)

diff --git a/src/backend/replication/logical/tablesync.c b/src/backend/replication/logical/tablesync.c
index 246510c82e..c3d6847b35 100644
--- a/src/backend/replication/logical/tablesync.c
+++ b/src/backend/replication/logical/tablesync.c
@@ -881,7 +881,7 @@ copy_table(Relation rel)
 	initStringInfo(&cmd);
 
 	/* Regular table with no row filter */
-	if (lrel.relkind == RELKIND_RELATION && list_length(qual) == 0)
+	if (lrel.relkind == RELKIND_RELATION && qual == NIL)
 		appendStringInfo(&cmd, "COPY %s TO STDOUT",
 						 quote_qualified_identifier(lrel.nspname, lrel.relname));
 	else
@@ -902,7 +902,7 @@ copy_table(Relation rel)
 		appendStringInfo(&cmd, " FROM %s",
 						 quote_qualified_identifier(lrel.nspname, lrel.relname));
 		/* list of AND'ed filters */
-		if (list_length(qual) > 0)
+		if (qual != NIL)
 		{
 			ListCell   *lc;
 			bool		first = true;
diff --git a/src/backend/replication/pgoutput/pgoutput.c b/src/backend/replication/pgoutput/pgoutput.c
index 6151f34925..593a8c96c8 100644
--- a/src/backend/replication/pgoutput/pgoutput.c
+++ b/src/backend/replication/pgoutput/pgoutput.c
@@ -568,7 +568,7 @@ pgoutput_row_filter(Relation relation, HeapTuple oldtuple, HeapTuple newtuple, L
 	bool		result = true;
 
 	/* Bail out if there is no row filter */
-	if (list_length(rowfilter) == 0)
+	if (rowfilter == NIL)
 		return true;
 
 	elog(DEBUG3, "table \"%s.%s\" has row filter",
@@ -1372,7 +1372,7 @@ rel_sync_cache_publication_cb(Datum arg, int cacheid, uint32 hashvalue)
 		entry->pubactions.pubdelete = false;
 		entry->pubactions.pubtruncate = false;
 
-		if (list_length(entry->qual) > 0)
+		if (entry->qual != NIL)
 			list_free_deep(entry->qual);
 		entry->qual = NIL;
 	}
-- 
2.30.2

From b82e7b05bb3fd73bede39f480c31d657fe410e41 Mon Sep 17 00:00:00 2001
From: Peter Eisentraut <[email-masked]>
Date: Thu, 25 Mar 2021 12:02:48 +0100
Subject: [PATCH 4/6] fixup! Row filter for logical replication

elog arguments are not evaluated unless the message level is
interesting, so the previous workaround is unnecessary.
---
 src/backend/replication/pgoutput/pgoutput.c | 16 +++++-----------
 1 file changed, 5 insertions(+), 11 deletions(-)

diff --git a/src/backend/replication/pgoutput/pgoutput.c b/src/backend/replication/pgoutput/pgoutput.c
index 593a8c96c8..00ad8f001f 100644
--- a/src/backend/replication/pgoutput/pgoutput.c
+++ b/src/backend/replication/pgoutput/pgoutput.c
@@ -603,17 +603,11 @@ pgoutput_row_filter(Relation relation, HeapTuple oldtuple, HeapTuple newtuple, L
 		/* Evaluates row filter */
 		result = pgoutput_row_filter_exec_expr(exprstate, ecxt);
 
-		if (message_level_is_interesting(DEBUG3))
-		{
-			char	   *s = NULL;
-
-			s = TextDatumGetCString(DirectFunctionCall2(pg_get_expr, CStringGetTextDatum(nodeToString(rfnode)), ObjectIdGetDatum(relation->rd_id)));
-			if (result)
-				elog(DEBUG3, "row filter \"%s\" matched", s);
-			else
-				elog(DEBUG3, "row filter \"%s\" not matched", s);
-			pfree(s);
-		}
+		elog(DEBUG3, "row filter \"%s\" %smatched",
+			 TextDatumGetCString(DirectFunctionCall2(pg_get_expr,
+													 CStringGetTextDatum(nodeToString(rfnode)),
+													 ObjectIdGetDatum(relation->rd_id))),
+			 result ? "" : " not");
 
2.30.2

From af012ad7d7592f88e353642fd715a3825ea2ec72 Mon Sep 17 00:00:00 2001
From: Peter Eisentraut <[email-masked]>
Date: Thu, 25 Mar 2021 12:07:11 +0100
Subject: [PATCH 5/6] fixup! Publication WHERE condition support for pg_dump

Make pg_dump backward compatible.

Also add necessary parentheses around expression.  pg_get_expr will
supply the parentheses in many cases, but it won't for things like
"WHERE TRUE".
---
 src/bin/pg_dump/pg_dump.c | 16 +++++++++++-----
 1 file changed, 11 insertions(+), 5 deletions(-)

diff --git a/src/bin/pg_dump/pg_dump.c b/src/bin/pg_dump/pg_dump.c
index 72d87f21c8..af57a50f27 100644
--- a/src/bin/pg_dump/pg_dump.c
+++ b/src/bin/pg_dump/pg_dump.c
@@ -4141,10 +4141,16 @@ getPublicationTables(Archive *fout, TableInfo tblinfo[], int numTables)
 	query = createPQExpBuffer();
 
 	/* Collect all publication membership info. */
-	appendPQExpBufferStr(query,
-						 "SELECT tableoid, oid, prpubid, prrelid, "
-						 "pg_catalog.pg_get_expr(prqual, prrelid) AS prrelqual "
-						 "FROM pg_catalog.pg_publication_rel");
+	if (fout->remoteVersion >= 140000)
+		appendPQExpBufferStr(query,
+							 "SELECT tableoid, oid, prpubid, prrelid, "
+							 "pg_catalog.pg_get_expr(prqual, prrelid) AS prrelqual "
+							 "FROM pg_catalog.pg_publication_rel");
+	else
+		appendPQExpBufferStr(query,
+							 "SELECT tableoid, oid, prpubid, prrelid, "
+							 "NULL AS prrelqual "
+							 "FROM pg_catalog.pg_publication_rel");
 	res = ExecuteSqlQuery(fout, query->data, PGRES_TUPLES_OK);
 
 	ntups = PQntuples(res);
@@ -4233,7 +4239,7 @@ dumpPublicationTable(Archive *fout, const PublicationRelInfo *pubrinfo)
 	appendPQExpBuffer(query, " %s",
 					  fmtQualifiedDumpable(tbinfo));
 	if (pubrinfo->pubrelqual)
-		appendPQExpBuffer(query, " WHERE %s", pubrinfo->pubrelqual);
+		appendPQExpBuffer(query, " WHERE (%s)", pubrinfo->pubrelqual);
 	appendPQExpBufferStr(query, ";\n");
 
 	/*
-- 
2.30.2

From 326c489012f020f8ef38e1187aea72b332d1b9a4 Mon Sep 17 00:00:00 2001
From: Peter Eisentraut <[email-masked]>
Date: Thu, 25 Mar 2021 12:08:51 +0100
Subject: [PATCH 6/6] fixup! Print publication WHERE condition in psql

Make psql backward compatible.  Also add necessary parentheses.
---
 src/bin/psql/describe.c                   | 11 +++++++----
 src/test/regress/expected/publication.out |  2 +-
 2 files changed, 8 insertions(+), 5 deletions(-)

diff --git a/src/bin/psql/describe.c b/src/bin/psql/describe.c
index 983ba512f7..6dce968414 100644
--- a/src/bin/psql/describe.c
+++ b/src/bin/psql/describe.c
@@ -6038,9 +6038,12 @@ describePublications(const char *pattern)
 		if (!puballtables)
 		{
 			printfPQExpBuffer(&buf,
-							  "SELECT n.nspname, c.relname,\n"
-							  "  pg_get_expr(pr.prqual, c.oid)\n"
-							  "FROM pg_catalog.pg_class c,\n"
+							  "SELECT n.nspname, c.relname");
+			if (pset.sversion >= 140000)
+				appendPQExpBuffer(&buf,
+								  ", pg_get_expr(pr.prqual, c.oid)");
+			appendPQExpBuffer(&buf,
+							  "\nFROM pg_catalog.pg_class c,\n"
 							  "     pg_catalog.pg_namespace n,\n"
 							  "     pg_catalog.pg_publication_rel pr\n"
 							  "WHERE c.relnamespace = n.oid\n"
@@ -6070,7 +6073,7 @@ describePublications(const char *pattern)
 								  PQgetvalue(tabres, j, 1));
 
 				if (!PQgetisnull(tabres, j, 2))
-					appendPQExpBuffer(&buf, "  WHERE %s",
+					appendPQExpBuffer(&buf, " WHERE (%s)",
 									  PQgetvalue(tabres, j, 2));
 
 				printTableAddFooter(&cont, buf.data);
diff --git a/src/test/regress/expected/publication.out b/src/test/regress/expected/publication.out
index c8cf1b685e..da49a2e215 100644
--- a/src/test/regress/expected/publication.out
+++ b/src/test/regress/expected/publication.out
@@ -181,7 +181,7 @@ ERROR:  cannot use a WHERE clause when removing table from publication "testpub5
 --------------------------+------------+---------+---------+---------+-----------+----------
  regress_publication_user | f          | t       | t       | t       | t         | f
 Tables:
-    "public.testpub_rf_tbl3"  WHERE ((e > 300) AND (e < 500))
+    "public.testpub_rf_tbl3" WHERE (((e > 300) AND (e < 500)))
 
 DROP TABLE testpub_rf_tbl1;
 DROP TABLE testpub_rf_tbl2;
-- 
2.30.2

---

From: "Euler Taveira" <[email-masked]>
Date: Sun, 28 Mar 2021 22:23:28 -0300

> mostly for debugging right now.)
I appreciate your work on it. I split into psql and pg_dump support just
because it was developed after the main patch. I expect them to be combined
into the main patch (0002) before committing it. This new patch set integrates
them into the main patch.

I totally forgot about the backward compatibility support. Good catch.  While
inspecting the code again, I did a small fix into the psql support. I added an
else as shown below so the query always returns the same number of columns and
we don't possibly have an issue while using a column number that is out of
range in PQgetisnull() a few lines later.

            if (pset.sversion >= 140000)
                appendPQExpBuffer(&buf,
                                  ", pg_get_expr(pr.prqual, c.oid)");
            else
                appendPQExpBuffer(&buf,
                                  ", NULL");

While testing the replication between v14 -> v10, I realized that even if the
tables in the publication have row filters, the data synchronization code won't
evaluate the row filter expressions. That's because the subscriber (v10) is
responsible to assemble the COPY command (possibly adding row filters) for data
synchronization and there is no such code in released versions. I added a new
sentence into copy_data parameter saying that row filters won't be used if
version is prior than 14. I also include this info into the commit message.

At this time, I didn't include the patch that changes the log_min_messages in
the row filter regression test. It was part of this patch set for testing
purposes only.

I don't expect the patch that measures row filter performance to be included
but I'm including it again in case someone wants to inspect the performance
numbers.

--
Euler Taveira
EDB   https://www.enterprisedb.com/
From 692572d0daaf8905e750889aa2f374764d86636c Mon Sep 17 00:00:00 2001
From: Euler Taveira <[email-masked]>
Date: Mon, 18 Jan 2021 12:07:51 -0300
Subject: [PATCH v13 1/2] Row filter for logical replication

If you choose to do the initial table synchronization, only data that satisfies
the row filters is sent. If the subscription has several publications in which
a table has been published with different WHERE clauses, rows must satisfy all
expressions to be copied. If subscriber is a pre-14 version, data
synchronization won't use row filters if they are defined in the publisher.
Previous versions cannot handle row filters.

If your publication contains partitioned table, the parameter
publish_via_partition_root determines if it uses the partition row filter (if
the parameter is false -- default) or the partitioned table row filter.
---
 doc/src/sgml/catalogs.sgml                  |   8 +
 doc/src/sgml/ref/alter_publication.sgml     |  11 +-
 doc/src/sgml/ref/create_publication.sgml    |  38 +++-
 doc/src/sgml/ref/create_subscription.sgml   |  11 +-
 src/backend/catalog/pg_publication.c        |  50 ++++-
 src/backend/commands/publicationcmds.c      |  98 +++++----
 src/backend/parser/gram.y                   |  26 ++-
 src/backend/parser/parse_agg.c              |  10 +
 src/backend/parser/parse_expr.c             |  13 ++
 src/backend/parser/parse_func.c             |   3 +
 src/backend/replication/logical/tablesync.c |  95 ++++++++-
 src/backend/replication/logical/worker.c    |  14 +-
 src/backend/replication/pgoutput/pgoutput.c | 202 ++++++++++++++++--
 src/bin/pg_dump/pg_dump.c                   |  24 ++-
 src/bin/pg_dump/pg_dump.h                   |   1 +
 src/bin/psql/describe.c                     |  15 +-
 src/include/catalog/pg_publication.h        |   9 +-
 src/include/catalog/pg_publication_rel.h    |   6 +
 src/include/nodes/nodes.h                   |   1 +
 src/include/nodes/parsenodes.h              |  11 +-
 src/include/parser/parse_node.h             |   1 +
 src/include/replication/logicalrelation.h   |   2 +
 src/test/regress/expected/publication.out   |  32 +++
 src/test/regress/sql/publication.sql        |  23 ++
 src/test/subscription/t/020_row_filter.pl   | 221 ++++++++++++++++++++
 25 files changed, 833 insertions(+), 92 deletions(-)
 create mode 100644 src/test/subscription/t/020_row_filter.pl

index f103d914a6..4f7ebcd967 100644
@@ -6213,6 +6213,14 @@ SCRAM-SHA-256$<replaceable>&lt;iteration count&gt;</replaceable>:<replaceable>&l
 
  <refsynopsisdiv>
 <synopsis>
 
      <para>
 
          <para>
 
+  <para>
 
  <refsect1>
 
+  <para>
index e812beee37..6fcd2fd447 100644
@@ -102,7 +102,16 @@ CREATE SUBSCRIPTION <replaceable class="parameter">subscription_name</replaceabl
+          to be copied. If any table in the publications has a
+          <literal>WHERE</literal> clause, data synchronization does not use it
+          if the subscriber is a <productname>PostgreSQL</productname> version
+          before 14.
index 86e415af89..78f5780fb7 100644
@@ -33,6 +33,9 @@
@@ -141,18 +144,20 @@ pg_relation_is_publishable(PG_FUNCTION_ARGS)

@@ -161,7 +166,7 @@ publication_add_relation(Oid pubid, Relation targetrel,
@@ -172,10 +177,27 @@ publication_add_relation(Oid pubid, Relation targetrel,

 	/* Form a tuple. */
@@ -187,7 +209,13 @@ publication_add_relation(Oid pubid, Relation targetrel,

@@ -202,14 +230,20 @@ publication_add_relation(Oid pubid, Relation targetrel,
 	recordDependencyOn(&myself, &referenced, DEPENDENCY_AUTO);

 
 
 	return myself;
 }

+	/*

 
-			foreach(newlc, rels)
-			{
 
 		/* And drop them. */
 
 	/*
 
 		/*

 	foreach(lc, rels)
 	{

 /*
 
 	foreach(lc, rels)
 	{

 
 	foreach(lc, rels)
 	{

index 7ff36bc842..e3833f5631 100644
@@ -426,13 +426,13 @@ static Node *makeRecursiveViewSelect(char *relname, List *aliases, Node *query);

 %type <list>	opt_fdw_options fdw_options
 %type <defelt>	fdw_option
@@ -9577,7 +9577,7 @@ opt_publication_for_tables:
 		;
 
 publication_for_tables:
-			FOR TABLE relation_expr_list
+			FOR TABLE publication_table_list
 				{
 					$$ = (Node *) $3;
 				}
@@ -9608,7 +9608,7 @@ AlterPublicationStmt:
 					n->options = $5;
 					$$ = (Node *)n;
 				}
-			| ALTER PUBLICATION name ADD_P TABLE relation_expr_list
+			| ALTER PUBLICATION name ADD_P TABLE publication_table_list
 				{
 					AlterPublicationStmt *n = makeNode(AlterPublicationStmt);
 					n->pubname = $3;
@@ -9616,7 +9616,7 @@ AlterPublicationStmt:
 					n->tableAction = DEFELEM_ADD;
 					$$ = (Node *)n;
 				}
-			| ALTER PUBLICATION name SET TABLE relation_expr_list
+			| ALTER PUBLICATION name SET TABLE publication_table_list
 				{
 					AlterPublicationStmt *n = makeNode(AlterPublicationStmt);
 					n->pubname = $3;
@@ -9624,7 +9624,7 @@ AlterPublicationStmt:
 					n->tableAction = DEFELEM_SET;
 					$$ = (Node *)n;
 				}
-			| ALTER PUBLICATION name DROP TABLE relation_expr_list
+			| ALTER PUBLICATION name DROP TABLE publication_table_list
 				{
 					AlterPublicationStmt *n = makeNode(AlterPublicationStmt);
 					n->pubname = $3;
@@ -9634,6 +9634,20 @@ AlterPublicationStmt:
 				}
 		;
 
+publication_table_list:
index ceb0bf597d..86c16c3def 100644
@@ -551,6 +551,13 @@ check_agglevels_and_constraints(ParseState *pstate, Node *expr)
 
 			break;
 
 		case EXPR_KIND_CYCLE_MARK:
 			errkind = true;
@@ -950,6 +957,9 @@ transformWindowFuncCall(ParseState *pstate, WindowFunc *wfunc,
 		case EXPR_KIND_CYCLE_MARK:
 			errkind = true;
 			break;
+		case EXPR_KIND_PUBLICATION_WHERE:
+			err = _("window functions are not allowed in publication WHERE expressions");
+			break;
 
 			/*
index 03373d551f..52c46fc7a7 100644
 
@@ -509,6 +516,7 @@ transformColumnRef(ParseState *pstate, ColumnRef *cref)
 
@@ -1769,6 +1777,9 @@ transformSubLink(ParseState *pstate, SubLink *sublink)
 
 			/*
@@ -3054,6 +3065,8 @@ ParseExprKindName(ParseExprKind exprKind)
 
 			/*
 			 * There is intentionally no default: case here, so that the
diff --git a/src/backend/parser/parse_func.c b/src/backend/parser/parse_func.c
index debef1d14f..bec927e1da 100644
--- a/src/backend/parser/parse_func.c
+++ b/src/backend/parser/parse_func.c
@@ -2533,6 +2533,9 @@ check_srf_call_placement(ParseState *pstate, Node *last_srf, int location)
 		case EXPR_KIND_CYCLE_MARK:
 			errkind = true;
 			break;
+		case EXPR_KIND_PUBLICATION_WHERE:
+			err = _("set-returning functions are not allowed in publication WHERE expressions");
+			break;
 
 			/*
index 8494db8f05..c3d6847b35 100644
 
 /*
 
 	lrel->nspname = nspname;
 	lrel->relname = relname;
@@ -790,6 +794,55 @@ fetch_remote_table_info(char *nspname, char *relname,
 	lrel->natts = natt;

@@ -803,6 +856,7 @@ copy_table(Relation rel)
 {
 	LogicalRepRelMapEntry *relmapentry;
 	LogicalRepRelation lrel;
+	List	   *qual = NIL;
 	WalRcvExecResult *res;
 	StringInfoData cmd;
 	CopyFromState cstate;
@@ -811,7 +865,7 @@ copy_table(Relation rel)

@@ -820,16 +874,23 @@ copy_table(Relation rel)
 
@@ -838,9 +899,31 @@ copy_table(Relation rel)
index 354fbe4b4b..04750f1778 100644

 

index 1b993fb032..00ad8f001f 100644
@@ -15,12 +15,18 @@
@@ -57,6 +63,10 @@ static void pgoutput_stream_abort(struct LogicalDecodingContext *ctx,
 
 static bool publications_valid;
 static bool in_streaming;
@@ -98,6 +108,7 @@ typedef struct RelationSyncEntry
 
 	bool		replicate_valid;
 
 	/*
 	 * OID of the relation to publish changes as.  For a partition, this may
@@ -121,7 +132,7 @@ static HTAB *RelationSyncCache = NULL;
 
@@ -491,6 +502,124 @@ send_relation_and_attrs(Relation relation, TransactionId xid,
 
+static ExprState *
@@ -518,7 +647,7 @@ pgoutput_change(LogicalDecodingContext *ctx, ReorderBufferTXN *txn,
 	if (in_streaming)
 		xid = change->txn->xid;

 	/* First check the table filter */
 	switch (change->action)
@@ -562,6 +691,10 @@ pgoutput_change(LogicalDecodingContext *ctx, ReorderBufferTXN *txn,
 						tuple = execute_attr_map_tuple(tuple, relentry->map);
 				}
 
@@ -588,6 +721,10 @@ pgoutput_change(LogicalDecodingContext *ctx, ReorderBufferTXN *txn,
 					}
 				}
 
@@ -610,6 +747,10 @@ pgoutput_change(LogicalDecodingContext *ctx, ReorderBufferTXN *txn,
 
@@ -657,12 +798,11 @@ pgoutput_truncate(LogicalDecodingContext *ctx, ReorderBufferTXN *txn,

 
@@ -672,10 +812,10 @@ pgoutput_truncate(LogicalDecodingContext *ctx, ReorderBufferTXN *txn,

@@ -944,16 +1084,21 @@ set_schema_sent_in_streamed_txn(RelationSyncEntry *entry, TransactionId xid)
  * when publishing.
  */

@@ -969,6 +1114,7 @@ get_rel_sync_entry(PGOutputData *data, Oid relid)
 
@@ -1000,6 +1146,9 @@ get_rel_sync_entry(PGOutputData *data, Oid relid)
 		{
 			Publication *pub = lfirst(lc);
 			bool		publish = false;
+			HeapTuple	rftuple;
+			Datum		rfdatum;
+			bool		rfisnull;
 
 			if (pub->alltables)
 			{
@@ -1059,9 +1208,29 @@ get_rel_sync_entry(PGOutputData *data, Oid relid)
 				entry->pubactions.pubtruncate |= pub->pubactions.pubtruncate;
 			}

 		list_free(pubids);
@@ -1167,6 +1336,7 @@ rel_sync_cache_publication_cb(Datum arg, int cacheid, uint32 hashvalue)
 {
 	HASH_SEQ_STATUS status;
 	RelationSyncEntry *entry;
+	MemoryContext oldctx;
 
 	/*
 	 * We can get here if the plugin was used in SQL interface as the
@@ -1176,6 +1346,8 @@ rel_sync_cache_publication_cb(Datum arg, int cacheid, uint32 hashvalue)
 	if (RelationSyncCache == NULL)
 		return;
 
@@ -1193,5 +1365,11 @@ rel_sync_cache_publication_cb(Datum arg, int cacheid, uint32 hashvalue)
index da6cc054b0..af57a50f27 100644
@@ -4140,9 +4141,16 @@ getPublicationTables(Archive *fout, TableInfo tblinfo[], int numTables)

 	ntups = PQntuples(res);
@@ -4151,6 +4159,7 @@ getPublicationTables(Archive *fout, TableInfo tblinfo[], int numTables)
 
@@ -4191,6 +4200,10 @@ getPublicationTables(Archive *fout, TableInfo tblinfo[], int numTables)
 
 		/* Decide whether we want to dump it */
 		selectDumpablePublicationTable(&(pubrinfo[j].dobj), fout);
@@ -4223,8 +4236,11 @@ dumpPublicationTable(Archive *fout, const PublicationRelInfo *pubrinfo)

 	/*
 
 /*
index 440249ff69..35d901295d 100644
@@ -6167,8 +6167,15 @@ describePublications(const char *pattern)
+								  ", NULL");
@@ -6197,6 +6204,10 @@ describePublications(const char *pattern)

 

 /* ----------------
 
+DECLARE_TOAST(pg_publication_rel, 8287, 8288);
+
 DECLARE_UNIQUE_INDEX_PKEY(pg_publication_rel_oid_index, 6112, on pg_publication_rel using btree(oid oid_ops));
 #define PublicationRelObjectIndexId 6112
 DECLARE_UNIQUE_INDEX(pg_publication_rel_prrelid_prpubid_index, 6113, on pg_publication_rel using btree(prrelid oid_ops, prpubid oid_ops));
diff --git a/src/include/nodes/nodes.h b/src/include/nodes/nodes.h
index 299956f329..f242092300 100644
--- a/src/include/nodes/nodes.h
+++ b/src/include/nodes/nodes.h
@@ -486,6 +486,7 @@ typedef enum NodeTag
 	T_PartitionRangeDatum,
 	T_PartitionCmd,
 	T_VacuumRelation,
+	T_PublicationTable,
 
 	/*
 	 * TAGS FOR REPLICATION GRAMMAR PARSE NODES (replnodes.h)
diff --git a/src/include/nodes/parsenodes.h b/src/include/nodes/parsenodes.h
index 12e0e026dc..a0161c50f8 100644
--- a/src/include/nodes/parsenodes.h
+++ b/src/include/nodes/parsenodes.h
@@ -3575,12 +3575,19 @@ typedef struct AlterTSConfigurationStmt
 } AlterTSConfigurationStmt;

@@ -3593,7 +3600,7 @@ typedef struct AlterPublicationStmt
 	List	   *options;		/* List of DefElem nodes */
 
index a71d7e1f74..01bed24717 100644
@@ -80,6 +80,7 @@ typedef enum ParseExprKind

index 63d6ab7a4e..da49a2e215 100644

From ef685a531a043ace67783c9ffe01e765617cfe15 Mon Sep 17 00:00:00 2001
From: Euler Taveira <[email-masked]>
Date: Sun, 31 Jan 2021 20:48:43 -0300
Subject: [PATCH v13 2/2] Measure row filter overhead

---

diff --git a/src/backend/replication/pgoutput/pgoutput.c b/src/backend/replication/pgoutput/pgoutput.c
index 00ad8f001f..a67459aec6 100644
--- a/src/backend/replication/pgoutput/pgoutput.c
+++ b/src/backend/replication/pgoutput/pgoutput.c
@@ -566,6 +566,8 @@ pgoutput_row_filter(Relation relation, HeapTuple oldtuple, HeapTuple newtuple, L
 	MemoryContext oldcxt;
 	ListCell   *lc;
 	bool		result = true;
+	instr_time	start_time;
+	instr_time	end_time;
 
 	/* Bail out if there is no row filter */
 	if (rowfilter == NIL)
@@ -575,6 +577,8 @@ pgoutput_row_filter(Relation relation, HeapTuple oldtuple, HeapTuple newtuple, L
 		 get_namespace_name(get_rel_namespace(RelationGetRelid(relation))),
 		 get_rel_name(relation->rd_id));

 	estate = create_estate_for_relation(relation);
@@ -617,6 +621,11 @@ pgoutput_row_filter(Relation relation, HeapTuple oldtuple, HeapTuple newtuple, L
 	ResetExprContext(ecxt);
 	FreeExecutorState(estate);

-- 
2.20.1

---

From: Rahila Syed <[email-masked]>
Date: Mon, 29 Mar 2021 15:15:40 +0530

Hi Euler,

While running some tests on v13 patches, I noticed that, in case the
published table data
already exists on the subscriber database before creating the subscription,
at the time of
CREATE subscription/table synchronization, an error as seen as follows

With the patch:

2021-03-29 14:32:56.265 IST [78467] STATEMENT:  CREATE_REPLICATION_SLOT
"pg_16406_sync_16390_6944995860755251708" LOGICAL pgoutput USE_SNAPSHOT
2021-03-29 14:32:56.279 IST [78467] LOG:  could not send data to client:
Broken pipe
2021-03-29 14:32:56.279 IST [78467] STATEMENT:  COPY (SELECT aid, bid,
abalance, filler FROM public.pgbench_accounts WHERE (aid > 0)) TO STDOUT
2021-03-29 14:32:56.279 IST [78467] FATAL:  connection to client lost
2021-03-29 14:32:56.279 IST [78467] STATEMENT:  COPY (SELECT aid, bid,
abalance, filler FROM public.pgbench_accounts WHERE (aid > 0)) TO STDOUT
2021-03-29 14:33:01.302 IST [78470] LOG:  logical decoding found consistent
point at 0/4E2B8460
2021-03-29 14:33:01.302 IST [78470] DETAIL:  There are no running
transactions.

Without the patch:

2021-03-29 15:05:01.581 IST [79029] ERROR:  duplicate key value violates
unique constraint "pgbench_branches_pkey"
2021-03-29 15:05:01.581 IST [79029] DETAIL:  Key (bid)=(1) already exists.
2021-03-29 15:05:01.581 IST [79029] CONTEXT:  COPY pgbench_branches, line 1
2021-03-29 15:05:01.583 IST [78538] LOG:  background worker "logical
replication worker" (PID 79029) exited with exit code 1
2021-03-29 15:05:06.593 IST [79031] LOG:  logical replication table
synchronization worker for subscription "test_sub2", table
"pgbench_branches" has started

Without the patch the COPY command throws an ERROR, but with the patch, a
similar scenario results in client connection being lost.

I didn't investigate it more, but looks like we should maintain the
existing behaviour when table synchronization fails
due to duplicate data.

Thank you,
Rahila Syed

---

From: "Euler Taveira" <[email-masked]>
Date: Mon, 29 Mar 2021 10:16:44 -0300

On Mon, Mar 29, 2021, at 6:45 AM, Rahila Syed wrote:
> While running some tests on v13 patches, I noticed that, in case the published table data 
> already exists on the subscriber database before creating the subscription, at the time of
> 2021-03-29 14:32:56.265 IST [78467] STATEMENT:  CREATE_REPLICATION_SLOT "pg_16406_sync_16390_6944995860755251708" LOGICAL pgoutput USE_SNAPSHOT
> 2021-03-29 14:32:56.279 IST [78467] LOG:  could not send data to client: Broken pipe
> 2021-03-29 14:32:56.279 IST [78467] STATEMENT:  COPY (SELECT aid, bid, abalance, filler FROM public.pgbench_accounts WHERE (aid > 0)) TO STDOUT
> 2021-03-29 14:32:56.279 IST [78467] STATEMENT:  COPY (SELECT aid, bid, abalance, filler FROM public.pgbench_accounts WHERE (aid > 0)) TO STDOUT
> 2021-03-29 14:33:01.302 IST [78470] LOG:  logical decoding found consistent point at 0/4E2B8460
> 2021-03-29 14:33:01.302 IST [78470] DETAIL:  There are no running transactions.
Rahila, I tried to reproduce this issue with the attached script but no luck. I always get

> Without the patch:
> 
> 2021-03-29 15:05:01.581 IST [79029] ERROR:  duplicate key value violates unique constraint "pgbench_branches_pkey"
> 2021-03-29 15:05:01.583 IST [78538] LOG:  background worker "logical replication worker" (PID 79029) exited with exit code 1
> 2021-03-29 15:05:06.593 IST [79031] LOG:  logical replication table synchronization worker for subscription "test_sub2", table "pgbench_branches" has started
... this message. The code that reports this error is from the COPY command.
Row filter modifications has no control over it. It seems somehow your
subscriber close the replication connection causing this issue. Can you
reproduce it consistently? If so, please share your steps.

---

From: Rahila Syed <[email-masked]>
Date: Mon, 29 Mar 2021 23:16:45 +0530

Hi,

> OK, Sorry for confusion. Actually both the errors are happening on
different servers. *Broken pipe*  error on publisher and
the following error on subscriber end. And the behaviour is consistent with
or without row filtering.

> Please ignore the report.

Thank you,
Rahila Syed

---

From: Amit Kapila <[email-masked]>
Date: Tue, 30 Mar 2021 16:53:56 +0530

On Mon, Mar 29, 2021 at 6:47 PM Euler Taveira <[email-masked]> wrote:
Few comments:
==============
1. How can we specify row filters for multiple tables for a
publication? Consider a case as below:
postgres=# CREATE TABLE tab_rowfilter_1 (a int primary key, b text);
CREATE TABLE
postgres=# CREATE TABLE tab_rowfilter_2 (c int primary key);
CREATE TABLE

postgres=# CREATE PUBLICATION tap_pub_1 FOR TABLE tab_rowfilter_1,
tab_rowfilter_2 WHERE (a > 1000 AND b <> 'filtered');
ERROR:  column "a" does not exist
LINE 1: ...FOR TABLE tab_rowfilter_1, tab_rowfilter_2 WHERE (a > 1000 A...

                                                             ^

postgres=# CREATE PUBLICATION tap_pub_1 FOR TABLE tab_rowfilter_1,
tab_rowfilter_2  WHERE (c > 1000);
CREATE PUBLICATION

It gives an error when I tried to specify the columns corresponding to
the first relation but is fine for columns for the second relation.
Then, I tried few more combinations like below but that didn't work.
CREATE PUBLICATION tap_pub_1 FOR TABLE tab_rowfilter_1 As t1,
tab_rowfilter_2 As t2 WHERE (t1.a > 1000 AND t1.b <> 'filtered');

Will users be allowed to specify join conditions among columns from
multiple tables?

2.
+ /*
+ errmsg("cannot use a WHERE clause when removing table from
publication \"%s\"",

Is there a reason to deal with this here separately rather than in the
ALTER PUBLICATION grammar?

-- 
With Regards,
Amit Kapila.

---

From: "Euler Taveira" <[email-masked]>
Date: Tue, 30 Mar 2021 22:45:55 -0300

On Tue, Mar 30, 2021, at 8:23 AM, Amit Kapila wrote:
> On Mon, Mar 29, 2021 at 6:47 PM Euler Taveira <[email-masked] <mailto:euler%40eulerto.com>> wrote:
> >
> Few comments:
> ==============
It is not possible. Row filter is a per table option. Isn't it clear from the
synopsis? The current design allows different row filter for tables in the same
publication. It is more flexible than a single row filter for a set of tables
(even if we would support such variant, there are some cases where the
condition should be different because the column names are not the same). You
can easily build a CREATE PUBLICATION command that adds the same row filter
multiple times using a DO block or use a similar approach in your favorite
language.

> Will users be allowed to specify join conditions among columns from
It seems you are envisioning row filter as a publication property instead of a
publication-relation property. Due to the flexibility that the later approach
provides, I decided to use it because it covers more use cases. Regarding
allowing joins, it could possibly slow down a critical path, no? This code path
is executed by every change. If there are interest in the join support, we
might add it in a future patch.

> ALTER PUBLICATION grammar?
Good question. IIRC the issue is that AlterPublicationStmt->tables has a list
element that was a relation_expr_list and was converted to
publication_table_list. If we share 'tables' with relation_expr_list (for ALTER
PUBLICATION ... DROP TABLE) and publication_table_list (for the other ALTER
PUBLICATION ... ADD|SET TABLE), the OpenTableList() has to know what list
element it is dealing with. I think I came to the conclusion that it is less
uglier to avoid changing OpenTableList() and CloseTableList().

[Doing some experimentation...]

Here is a patch that remove the referred code. It uses 2 distinct list
elements: relation_expr_list for ALTER PUBLICATION ... DROP TABLE and
publication_table_list for for ALTER PUBLICATION ... ADD|SET TABLE. A new
parameter was introduced to deal with the different elements of the list
'tables'.

--
Euler Taveira
EDB   https://www.enterprisedb.com/
From 99d36d0f5cb5f706c73fcdbb05772580f6814fe6 Mon Sep 17 00:00:00 2001
From: Euler Taveira <[email-masked]>
Date: Mon, 18 Jan 2021 12:07:51 -0300
Subject: [PATCH] Row filter for logical replication

 src/backend/commands/publicationcmds.c      | 125 +++++++----
 src/backend/parser/gram.y                   |  24 ++-
 src/test/regress/expected/publication.out   |  34 +++
 25 files changed, 851 insertions(+), 101 deletions(-)

  <refsynopsisdiv>
 <synopsis>
 
      <para>
 
          <para>
 
+  <para>
 
  <refsect1>
 
+  <para>

 
 
 	/* Form a tuple. */

 

 
 	return myself;
 }
diff --git a/src/backend/commands/publicationcmds.c b/src/backend/commands/publicationcmds.c
index 95c253c8e0..1a19ed9972 100644
--- a/src/backend/commands/publicationcmds.c
+++ b/src/backend/commands/publicationcmds.c
@@ -48,8 +48,8 @@
 /* Same as MAXNUMMESSAGES in sinvaladt.c */
 #define MAX_RELCACHE_INVAL_MSGS 4096
 
-static List *OpenTableList(List *tables);
-static void CloseTableList(List *rels);
+static List *OpenTableList(List *tables, bool is_drop);
+static void CloseTableList(List *rels, bool is_drop);
 static void PublicationAddTables(Oid pubid, List *rels, bool if_not_exists,
 								 AlterPublicationStmt *stmt);
 static void PublicationDropTables(Oid pubid, List *rels, bool missing_ok);
@@ -232,9 +232,9 @@ CreatePublication(CreatePublicationStmt *stmt)

-		rels = OpenTableList(stmt->tables);
+		rels = OpenTableList(stmt->tables, false);
 		PublicationAddTables(puboid, rels, true, NULL);
-		CloseTableList(rels);
+		CloseTableList(rels, false);
 	}
 
 	table_close(rel, RowExclusiveLock);
@@ -372,7 +372,10 @@ AlterPublicationTables(AlterPublicationStmt *stmt, Relation rel,

-	rels = OpenTableList(stmt->tables);
+	if (stmt->tableAction == DEFELEM_DROP)
+		rels = OpenTableList(stmt->tables, true);
+	else
+		rels = OpenTableList(stmt->tables, false);
 
 	if (stmt->tableAction == DEFELEM_ADD)
 		PublicationAddTables(pubid, rels, false, stmt);
@@ -385,31 +388,24 @@ AlterPublicationTables(AlterPublicationStmt *stmt, Relation rel,
 		List	   *delrels = NIL;
 		ListCell   *oldlc;

-			foreach(newlc, rels)
-			{
 
 		/* And drop them. */
@@ -421,10 +417,10 @@ AlterPublicationTables(AlterPublicationStmt *stmt, Relation rel,
 		 */
 		PublicationAddTables(pubid, rels, true, stmt);
 
-		CloseTableList(delrels);
+		CloseTableList(delrels, false);
 	}
 
-	CloseTableList(rels);
+	CloseTableList(rels, false);
 }
 
 /*
@@ -500,26 +496,42 @@ RemovePublicationRelById(Oid proid)
 
 /*
  * Open relations specified by a RangeVar list.
+ * AlterPublicationStmt->tables has a different list element, hence, is_drop
+ * indicates if it has a RangeVar (true) or PublicationTable (false).
  * The returned tables are locked in ShareUpdateExclusiveLock mode in order to
  * add them to a publication.
  */
 static List *
-OpenTableList(List *tables)
+OpenTableList(List *tables, bool is_drop)
 {
 	List	   *relids = NIL;
 	List	   *rels = NIL;
 	ListCell   *lc;
+	PublicationRelationInfo *pri;
 
 	/*
+		PublicationTable *t;
+		RangeVar   *rv;
+		bool		recurse;
 		Relation	rel;
 		Oid			myrelid;
 
+		if (is_drop)
+		{
+			rv = castNode(RangeVar, lfirst(lc));
+		}
+		else
+		{
+			t = lfirst(lc);
+			rv = castNode(RangeVar, t->relation);
+		}
+
+		recurse = rv->inh;
+
 		/* Allow query cancel in case this takes a long time */
 		CHECK_FOR_INTERRUPTS();
 
@@ -538,8 +550,12 @@ OpenTableList(List *tables)
+		if (!is_drop)
 
 		/*
@@ -572,7 +588,13 @@ OpenTableList(List *tables)
 
 				/* find_all_inheritors already got lock */
 				rel = table_open(childrelid, NoLock);
-				rels = lappend(rels, rel);
+				pri = palloc(sizeof(PublicationRelationInfo));
+				pri->relid = childrelid;
+				pri->relation = rel;
+				/* child inherits WHERE clause from parent */
+				if (!is_drop)
+					pri->whereClause = t->whereClause;
+				rels = lappend(rels, pri);
 				relids = lappend_oid(relids, childrelid);
 			}
 		}
@@ -587,16 +609,28 @@ OpenTableList(List *tables)
  * Close all relations in the list.
  */
 static void
-CloseTableList(List *rels)
+CloseTableList(List *rels, bool is_drop)
 {
 	ListCell   *lc;
 
 	foreach(lc, rels)
 	{
-		Relation	rel = (Relation) lfirst(lc);
+		if (is_drop)
+		{
+			Relation    rel = (Relation) lfirst(lc);
+			
+			table_close(rel, NoLock);
+		}
+		else
+		{
+			PublicationRelationInfo *pri = (PublicationRelationInfo *) lfirst(lc);
 
+	if (!is_drop)
+		list_free_deep(rels);
 }
 
 /*
@@ -612,15 +646,15 @@ PublicationAddTables(Oid pubid, List *rels, bool if_not_exists,
 
 	foreach(lc, rels)
 	{

-		obj = publication_add_relation(pubid, rel, if_not_exists);
+		obj = publication_add_relation(pubid, pri, if_not_exists);
 		if (stmt)
 		{
 			EventTriggerCollectSimpleCommand(obj, InvalidObjectAddress,
@@ -644,11 +678,10 @@ PublicationDropTables(Oid pubid, List *rels, bool missing_ok)
 
 	foreach(lc, rels)
 	{
 
@@ -658,7 +691,7 @@ PublicationDropTables(Oid pubid, List *rels, bool missing_ok)
 
index 7ff36bc842..a2899702c1 100644

 
 publication_for_tables:
 
+publication_table_list:
 
 			break;

 			/*

 
 			/*
 
 			/*
 
 			/*
 
 /*
 
 	lrel->nspname = nspname;
 	lrel->relname = relname;

 

 

 

 
 	bool		replicate_valid;
 
 	/*

+static ExprState *

 

 

 

 

 			if (pub->alltables)
 			{

 		list_free(pubids);
 
 	/*
 
index 25717ce0e6..4b2316b01e 100644
@@ -4220,6 +4220,7 @@ getPublicationTables(Archive *fout, TableInfo tblinfo[], int numTables)
@@ -4230,9 +4231,16 @@ getPublicationTables(Archive *fout, TableInfo tblinfo[], int numTables)

 	ntups = PQntuples(res);
@@ -4241,6 +4249,7 @@ getPublicationTables(Archive *fout, TableInfo tblinfo[], int numTables)
 
@@ -4281,6 +4290,10 @@ getPublicationTables(Archive *fout, TableInfo tblinfo[], int numTables)
 
 		/* Decide whether we want to dump it */
 		selectDumpablePublicationTable(&(pubrinfo[j].dobj), fout);
@@ -4313,8 +4326,11 @@ dumpPublicationTable(Archive *fout, const PublicationRelInfo *pubrinfo)

 	/*
 
 /*

 

 /* ----------------

 	/*
index 334262b1dd..f34e92b2b2 100644

 

+extern EState *create_estate_for_relation(Relation rel);
+
 #endif							/* LOGICALRELATION_H */
diff --git a/src/test/regress/expected/publication.out b/src/test/regress/expected/publication.out
index 63d6ab7a4e..96d869dd27 100644
--- a/src/test/regress/expected/publication.out
+++ b/src/test/regress/expected/publication.out
@@ -156,6 +156,40 @@ Tables:
 
+ERROR:  syntax error at or near "WHERE"
+LINE 1: ...R PUBLICATION testpub5 DROP TABLE testpub_rf_tbl3 WHERE (e <...

---

From: Amit Kapila <[email-masked]>
Date: Wed, 31 Mar 2021 18:16:44 +0530

On Wed, Mar 31, 2021 at 7:17 AM Euler Taveira <[email-masked]> wrote:
> synopsis?

Sorry, it seems I didn't read it properly earlier, now I got it.

> Here is a patch that remove the referred code.
>

Thanks, few more comments:
1. In pgoutput_change, we are always sending schema even though we
don't send actual data because of row filters. It may not be a problem
in many cases but I guess for some odd cases we can avoid sending
extra information.

2. In get_rel_sync_entry(), we are caching the qual for rel_sync_entry
even though we won't publish it which seems unnecessary?

3.
@@ -1193,5 +1365,11 @@ rel_sync_cache_publication_cb(Datum arg, int
cacheid, uint32 hashvalue)
  entry->pubactions.pubupdate = false;
  entry->pubactions.pubdelete = false;
  entry->pubactions.pubtruncate = false;
+
+ if (entry->qual != NIL)
+ list_free_deep(entry->qual);

Seeing one previous comment in this thread [1], I am wondering if
list_free_deep is enough here?

4. Can we write explicitly in the docs that row filters won't apply
for Truncate operation?

5. Getting some whitespace errors:
git am /d/PostgreSQL/Patches/logical_replication/row_filter/v14-0001-Row-filter-for-logical-replication.patch
.git/rebase-apply/patch:487: trailing whitespace.

warning: 1 line adds whitespace errors.
Applying: Row filter for logical replication

[1] - https://www.postgresql.org/message-id/20181123161933.jpepibtyayflz2xg%40alvherre.pgsql

-- 
With Regards,
Amit Kapila.

---

From: Andres Freund <[email-masked]>
Date: Wed, 31 Mar 2021 12:17:10 -0700

Hi,

As far as I can tell you have not *AT ALL* addressed that it is *NOT
SAFE* to evaluate arbitrary expressions from within an output
plugin. Despite that having been brought up multiple times.

> +		result = pgoutput_row_filter_exec_expr(exprstate, ecxt);

Also, this still seems like an *extremely* expensive thing to do for
each tuple. It'll often be *vastly* faster to just send the data than to
the other side.

This just cannot be done once per tuple. It has to be cached.

I don't see how these issues can be addressed in the next 7 days,
therefore I think this unfortunately needs to be marked as returned with
feedback.

Greetings,

Andres Freund

---

From: Peter Smith <[email-masked]>
Date: Mon, 10 May 2021 18:19:42 +1000

On Wed, Mar 31, 2021 at 12:47 PM Euler Taveira <[email-masked]> wrote:
....

> parameter was introduced to deal with the different elements of the list

AFAIK this is the latest patch available, but FYI it no longer applies
cleanly on HEAD.

git apply ../patches_misc/0001-Row-filter-for-logical-replication.patch
../patches_misc/0001-Row-filter-for-logical-replication.patch:518:
trailing whitespace.
error: patch failed: src/backend/parser/gram.y:426
error: src/backend/parser/gram.y: patch does not apply
error: patch failed: src/backend/replication/logical/worker.c:340
error: src/backend/replication/logical/worker.c: patch does not apply

--------
Kind Regards,
Peter Smith.
Fujitsu Australia.

---

From: "Euler Taveira" <[email-masked]>
Date: Mon, 10 May 2021 10:41:24 -0300

> AFAIK this is the latest patch available, but FYI it no longer applies
Peter, the last patch is broken since f3b141c4825. I'm still working on it for
the next CF. I already addressed the points suggested by Amit in his last
review; however, I'm still working on a cache for evaluating expression as
suggested by Andres. I hope to post a new patch soon.

---

From: Peter Smith <[email-masked]>
Date: Wed, 9 Jun 2021 10:02:52 +1000

> suggested by Andres. I hope to post a new patch soon.

Is there any ETA for your new patch?

In the interim can you rebase the old patch just so it builds and I can try it?

------
Kind Regards,
Peter Smith.
Fujitsu Australia

---

From: Amit Kapila <[email-masked]>
Date: Fri, 18 Jun 2021 17:10:38 +0530

> In the interim can you rebase the old patch just so it builds and I can try it?

I have rebased the patch so that you can try it out. The main thing I
have done is to remove changes in worker.c and created a specialized
function to create estate for pgoutput.c as I don't think we need what
is done in worker.c.

Euler, do let me know if you are not happy with the change in pgoutput.c?

-- 
With Regards,
Amit Kapila.

---

From: Peter Smith <[email-masked]>
Date: Tue, 22 Jun 2021 12:38:47 +1000

On Fri, Jun 18, 2021 at 9:40 PM Amit Kapila <[email-masked]> wrote:
[...]
> function to create estate for pgoutput.c as I don't think we need what

Thanks for the recent rebase.

- The v15 patch applies OK (albeit with whitespace warning)
- make check is passing OK
- the new TAP tests 020_row_filter is passing OK.

------
Kind Regards,
Peter Smith.
Fujitsu Australia

---

From: "Euler Taveira" <[email-masked]>
Date: Wed, 30 Jun 2021 21:42:58 -0300

> Euler, do let me know if you are not happy with the change in pgoutput.c?
Amit, thanks for rebasing this patch. I already had a similar rebased patch in
my local tree. A recent patch broke your version v15 so I rebased it.

I like the idea of a simple create_estate_for_relation() function (I fixed an
oversight regarding GetCurrentCommandId(false) because it is used only for
read-only purposes). This patch also replaces all references to version 14.

Commit ef948050 made some changes in the snapshot handling. Set the current
active snapshot might not be required but future changes to allow functions
will need it.

As the previous patches, it includes commits (0002 and 0003) that are not
intended to be committed. They are available for test-only purposes.

--
Euler Taveira
EDB   https://www.enterprisedb.com/
From 9c27d11efd2ac8257cc2664e5da82fd19f012ebc Mon Sep 17 00:00:00 2001
From: Euler Taveira <[email-masked]>
Date: Mon, 18 Jan 2021 12:07:51 -0300
Subject: [PATCH 1/3] Row filter for logical replication

expressions to be copied. If subscriber is a pre-15 version, data

If your publication contains partitioned table, the parameter
publish_via_partition_root determines if it uses the partition row filter (if
the parameter is false -- default) or the partitioned table row filter.
---
 doc/src/sgml/catalogs.sgml                  |   8 +
 doc/src/sgml/ref/alter_publication.sgml     |  11 +-
 doc/src/sgml/ref/create_publication.sgml    |  32 ++-
 doc/src/sgml/ref/create_subscription.sgml   |  11 +-
 src/backend/catalog/pg_publication.c        |  50 +++-
 src/backend/commands/publicationcmds.c      | 125 ++++++----
 src/backend/parser/gram.y                   |  24 +-
 src/backend/parser/parse_agg.c              |  10 +
 src/backend/parser/parse_expr.c             |  13 ++
 src/backend/parser/parse_func.c             |   3 +
 src/backend/replication/logical/tablesync.c |  94 +++++++-
 src/backend/replication/pgoutput/pgoutput.c | 241 ++++++++++++++++++--
 src/bin/pg_dump/pg_dump.c                   |  24 +-
 src/bin/pg_dump/pg_dump.h                   |   1 +
 src/bin/psql/describe.c                     |  15 +-
 src/include/catalog/pg_publication.h        |   9 +-
 src/include/catalog/pg_publication_rel.h    |   6 +
 src/include/nodes/nodes.h                   |   1 +
 src/include/nodes/parsenodes.h              |  11 +-
 src/include/parser/parse_node.h             |   1 +
 src/test/regress/expected/publication.out   |  34 +++
 src/test/regress/sql/publication.sql        |  23 ++
 src/test/subscription/t/020_row_filter.pl   | 221 ++++++++++++++++++
 23 files changed, 872 insertions(+), 96 deletions(-)
 create mode 100644 src/test/subscription/t/020_row_filter.pl

index f517a7d4af..dbf2f46c00 100644
@@ -6233,6 +6233,14 @@ SCRAM-SHA-256$<replaceable>&lt;iteration count&gt;</replaceable>:<replaceable>&l
 
  <refsynopsisdiv>
 <synopsis>
index ff82fbca55..5c2b7d0bd2 100644
+      expression. It has no effect on <literal>TRUNCATE</literal> commands.
      </para>
 
      <para>
@@ -131,9 +135,9 @@ CREATE PUBLICATION <replaceable class="parameter">name</replaceable>
 
          <para>
@@ -182,6 +186,14 @@ CREATE PUBLICATION <replaceable class="parameter">name</replaceable>
    disallowed on those tables.
   </para>
 
+  <para>
@@ -197,6 +209,11 @@ CREATE PUBLICATION <replaceable class="parameter">name</replaceable>
 
  <refsect1>
@@ -209,6 +226,13 @@ CREATE PUBLICATION mypublication FOR TABLE users, departments;
 </programlisting>
   </para>
 
+  <para>
index e812beee37..7183700ed9 100644
+          before 15.

 
 
 	/* Form a tuple. */

 

 
 	return myself;
 }
index 95c253c8e0..9637d3ddba 100644

 

 

 
-			foreach(newlc, rels)
-			{
 
 		/* And drop them. */

-	CloseTableList(rels);
 
 /*
 
 /*
 
 	/*
 	 * Open, share-lock, and check all the explicitly-specified relations
 	 */
 	foreach(lc, tables)
 	{
-		RangeVar   *rv = castNode(RangeVar, lfirst(lc));
-		bool		recurse = rv->inh;
+		PublicationTable *t = NULL;
+		RangeVar   *rv;
+		bool		recurse;
 		Relation	rel;
 		Oid			myrelid;
 
+		if (is_drop)
+		{

 		/*

 	foreach(lc, rels)
 	{

 /*
 
 	foreach(lc, rels)
 	{

 
 	foreach(lc, rels)
 	{

index eb24195438..d82ea003db 100644
@@ -426,14 +426,14 @@ static Node *makeRecursiveViewSelect(char *relname, List *aliases, Node *query);
 
 %type <node>	opt_routine_body
 
 %type <list>	opt_fdw_options fdw_options
 %type <defelt>	fdw_option
@@ -9612,7 +9612,7 @@ opt_publication_for_tables:
 		;
 
 publication_for_tables:
-			FOR TABLE relation_expr_list
+			FOR TABLE publication_table_list
 				{
 					$$ = (Node *) $3;
 				}
@@ -9643,7 +9643,7 @@ AlterPublicationStmt:
 					n->options = $5;
 					$$ = (Node *)n;
 				}
-			| ALTER PUBLICATION name ADD_P TABLE relation_expr_list
+			| ALTER PUBLICATION name ADD_P TABLE publication_table_list
 				{
 					AlterPublicationStmt *n = makeNode(AlterPublicationStmt);
 					n->pubname = $3;
@@ -9651,7 +9651,7 @@ AlterPublicationStmt:
 					n->tableAction = DEFELEM_ADD;
 					$$ = (Node *)n;
 				}
-			| ALTER PUBLICATION name SET TABLE relation_expr_list
+			| ALTER PUBLICATION name SET TABLE publication_table_list
 				{
 					AlterPublicationStmt *n = makeNode(AlterPublicationStmt);
 					n->pubname = $3;
@@ -9669,6 +9669,20 @@ AlterPublicationStmt:
 				}
 		;
 
+publication_table_list:
index a25f8d5b98..5619ac6904 100644
 
 			break;

 			/*
index f928c32311..fc4170e723 100644

 
 			/*
@@ -3089,6 +3100,8 @@ ParseExprKindName(ParseExprKind exprKind)
 
 			/*
 			 * There is intentionally no default: case here, so that the
diff --git a/src/backend/parser/parse_func.c b/src/backend/parser/parse_func.c
index 3cec8de7da..e946f17c64 100644
--- a/src/backend/parser/parse_func.c
+++ b/src/backend/parser/parse_func.c
@@ -2655,6 +2655,9 @@ check_srf_call_placement(ParseState *pstate, Node *last_srf, int location)
 		case EXPR_KIND_CYCLE_MARK:
 			errkind = true;
 			break;
+		case EXPR_KIND_PUBLICATION_WHERE:
+			err = _("set-returning functions are not allowed in publication WHERE expressions");
+			break;
 
 			/*
 			 * There is intentionally no default: case here, so that the
diff --git a/src/backend/replication/logical/tablesync.c b/src/backend/replication/logical/tablesync.c
index 682c107e74..980826a502 100644
--- a/src/backend/replication/logical/tablesync.c
+++ b/src/backend/replication/logical/tablesync.c
@@ -691,19 +691,23 @@ copy_read_data(void *outbuf, int minread, int maxread)
 
 /*
 
 	lrel->nspname = nspname;
 	lrel->relname = relname;
@@ -799,6 +803,55 @@ fetch_remote_table_info(char *nspname, char *relname,
 	lrel->natts = natt;
 
+	if (walrcv_server_version(LogRepWorkerWalRcvConn) >= 150000)
+		res = walrcv_exec(LogRepWorkerWalRcvConn, cmd.data, 1, qualRow);
 
@@ -812,6 +865,7 @@ copy_table(Relation rel)
 {
 	LogicalRepRelMapEntry *relmapentry;
 	LogicalRepRelation lrel;
+	List	   *qual = NIL;
 	WalRcvExecResult *res;
 	StringInfoData cmd;
 	CopyFromState cstate;
@@ -820,7 +874,7 @@ copy_table(Relation rel)

@@ -829,16 +883,23 @@ copy_table(Relation rel)
 
@@ -847,8 +908,29 @@ copy_table(Relation rel)
 	res = walrcv_exec(LogRepWorkerWalRcvConn, cmd.data, 0, NULL);
index abd5217ab1..10f85365fc 100644
@@ -13,18 +13,26 @@
 #include "postgres.h"
 
+#include "access/xact.h"
+#include "utils/snapmgr.h"
 
@@ -61,6 +69,11 @@ static void pgoutput_stream_abort(struct LogicalDecodingContext *ctx,
 static void pgoutput_stream_commit(struct LogicalDecodingContext *ctx,
 								   ReorderBufferTXN *txn,
 								   XLogRecPtr commit_lsn);
+static EState *create_estate_for_relation(Relation rel);
+static ExprState *pgoutput_row_filter_prepare_expr(Node *rfnode, EState *estate);
+static bool pgoutput_row_filter_exec_expr(ExprState *state, ExprContext *econtext);
+static bool pgoutput_row_filter(Relation relation, HeapTuple oldtuple,
+								HeapTuple newtuple, List *rowfilter);
 
 static bool publications_valid;
 static bool in_streaming;
@@ -99,6 +112,7 @@ typedef struct RelationSyncEntry
 
 	bool		replicate_valid;
 
 	/*
 	 * OID of the relation to publish changes as.  For a partition, this may
@@ -122,7 +136,7 @@ static HTAB *RelationSyncCache = NULL;
 
@@ -520,6 +534,148 @@ send_relation_and_attrs(Relation relation, TransactionId xid,
 
+static EState *
+	estate = CreateExecutorState();
+	rte = makeNode(RangeTblEntry);
+	rte->rtekind = RTE_RELATION;
+	rte->rellockmode = AccessShareLock;
+	ExecInitRangeTable(estate, list_make1(rte));
+	estate->es_output_cid = GetCurrentCommandId(false);
+	return estate;
+	PushActiveSnapshot(GetTransactionSnapshot());
+			 result ? "" : "not ");
+	/* Cleanup allocated resources */
+	PopActiveSnapshot();
@@ -547,7 +703,7 @@ pgoutput_change(LogicalDecodingContext *ctx, ReorderBufferTXN *txn,
 	if (in_streaming)
 		xid = change->txn->xid;

 	/* First check the table filter */
 	switch (change->action)
@@ -571,8 +727,6 @@ pgoutput_change(LogicalDecodingContext *ctx, ReorderBufferTXN *txn,
 	/* Avoid leaking memory by using and resetting our own context */
 	old = MemoryContextSwitchTo(data->context);
 
-	maybe_send_schema(ctx, txn, change, relation, relentry);
-
 	/* Send the data */
 	switch (change->action)
 	{
@@ -580,6 +734,16 @@ pgoutput_change(LogicalDecodingContext *ctx, ReorderBufferTXN *txn,
 			{
 				HeapTuple	tuple = &change->data.tp.newtuple->tuple;
 
+				/* Check row filter. */
+				if (!pgoutput_row_filter(relation, NULL, tuple, relentry->qual))
+					return;
+
+				/*
+				 * Schema should be sent before the logic that replaces the
+				 * relation because it also sends the ancestor's relation.
+				 */
+				maybe_send_schema(ctx, txn, change, relation, relentry);
+
 				/* Switch relation if publishing via root. */
 				if (relentry->publish_as_relid != RelationGetRelid(relation))
 				{
@@ -603,6 +767,12 @@ pgoutput_change(LogicalDecodingContext *ctx, ReorderBufferTXN *txn,
 				&change->data.tp.oldtuple->tuple : NULL;
 				HeapTuple	newtuple = &change->data.tp.newtuple->tuple;
 
+				/* Check row filter. */
+				if (!pgoutput_row_filter(relation, oldtuple, newtuple, relentry->qual))
+					return;
+
+				maybe_send_schema(ctx, txn, change, relation, relentry);
+
 				/* Switch relation if publishing via root. */
 				if (relentry->publish_as_relid != RelationGetRelid(relation))
 				{
@@ -631,6 +801,12 @@ pgoutput_change(LogicalDecodingContext *ctx, ReorderBufferTXN *txn,
 			{
 				HeapTuple	oldtuple = &change->data.tp.oldtuple->tuple;
 
+				/* Check row filter. */
+				if (!pgoutput_row_filter(relation, oldtuple, NULL, relentry->qual))
+					return;
+
+				maybe_send_schema(ctx, txn, change, relation, relentry);
+
 				/* Switch relation if publishing via root. */
 				if (relentry->publish_as_relid != RelationGetRelid(relation))
 				{
@@ -689,12 +865,11 @@ pgoutput_truncate(LogicalDecodingContext *ctx, ReorderBufferTXN *txn,
 	for (i = 0; i < nrelations; i++)
 	{
 		Relation	relation = relations[i];
-		Oid			relid = RelationGetRelid(relation);

 
@@ -704,10 +879,10 @@ pgoutput_truncate(LogicalDecodingContext *ctx, ReorderBufferTXN *txn,

@@ -1005,16 +1180,21 @@ set_schema_sent_in_streamed_txn(RelationSyncEntry *entry, TransactionId xid)
  * when publishing.
  */

+	relid = RelationGetRelid(rel);
+	am_partition = get_rel_relispartition(relid);
+	relkind = get_rel_relkind(relid);
+
 	/* Find cached relation info, creating if not found */
 	entry = (RelationSyncEntry *) hash_search(RelationSyncCache,
 											  (void *) &relid,
@@ -1030,6 +1210,7 @@ get_rel_sync_entry(PGOutputData *data, Oid relid)
 		entry->replicate_valid = false;
 		entry->pubactions.pubinsert = entry->pubactions.pubupdate =
 			entry->pubactions.pubdelete = entry->pubactions.pubtruncate = false;
+		entry->qual = NIL;
 		entry->publish_as_relid = InvalidOid;
 		entry->map = NULL;		/* will be set by maybe_send_schema() if
 								 * needed */
@@ -1063,6 +1244,9 @@ get_rel_sync_entry(PGOutputData *data, Oid relid)
 		{
 			Publication *pub = lfirst(lc);
 			bool		publish = false;
+			HeapTuple	rftuple;
+			Datum		rfdatum;
+			bool		rfisnull;
 
 			if (pub->alltables)
 			{
@@ -1122,9 +1306,29 @@ get_rel_sync_entry(PGOutputData *data, Oid relid)
 				entry->pubactions.pubtruncate |= pub->pubactions.pubtruncate;
 			}

 		list_free(pubids);
@@ -1242,6 +1446,7 @@ rel_sync_cache_publication_cb(Datum arg, int cacheid, uint32 hashvalue)
 {
 	HASH_SEQ_STATUS status;
 	RelationSyncEntry *entry;
+	MemoryContext oldctx;
 
 	/*
 	 * We can get here if the plugin was used in SQL interface as the
@@ -1251,6 +1456,8 @@ rel_sync_cache_publication_cb(Datum arg, int cacheid, uint32 hashvalue)
 	if (RelationSyncCache == NULL)
 		return;
 
+	oldctx = MemoryContextSwitchTo(CacheMemoryContext);
+
 	/*
 	 * There is no way to find which entry in our cache the hash belongs to so
 	 * mark the whole cache as invalid.
@@ -1268,5 +1475,11 @@ rel_sync_cache_publication_cb(Datum arg, int cacheid, uint32 hashvalue)
 		entry->pubactions.pubupdate = false;
 		entry->pubactions.pubdelete = false;
 		entry->pubactions.pubtruncate = false;
+
+		if (entry->qual != NIL)
+			list_free_deep(entry->qual);
+		entry->qual = NIL;
 	}
+
+	MemoryContextSwitchTo(oldctx);
 }
diff --git a/src/bin/pg_dump/pg_dump.c b/src/bin/pg_dump/pg_dump.c
index 321152151d..6f944ec60d 100644
--- a/src/bin/pg_dump/pg_dump.c
+++ b/src/bin/pg_dump/pg_dump.c
@@ -4172,6 +4172,7 @@ getPublicationTables(Archive *fout, TableInfo tblinfo[], int numTables)
 	int			i_oid;
 	int			i_prpubid;
 	int			i_prrelid;
+	int			i_prrelqual;
 	int			i,
 				j,
 				ntups;
@@ -4182,9 +4183,16 @@ getPublicationTables(Archive *fout, TableInfo tblinfo[], int numTables)
 	query = createPQExpBuffer();
 
+	if (fout->remoteVersion >= 150000)
 
 	ntups = PQntuples(res);
@@ -4193,6 +4201,7 @@ getPublicationTables(Archive *fout, TableInfo tblinfo[], int numTables)
 
@@ -4233,6 +4242,10 @@ getPublicationTables(Archive *fout, TableInfo tblinfo[], int numTables)
 
 		/* Decide whether we want to dump it */
 		selectDumpablePublicationTable(&(pubrinfo[j].dobj), fout);
@@ -4265,8 +4278,11 @@ dumpPublicationTable(Archive *fout, const PublicationRelInfo *pubrinfo)

 	/*
index ba9bc6ddd2..7d72d498c1 100644
 
 /*
diff --git a/src/bin/psql/describe.c b/src/bin/psql/describe.c
index 2abf255798..e2e64cb3bf 100644
--- a/src/bin/psql/describe.c
+++ b/src/bin/psql/describe.c
@@ -6329,8 +6329,15 @@ describePublications(const char *pattern)
 		if (!puballtables)
 		{
 			printfPQExpBuffer(&buf,
-							  "SELECT n.nspname, c.relname\n"
-							  "FROM pg_catalog.pg_class c,\n"
+							  "SELECT n.nspname, c.relname");
+			if (pset.sversion >= 150000)
+				appendPQExpBuffer(&buf,
+								  ", pg_get_expr(pr.prqual, c.oid)");
+			else
+				appendPQExpBuffer(&buf,
+								  ", NULL");
+			appendPQExpBuffer(&buf,
+							  "\nFROM pg_catalog.pg_class c,\n"
 							  "     pg_catalog.pg_namespace n,\n"
 							  "     pg_catalog.pg_publication_rel pr\n"
 							  "WHERE c.relnamespace = n.oid\n"
@@ -6359,6 +6366,10 @@ describePublications(const char *pattern)
 								  PQgetvalue(tabres, j, 0),
 								  PQgetvalue(tabres, j, 1));
 
index f332bad4d4..333c2b581d 100644
@@ -83,6 +83,13 @@ typedef struct Publication
 
@@ -108,7 +115,7 @@ extern List *GetAllTablesPublications(void);

index b5d5504cbb..154bb61777 100644
 
 /* ----------------
 
+DECLARE_TOAST(pg_publication_rel, 8287, 8288);
+
 DECLARE_UNIQUE_INDEX_PKEY(pg_publication_rel_oid_index, 6112, PublicationRelObjectIndexId, on pg_publication_rel using btree(oid oid_ops));
 DECLARE_UNIQUE_INDEX(pg_publication_rel_prrelid_prpubid_index, 6113, PublicationRelPrrelidPrpubidIndexId, on pg_publication_rel using btree(prrelid oid_ops, prpubid oid_ops));
 
diff --git a/src/include/nodes/nodes.h b/src/include/nodes/nodes.h
index d9e417bcd7..2037705f45 100644
--- a/src/include/nodes/nodes.h
+++ b/src/include/nodes/nodes.h
@@ -491,6 +491,7 @@ typedef enum NodeTag
 	T_PartitionRangeDatum,
 	T_PartitionCmd,
 	T_VacuumRelation,
+	T_PublicationTable,
 
 	/*
 	 * TAGS FOR REPLICATION GRAMMAR PARSE NODES (replnodes.h)
diff --git a/src/include/nodes/parsenodes.h b/src/include/nodes/parsenodes.h
index def9651b34..cf815cc0f2 100644
--- a/src/include/nodes/parsenodes.h
+++ b/src/include/nodes/parsenodes.h
@@ -3624,12 +3624,19 @@ typedef struct AlterTSConfigurationStmt
 } AlterTSConfigurationStmt;

@@ -3642,7 +3649,7 @@ typedef struct AlterPublicationStmt
 	List	   *options;		/* List of DefElem nodes */
 
index 1500de2dd0..4537543a7b 100644

 

From 0ff56520b40129c1fc1fbb1379214a1b94413a9f Mon Sep 17 00:00:00 2001
From: Euler Taveira <[email-masked]>
Date: Sun, 31 Jan 2021 20:48:43 -0300
Subject: [PATCH 2/3] Measure row filter overhead

---

diff --git a/src/backend/replication/pgoutput/pgoutput.c b/src/backend/replication/pgoutput/pgoutput.c
index 10f85365fc..4677d21b62 100644
--- a/src/backend/replication/pgoutput/pgoutput.c
+++ b/src/backend/replication/pgoutput/pgoutput.c
@@ -618,6 +618,8 @@ pgoutput_row_filter(Relation relation, HeapTuple oldtuple, HeapTuple newtuple, L
 	MemoryContext oldcxt;
 	ListCell   *lc;
 	bool		result = true;
+	instr_time	start_time;
+	instr_time	end_time;
 
 	/* Bail out if there is no row filter */
 	if (rowfilter == NIL)
@@ -627,6 +629,8 @@ pgoutput_row_filter(Relation relation, HeapTuple oldtuple, HeapTuple newtuple, L
 		 get_namespace_name(get_rel_namespace(RelationGetRelid(relation))),
 		 get_rel_name(relation->rd_id));

 	PushActiveSnapshot(GetTransactionSnapshot());
@@ -673,6 +677,11 @@ pgoutput_row_filter(Relation relation, HeapTuple oldtuple, HeapTuple newtuple, L
 	FreeExecutorState(estate);
 	PopActiveSnapshot();

-- 
2.20.1

From 4275d48c9f59b6f3b2bc99b4563119da3909dd56 Mon Sep 17 00:00:00 2001
From: Euler Taveira <[email-masked]>
Date: Fri, 26 Feb 2021 21:08:10 -0300
Subject: [PATCH 3/3] Debug messages

---

index fc4170e723..bd02576c86 100644
 
+	/*
+	 * T_FuncCall: 349
+	 * EXPR_KIND_PUBLICATION_WHERE: 42
+	 */
+	elog(DEBUG3, "nodeTag(expr): %d ; pstate->p_expr_kind: %d ; nodeToString(expr): %s", nodeTag(expr), pstate->p_expr_kind, nodeToString(expr));
+
 	/* Guard against stack overflow due to overly complex expressions */
 	check_stack_depth();
 
diff --git a/src/test/subscription/t/020_row_filter.pl b/src/test/subscription/t/020_row_filter.pl
index 35a41741d3..8c305cf1dc 100644
--- a/src/test/subscription/t/020_row_filter.pl
+++ b/src/test/subscription/t/020_row_filter.pl
@@ -8,6 +8,7 @@ use Test::More tests => 7;
 # create publisher node
 my $node_publisher = get_new_node('publisher');
 $node_publisher->init(allows_streaming => 'logical');
+$node_publisher->append_conf('postgresql.conf', 'log_min_messages = DEBUG3');
 $node_publisher->start;
 
 # create subscriber node
-- 
2.20.1

---

From: Peter Smith <[email-masked]>
Date: Fri, 2 Jul 2021 17:29:59 +1000

Hi.

I have been looking at the latest patch set (v16). Below are my review
comments and some patches.

The patches are:
v16-0001. This is identical to your previously posted 0001 patch. (I
only attached it again hoping it can allow the cfbot to keep working
OK).
v16-0002,0003. These are for demonstrating some of the review comments
v16-0004. This is a POC plan cache for your consideration.

//////////

REVIEW COMMENTS
===============

1. Patch 0001 comment - typo

you can optionally filter rows that does not satisfy a WHERE condition

typo: does/does

~~

2. Patch 0001 comment - typo

The WHERE clause should probably contain only columns that are part of
the primary key or that are covered by REPLICA IDENTITY. Otherwise,
and DELETEs won't be replicated.

typo: "Otherwise, and DELETEs" ??

~~

3. Patch 0001 comment - typo and clarification

Typo: "contains partitioned table" -> "contains a partitioned table"

Also, perhaps the text "or the partitioned table row filter." should
say "or the root partitioned table row filter." to disambiguate the
case where there are more levels of partitions like A->B->C. e.g. What
filter does C use?

~~

4. src/backend/catalog/pg_publication.c - misleading names

Leaving this parameter name as "targetrel" seems a bit misleading now
in the function code. Maybe this should be called something like "pri"
which is consistent with other places where you have declared
PublicationRelationInfo.

Also, consider declaring some local variables so that the patch may
have less impact on existing code. e.g.
Oid relid = pri->relid
Relation *targetrel = relationinfo->relation

~~

5. src/backend/commands/publicationcmds.c - simplify code

Consider writing that code more simply as just:

rels = OpenTableList(stmt->tables, stmt->tableAction == DEFELEM_DROP);

~~

6. src/backend/commands/publicationcmds.c - bug?

- CloseTableList(rels);

Is this a potential bug? When you called OpenTableList the 2nd param
was maybe true/false, so is it correct to be unconditionally false
here? I am not sure.

~~

7. src/backend/commands/publicationcmds.c - OpenTableList function comment.

I am not sure about this. Should that comment instead say "indicates
if it has a Relation (true) or PublicationTable (false)"?

~~

8. src/backend/commands/publicationcmds.c - OpenTableList

+ if (is_drop)
+ {

For some reason it feels kind of clunky to me for this function to be
processing the list differently according to the 2nd param. e.g. the
name "is_drop" seems quite unrelated to the function code, and more to
do with where it was called from. Sorry, I don't have any better ideas
for improvement atm.

~~

9. src/backend/commands/publicationcmds.c - OpenTableList bug?

I felt maybe this is a possible bug here because there seems no code
explicitly assigning the whereClause = NULL  if "is_drop" is true so
maybe it can have a garbage value which could cause problems later.
Maybe this is fixed by using palloc0.

Same thing is 2x in this function.

~~

10. src/backend/commands/publicationcmds.c - CloseTableList function comment

Probably the meaning of "is_drop" should be described in this function comment.

~~

11. src/backend/replication/pgoutput/pgoutput.c - get_rel_sync_entry signature.

I see that this function signature is modified but I did not see how
this parameter refactoring is actually related to the RowFilter patch.
Perhaps I am mistaken, but IIUC this only changes the relid =
RelationGetRelid(rel); to be done inside this function instead of
being done outside by the callers.

It impacts other code like in pgoutput_truncate:

@@ -689,12 +865,11 @@ pgoutput_truncate(LogicalDecodingContext *ctx,
ReorderBufferTXN *txn,
  for (i = 0; i < nrelations; i++)
  {
  Relation relation = relations[i];
- Oid relid = RelationGetRelid(relation);

@@ -704,10 +879,10 @@ pgoutput_truncate(LogicalDecodingContext *ctx,

- relids[nrelids++] = relid;
+ relids[nrelids++] = relentry->relid;
  maybe_send_schema(ctx, txn, change, relation, relentry);
  }
So maybe this is a good refactor or maybe not, but I felt this should
not be included as part of the RowFilter patch unless it is really
necessary.

~~

12. src/backend/replication/pgoutput/pgoutput.c - missing function comments

The static functions create_estate_for_relation and
pgoutput_row_filter_prepare_expr probably should be commented.

~~

13. src/backend/replication/pgoutput/pgoutput.c -
pgoutput_row_filter_prepare_expr function name

+static ExprState *pgoutput_row_filter_prepare_expr(Node *rfnode,
EState *estate);

This function has an unfortunate name with the word "prepare" in it. I
wonder if a different name can be found for this function to avoid any
confusion with pgoutput functions (coming soon) which are related to
the two-phase commit "prepare".

~~

14. src/bin/psql/describe.c

Because the where-clause value already has enclosing parentheses so
using " WHERE (%s)" seems overkill here. e.g. you can see the effect
in your src/test/regress/expected/publication.out file. I think this
should be changed to " WHERE %s" to give better output.

~~

15. src/include/catalog/pg_publication.h - new typedef

The new PublicationRelationInfo should also be added
src/tools/pgindent/typedefs.list

~~

16. src/include/nodes/parsenodes.h - new typedef

The new PublicationTable should also be added src/tools/pgindent/typedefs.list

~~

17. sql/publication.sql - show more output

+CREATE PUBLICATION testpub5 FOR TABLE testpub_rf_tbl1,
testpub_rf_tbl2 WHERE (c <> 'test' AND d < 5);
+RESET client_min_messages;
+ALTER PUBLICATION testpub5 ADD TABLE testpub_rf_tbl3 WHERE (e > 1000
AND e < 2000);
+ALTER PUBLICATION testpub5 DROP TABLE testpub_rf_tbl2;
+-- remove testpub_rf_tbl1 and add testpub_rf_tbl3 again (another
WHERE expression)
+ALTER PUBLICATION testpub5 SET TABLE testpub_rf_tbl3 WHERE (e > 300
AND e < 500);
+-- fail - functions disallowed
+ALTER PUBLICATION testpub5 ADD TABLE testpub_rf_tbl4 WHERE (length(g) < 6);
+-- fail - WHERE not allowed in DROP
+ALTER PUBLICATION testpub5 DROP TABLE testpub_rf_tbl3 WHERE (e < 27);
+\dRp+ testpub5

I felt that it would be better to have a "\dRp+ testpub5" after each
of the valid ALTER PUBLICATION steps to show the intermediate results
also; not just the final one at the end.

(PSA a temp patch showing what I mean by this review comment)

~~

18. src/test/subscription/t/020_row_filter.pl - rename file

I think this file should be renamed to 021_row_filter.pl as there is
already an 020 TAP test present.

~~

19. src/test/subscription/t/020_row_filter.pl - test comments

AFAIK the test cases are all OK, but it was really quite hard to
review these TAP tests to try to determine what the expected results
should be.

I found that I had to add my own comments to the file so I could
understand what was going on, so I think the TAP test can benefit lots
from having many more comments describing how the expected results are
determined.

Also, the filtering does not take place at the INSERT but really it is
affected only by which publications the subscription has subscribed
to. So I thought some of the existing comments (although correct) are
misplaced.

(PSA a temp patch showing what I mean by this review comment)

~~~

20. src/test/subscription/t/020_row_filter.pl - missing test case?

There are some partition tests, but I did not see any test that was
like 3 levels deep like A->B->C, so I was not sure if there is any
case C would ever make use of the filter of its parent B, or would it
only use the filter of the root A?

~~

21. src/test/subscription/t/020_row_filter.pl - missing test case?

If the same table is in multiple publications they can each have a row
filter. And a subscription might subscribe to some but not all of
those publications. I think this scenario is only partly tested.

e.g.
pub_1 has tableX with RowFilter1
pub_2 has tableX with RowFilter2

Then sub_12 subscribes to pub_1, pub_2
This is already tested in your TAP test (I think) and it makes sure
both filters are applied

But if there was also
pub_3 has tableX with RowFilter3

Then sub_12 still should only be checking the filtered RowFilter1 AND
RowFilter2 (but NOT row RowFilter3). I think this scenario is not
tested.

////////////////

POC PATCH FOR PLAN CACHE
========================

PSA a POC patch for a plan cache which gets used inside the
pgoutput_row_filter function instead of calling prepare for every row.
I think this is implementing something like Andes was suggesting a
while back [1].

Measurements with/without this plan cache:

Time spent processing within the pgoutput_row_filter function
- Data was captured using the same technique as the
0002-Measure-row-filter-overhead.patch.
- Inserted 1000 rows, sampled data for the first 100 times in this function.
not cached: average ~ 28.48 us
cached: average ~ 9.75 us

Replication times:
- Using tables and row filters same as in Onder's commands_to_test_perf.sql [2]
100K rows - not cached: ~ 42sec, 43sec, 44sec
100K rows - cached: ~ 41sec, 42sec, 42 sec.

There does seem to be a tiny gain achieved by having the plan cache,
but I think the gain might be a lot less than what people were
expecting.

Unless there are millions of rows the speedup may be barely noticeable.

--------
[1] https://www.postgresql.org/message-id/20210128022032.eq2qqc6zxkqn5syt%40alap3.anarazel.de
[2] https://www.postgresql.org/message-id/CACawEhW_iMnY9XK2tEb1ig%2BA%2BgKeB4cxdJcxMsoCU0SaKPExxg%40mail.gmail.com

Kind Regards,
Peter Smith.
Fujitsu Australia

---

From: Greg Nancarrow <[email-masked]>
Date: Mon, 5 Jul 2021 13:14:02 +1000

> intended to be committed. They are available for test-only purposes.

I have some review comments on the "Row filter for logical replication" patch:

(1) Suggested update to patch comment:
(There are some missing words and things which could be better expressed)

This feature adds row filtering for publication tables.
When a publication is defined or modified, rows that don't satisfy a WHERE
clause may be optionally filtered out. This allows a database or set of
tables to be partially replicated. The row filter is per table, which allows
different row filters to be defined for different tables. A new row filter
can be added simply by specifying a WHERE clause after the table name.
The WHERE clause must be enclosed by parentheses.

The WHERE clause should probably contain only columns that are part of the
primary key or that are covered by REPLICA IDENTITY. Otherwise, any DELETEs
won't be replicated. DELETE uses the old row version (that is limited to
primary key or REPLICA IDENTITY) to evaluate the row filter. INSERT and UPDATE
use the new row version to evaluate the row filter, hence, you can use any
column. If the row filter evaluates to NULL, it returns false. For simplicity,
functions are not allowed; that could possibly be addressed in a future patch.

If your publication contains a partitioned table, the publication parameter
publish_via_partition_root determines if it uses the partition row filter (if
the parameter is false, the default) or the root partitioned table row filter.

(2) Some inconsistent error message wording:

Currently:
err = _("cannot use subquery in publication WHERE expression");

Suggest changing it to:
err = _("subqueries are not allowed in publication WHERE expressions");

Other examples from the patch:
err = _("aggregate functions are not allowed in publication WHERE expressions");
err = _("grouping operations are not allowed in publication WHERE expressions");
err = _("window functions are not allowed in publication WHERE expressions");
errmsg("functions are not allowed in publication WHERE expressions"),
err = _("set-returning functions are not allowed in publication WHERE
expressions");

(3) The current code still allows arbitrary code execution, e.g. via a
user-defined operator:

e.g.
publisher:

CREATE OR REPLACE FUNCTION myop(left_arg INTEGER, right_arg INTEGER)
RETURNS BOOL AS
$$
BEGIN
  RAISE NOTICE 'I can do anything here!';
  RETURN left_arg > right_arg;
 END;
$$ LANGUAGE PLPGSQL VOLATILE;

CREATE OPERATOR >>>> (
  PROCEDURE = myop,
  LEFTARG = INTEGER,
  RIGHTARG = INTEGER
);

CREATE PUBLICATION tap_pub FOR TABLE test_tab WHERE (a >>>> 5);

subscriber:
CREATE SUBSCRIPTION tap_sub CONNECTION 'host=localhost dbname=test_pub
application_name=tap_sub' PUBLICATION tap_pub;

Perhaps add the following after the existing shell error-check in make_op():

/* User-defined operators are not allowed in publication WHERE clauses */
if (pstate->p_expr_kind == EXPR_KIND_PUBLICATION_WHERE && opform->oid
>= FirstNormalObjectId)
    ereport(ERROR,
    (errcode(ERRCODE_FEATURE_NOT_SUPPORTED),
    errmsg("user-defined operators are not allowed in publication
WHERE expressions"),
    parser_errposition(pstate, location)));

Also, I believe it's also allowing user-defined CASTs (so could add a
similar check to above in transformTypeCast()).
Ideally, it would be preferable to validate/check publication WHERE
expressions in one central place, rather than scattered all over the
place, but that might be easier said than done.
You need to update the patch comment accordingly.

(4) src/backend/replication/pgoutput/pgoutput.c
pgoutput_change()

The 3 added calls to pgoutput_row_filter() are returning from
pgoutput_change(), if false is returned, but instead they should break
from the switch, otherwise cleanup code is missed. This is surely a
bug.

e.g.
(3 similar cases of this)

should be:

Regards,
Greg Nancarrow
Fujitsu Australia

---

From: Greg Nancarrow <[email-masked]>
Date: Wed, 7 Jul 2021 15:24:24 +1000

Hi,

I did some testing of the performance of the row filtering, in the
case of the publisher INSERTing 100,000 rows, using a similar test
setup and timing as previously used in the “commands_to_perf_test.sql“
script posted by Önder Kalacı.

I found that with the call to ExecInitExtraTupleSlot() in
pgoutput_row_filter(), then the performance of pgoutput_row_filter()
degrades considerably over the 100,000 invocations, and on my system
it took about 43 seconds to filter and send to the subscriber.
However, by caching the tuple table slot in RelationSyncEntry, this
duration can be dramatically reduced by 38+ seconds.
A further improvement can be made using this in combination with
Peter's plan cache (v16-0004).
I've attached a patch for this, which relies on the latest v16-0001
and v16-0004 patches posted by Peter Smith (noting that v16-0001 is
identical to your previously-posted 0001 patch).
Also attached is a graph (created by Peter Smith – thanks!) detailing
the performance improvement.

Regards,
Greg Nancarrow
Fujitsu Australia

---

From: "Euler Taveira" <[email-masked]>
Date: Wed, 07 Jul 2021 21:33:41 -0300

> the performance improvement.
Greg, I like your suggestion and already integrate it (I replaced
ExecAllocTableSlot() with MakeSingleTupleTableSlot() because we don't need the
List). I'm still working on a new version to integrate all suggestions that you
and Peter did. I have a similar code to Peter's plan cache and I'm working on
merging both ideas together. I'm done for today but I'll continue tomorrow.

---

From: Greg Nancarrow <[email-masked]>
Date: Thu, 8 Jul 2021 11:03:34 +1000

On Thu, Jul 8, 2021 at 10:34 AM Euler Taveira <[email-masked]> wrote:
> List).

Yes I agree, I found the same thing, it's not needed.

>I'm still working on a new version to integrate all suggestions that you

I also realised that my 0005 patch wasn't handling RelationSyncEntry
invalidation, so I've updated it.
For completeness, I'm posting the complete patch set with the updates,
so you can look at it and compare with yours, and also it'll keep the
cfbot happy until you post your updated patch.

Regards,
Greg Nancarrow
Fujitsu Australia

---

From: "Euler Taveira" <[email-masked]>
Date: Sun, 11 Jul 2021 16:39:24 -0300

> comments and some patches.
Peter, thanks for your detailed review. Comments are inline.

> you can optionally filter rows that does not satisfy a WHERE condition
Fixed.

> typo: "Otherwise, and DELETEs" ??
Fixed.

> Typo: "contains partitioned table" -> "contains a partitioned table"
Fixed.

> case where there are more levels of partitions like A->B->C. e.g. What
I agree it can be confusing. BTW, CREATE PUBLICATION does not mention that the
root partitioned table is used. We should improve that sentence too.

> Relation *targetrel = relationinfo->relation
Done.

> rels = OpenTableList(stmt->tables, stmt->tableAction == DEFELEM_DROP);
It is not a common pattern to use an expression as a function argument in
Postgres. I prefer to use a variable with a suggestive name.

Good catch.

> if it has a Relation (true) or PublicationTable (false)"?
Fixed.

>8

> do with where it was called from. Sorry, I don't have any better ideas
My suggestion is to rename it to "pub_drop_table".

>8

> Maybe this is fixed by using palloc0.
Fixed.

>8

> Probably the meaning of "is_drop" should be described in this function comment.
Done.

> being done outside by the callers.
It is not critical for this patch so I removed it.

> pgoutput_row_filter_prepare_expr probably should be commented.
Done.

> the two-phase commit "prepare".
The word "prepare" is related to the executor context. The function name
contains "row_filter" that is sufficient to distinguish it from any other
function whose context is "prepare". I replaced "prepare" with "init".

> should be changed to " WHERE %s" to give better output.
Peter E suggested that extra parenthesis be added. See 0005 [1].

> src/tools/pgindent/typedefs.list
Patches usually don't update typedefs.list. Check src/tools/pgindent/README.

> The new PublicationTable should also be added src/tools/pgindent/typedefs.list
Idem.

> also; not just the final one at the end.
Done.

> already an 020 TAP test present.
Done.

> review these TAP tests to try to determine what the expected results
I included your comments but heavily changed it.

> only use the filter of the root A?
I didn't include it yet. There is an issue with initial synchronization and
partitioned table when you set publish_via_partition_root. I'll start another
thread for this issue.

> those publications. I think this scenario is only partly tested.
8<
> RowFilter2 (but NOT row RowFilter3). I think this scenario is not
I added a new publication tap_pub_not_used to cover this case.

> POC PATCH FOR PLAN CACHE
I also had a WIP patch for it (that's very similar to your patch) so I merged
it.

This cache mechanism consists of caching ExprState and avoid calling
pgoutput_row_filter_init_expr() for every single row. Greg N suggested in
another email that tuple table slot should also be cached to avoid a few cycles
too. It is also included in this new patch.

> but I think the gain might be a lot less than what people were
I did another measure using as baseline the previous patch (v16).

without cache (v16)
---------------------------

mean:           1.46 us
stddev:         2.13 us
median:         1.39 us
min-max:        [0.69 .. 1456.69] us
percentile(99): 3.15 us
mode:           0.91 us

with cache (v18)
-----------------------

mean:           0.63 us
stddev:         1.07 us
median:         0.55 us
min-max:        [0.29 .. 844.87] us
percentile(99): 1.38 us
mode:           0.41 us

It represents -57%. It is a really good optimization for just a few extra lines
of code.

[1] https://www.postgresql.org/message-id/57373e8b-1264-cd37-404e-8edbcf7884cc%40enterprisedb.com

--
Euler Taveira
EDB   https://www.enterprisedb.com/
From 6176df1880ba690a7e5d550b7ee8c533dd33712e Mon Sep 17 00:00:00 2001
From: Euler Taveira <[email-masked]>
Date: Mon, 18 Jan 2021 12:07:51 -0300
Subject: [PATCH v18 1/2] Row filter for logical replication

This feature adds row filter for publication tables. When a publication
is defined or modified, rows that don't satisfy a WHERE clause may be
optionally filtered out. This allows a database or set of tables to be
partially replicated. The row filter is per table, which allows
different row filters to be defined for different tables. A new row
filter can be added simply by specifying a WHERE clause after the table
name. The WHERE clause must be enclosed by parentheses.

If your publication contains a partitioned table, the publication parameter
publish_via_partition_root determines if it uses the partition row filter (if
the parameter is false, the default) or the root partitioned table row filter.
---
 doc/src/sgml/catalogs.sgml                  |   8 +
 doc/src/sgml/ref/alter_publication.sgml     |  11 +-
 doc/src/sgml/ref/create_publication.sgml    |  32 ++-
 doc/src/sgml/ref/create_subscription.sgml   |  11 +-
 src/backend/catalog/pg_publication.c        |  42 ++-
 src/backend/commands/publicationcmds.c      | 112 +++++---
 src/backend/parser/gram.y                   |  24 +-
 src/backend/parser/parse_agg.c              |  10 +
 src/backend/parser/parse_expr.c             |  13 +
 src/backend/parser/parse_func.c             |   3 +
 src/backend/parser/parse_oper.c             |   7 +
 src/backend/replication/logical/tablesync.c |  94 +++++-
 src/backend/replication/pgoutput/pgoutput.c | 268 +++++++++++++++++-
 src/bin/pg_dump/pg_dump.c                   |  24 +-
 src/bin/pg_dump/pg_dump.h                   |   1 +
 src/bin/psql/describe.c                     |  15 +-
 src/include/catalog/pg_publication.h        |   9 +-
 src/include/catalog/pg_publication_rel.h    |   6 +
 src/include/nodes/nodes.h                   |   1 +
 src/include/nodes/parsenodes.h              |  11 +-
 src/include/parser/parse_node.h             |   1 +
 src/test/regress/expected/publication.out   |  71 +++++
 src/test/regress/sql/publication.sql        |  32 +++
 src/test/subscription/t/021_row_filter.pl   | 298 ++++++++++++++++++++
 24 files changed, 1024 insertions(+), 80 deletions(-)
 create mode 100644 src/test/subscription/t/021_row_filter.pl

  <refsynopsisdiv>
 <synopsis>
 
      <para>
 
          <para>
 
+  <para>
 
  <refsect1>
 
+  <para>
index 86e415af89..a15f00f637 100644
@@ -141,21 +144,27 @@ pg_relation_is_publishable(PG_FUNCTION_ARGS)
+publication_add_relation(Oid pubid, PublicationRelationInfo *pri,
+	Relation	targetrel = pri->relation;

+	relid = RelationGetRelid(targetrel);
+
 	/*
 	 * Check for duplicates. Note that this does not really prevent
 	 * duplicates, it's here just to provide nicer error message in common
@@ -177,6 +186,23 @@ publication_add_relation(Oid pubid, Relation targetrel,
 
 	check_publication_add_relation(targetrel);
 
+	/* Set up a pstate to parse with */
+	pstate = make_parsestate(NULL);
+	pstate->p_sourcetext = nodeToString(pri->whereClause);
+
+	nsitem = addRangeTableEntryForRelation(pstate, targetrel,
+										   AccessShareLock,
+										   NULL, false, false);
+	addNSItemToQuery(pstate, nsitem, false, true, true);
+
+	whereclause = transformWhereClause(pstate,
+									   copyObject(pri->whereClause),
+									   EXPR_KIND_PUBLICATION_WHERE,
+									   "PUBLICATION");
+
+	/* Fix up collation information */
+	assign_expr_collations(pstate, whereclause);
+
 	/* Form a tuple. */
 	memset(values, 0, sizeof(values));
 	memset(nulls, false, sizeof(nulls));
@@ -189,6 +215,12 @@ publication_add_relation(Oid pubid, Relation targetrel,
 	values[Anum_pg_publication_rel_prrelid - 1] =
 		ObjectIdGetDatum(relid);

 	/* Insert tuple into catalog. */
@@ -205,6 +237,12 @@ publication_add_relation(Oid pubid, Relation targetrel,
 	ObjectAddressSet(referenced, RelationRelationId, relid);
 	recordDependencyOn(&myself, &referenced, DEPENDENCY_AUTO);

index 95c253c8e0..12adcf1f36 100644
@@ -48,7 +48,7 @@
 
-static List *OpenTableList(List *tables);
+static List *OpenTableList(List *tables, bool pub_drop_table);
 static void CloseTableList(List *rels);
 static void PublicationAddTables(Oid pubid, List *rels, bool if_not_exists,
 								 AlterPublicationStmt *stmt);
@@ -232,7 +232,7 @@ CreatePublication(CreatePublicationStmt *stmt)

-		rels = OpenTableList(stmt->tables);
+		rels = OpenTableList(stmt->tables, false);
 		PublicationAddTables(puboid, rels, true, NULL);
 		CloseTableList(rels);
 	}
@@ -361,6 +361,7 @@ AlterPublicationTables(AlterPublicationStmt *stmt, Relation rel,
 	List	   *rels = NIL;
 	Form_pg_publication pubform = (Form_pg_publication) GETSTRUCT(tup);
 	Oid			pubid = pubform->oid;
+	bool		isdrop = (stmt->tableAction == DEFELEM_DROP);
 
 	/* Check that user is allowed to manipulate the publication tables. */
 	if (pubform->puballtables)
@@ -372,7 +373,7 @@ AlterPublicationTables(AlterPublicationStmt *stmt, Relation rel,

-	rels = OpenTableList(stmt->tables);
+	rels = OpenTableList(stmt->tables, isdrop);
 
 	if (stmt->tableAction == DEFELEM_ADD)
 		PublicationAddTables(pubid, rels, false, stmt);
@@ -385,31 +386,24 @@ AlterPublicationTables(AlterPublicationStmt *stmt, Relation rel,
 		List	   *delrels = NIL;
 		ListCell   *oldlc;

-			foreach(newlc, rels)
-			{
 
 		/* And drop them. */
@@ -500,26 +494,44 @@ RemovePublicationRelById(Oid proid)
 
 /*
  * Open relations specified by a RangeVar list.
+ *
+ * Publication node can have a different list element, hence, pub_drop_table
+ * indicates if it has a Relation (true) or PublicationTable (false).
+ *
  * The returned tables are locked in ShareUpdateExclusiveLock mode in order to
  * add them to a publication.
  */
 static List *
-OpenTableList(List *tables)
+OpenTableList(List *tables, bool pub_drop_table)
 {
 	List	   *relids = NIL;
 	List	   *rels = NIL;
 	ListCell   *lc;
+	PublicationRelationInfo *pri;
 
 	/*
 
+		if (pub_drop_table)
+		{
 
@@ -538,8 +550,14 @@ OpenTableList(List *tables)
 			table_close(rel, ShareUpdateExclusiveLock);
 			continue;
 		}
-
-		rels = lappend(rels, rel);
+		pri = palloc(sizeof(PublicationRelationInfo));
+		pri->relid = myrelid;
+		pri->relation = rel;
+		if (pub_drop_table)
+			pri->whereClause = NULL;
+		else
+			pri->whereClause = t->whereClause;
+		rels = lappend(rels, pri);
 		relids = lappend_oid(relids, myrelid);
 
 		/*
@@ -572,7 +590,15 @@ OpenTableList(List *tables)
 
+				if (pub_drop_table)
+					pri->whereClause = NULL;
@@ -585,6 +611,9 @@ OpenTableList(List *tables)
 
 /*
  * Close all relations in the list.
+ *
+ * Publication node can have a different list element, hence, pub_drop_table
+ * indicates if it has a Relation (true) or PublicationTable (false).
  */
 static void
 CloseTableList(List *rels)
@@ -593,10 +622,12 @@ CloseTableList(List *rels)
 
 	foreach(lc, rels)
 	{

 /*
@@ -612,15 +643,15 @@ PublicationAddTables(Oid pubid, List *rels, bool if_not_exists,
 
 	foreach(lc, rels)
 	{

-		obj = publication_add_relation(pubid, rel, if_not_exists);
+		obj = publication_add_relation(pubid, pri, if_not_exists);
 		if (stmt)
 		{
 			EventTriggerCollectSimpleCommand(obj, InvalidObjectAddress,
@@ -644,11 +675,10 @@ PublicationDropTables(Oid pubid, List *rels, bool missing_ok)
 
 	foreach(lc, rels)
 	{
 
@@ -658,7 +688,7 @@ PublicationDropTables(Oid pubid, List *rels, bool missing_ok)

 
 
 publication_for_tables:
 
+publication_table_list:
index 24268eb502..8fb953b54f 100644
 
 			break;

 			/*

 
 			/*
 
 			/*
 
 			/*
 			 * There is intentionally no default: case here, so that the
diff --git a/src/backend/parser/parse_oper.c b/src/backend/parser/parse_oper.c
index bc34a23afc..29f8835ce1 100644
--- a/src/backend/parser/parse_oper.c
+++ b/src/backend/parser/parse_oper.c
@@ -718,6 +718,13 @@ make_op(ParseState *pstate, List *opname, Node *ltree, Node *rtree,
 											opform->oprright)),
 				 parser_errposition(pstate, location)));
 
+	/* Check it's not a custom operator for publication WHERE expressions */
+	if (pstate->p_expr_kind == EXPR_KIND_PUBLICATION_WHERE && opform->oid >= FirstNormalObjectId)
+		ereport(ERROR,
+				(errcode(ERRCODE_FEATURE_NOT_SUPPORTED),
+				 errmsg("user-defined operators are not allowed in publication WHERE expressions"),
+				 parser_errposition(pstate, location)));
+
 	/* Do typecasting and build the expression tree */
 	if (ltree == NULL)
 	{
diff --git a/src/backend/replication/logical/tablesync.c b/src/backend/replication/logical/tablesync.c
index 682c107e74..980826a502 100644
--- a/src/backend/replication/logical/tablesync.c
+++ b/src/backend/replication/logical/tablesync.c
@@ -691,19 +691,23 @@ copy_read_data(void *outbuf, int minread, int maxread)
 
 /*
 
 	lrel->nspname = nspname;
 	lrel->relname = relname;

 

index abd5217ab1..08c018a300 100644
@@ -13,18 +13,27 @@
 #include "postgres.h"

@@ -99,6 +108,9 @@ typedef struct RelationSyncEntry
 
 	bool		replicate_valid;
 	PublicationActions pubactions;
+	List	   *qual;				/* row filter */
+	List	   *exprstate;			/* ExprState for row filter */
+	TupleTableSlot	*scantuple;		/* tuple table slot for row filter */
 
 	/*
 	 * OID of the relation to publish changes as.  For a partition, this may
@@ -122,7 +134,7 @@ static HTAB *RelationSyncCache = NULL;
 
 static void init_rel_sync_cache(MemoryContext decoding_context);
 static void cleanup_rel_sync_cache(TransactionId xid, bool is_commit);
-static RelationSyncEntry *get_rel_sync_entry(PGOutputData *data, Oid relid);
+static RelationSyncEntry *get_rel_sync_entry(PGOutputData *data, Relation relation);
 static void rel_sync_cache_relation_cb(Datum arg, Oid relid);
 static void rel_sync_cache_publication_cb(Datum arg, int cacheid,
 										  uint32 hashvalue);
@@ -131,6 +143,13 @@ static void set_schema_sent_in_streamed_txn(RelationSyncEntry *entry,
 static bool get_schema_sent_in_streamed_txn(RelationSyncEntry *entry,
 											TransactionId xid);
 
+/* row filter routines */
+static EState *create_estate_for_relation(Relation rel);
+static ExprState *pgoutput_row_filter_init_expr(Node *rfnode);
+static bool pgoutput_row_filter_exec_expr(ExprState *state, ExprContext *econtext);
+static bool pgoutput_row_filter(Relation relation, HeapTuple oldtuple,
+								HeapTuple newtuple, RelationSyncEntry *entry);
+
 /*
  * Specify output plugin callbacks
  */
@@ -520,6 +539,154 @@ send_relation_and_attrs(Relation relation, TransactionId xid,
 	OutputPluginWrite(ctx, false);
 }
 
+/*
+ * Executor state preparation for evaluation of row filter expressions for the
+ * specified relation.
+ */
+static EState *
+create_estate_for_relation(Relation rel)
+{
+	EState			*estate;
+	RangeTblEntry	*rte;
+
+	estate = CreateExecutorState();
+
+	rte = makeNode(RangeTblEntry);
+	rte->rtekind = RTE_RELATION;
+	rte->relid = RelationGetRelid(rel);
+	rte->relkind = rel->rd_rel->relkind;
+	rte->rellockmode = AccessShareLock;
+	ExecInitRangeTable(estate, list_make1(rte));
+
+	estate->es_output_cid = GetCurrentCommandId(false);
+
+	return estate;
+}
+
+/*
+ * Initialize for row filter expression execution.
+ */
+static ExprState *
+pgoutput_row_filter_init_expr(Node *rfnode)
+{
+	ExprState	   *exprstate;
+	Oid				exprtype;
+	Expr		   *expr;
+	MemoryContext	oldctx;
+
+	/* Prepare expression for execution */
+	exprtype = exprType(rfnode);
+	expr = (Expr *) coerce_to_target_type(NULL, rfnode, exprtype, BOOLOID, -1, COERCION_ASSIGNMENT, COERCE_IMPLICIT_CAST, -1);
+
+	if (expr == NULL)
+		ereport(ERROR,
+				(errcode(ERRCODE_CANNOT_COERCE),
+				 errmsg("row filter returns type %s that cannot be coerced to the expected type %s",
+						format_type_be(exprtype),
+						format_type_be(BOOLOID)),
+				 errhint("You will need to rewrite the row filter.")));
+
+	/*
+	 * Cache ExprState using CacheMemoryContext. This is the same code as
+	 * ExecPrepareExpr() but it is not used because it doesn't use an EState.
+	 * It should probably be another function in the executor to handle the
+	 * execution outside a normal Plan tree context.
+	 */
+	oldctx = MemoryContextSwitchTo(CacheMemoryContext);
+	expr = expression_planner(expr);
+	exprstate = ExecInitExpr(expr, NULL);
+	MemoryContextSwitchTo(oldctx);
+
+	return exprstate;
+}
+
+/*
+ * Evaluates row filter.
+ *
+ * If the row filter evaluates to NULL, it is taken as false i.e. the change
+ * isn't replicated.
+ */
+static bool
+pgoutput_row_filter_exec_expr(ExprState *state, ExprContext *econtext)
+{
+	Datum		ret;
+	bool		isnull;
+
+	Assert(state != NULL);
+
+	ret = ExecEvalExprSwitchContext(state, econtext, &isnull);
+
+	elog(DEBUG3, "row filter evaluates to %s (isnull: %s)",
+		 DatumGetBool(ret) ? "true" : "false",
+		 isnull ? "true" : "false");
+
+	if (isnull)
+		return false;
+
+	return DatumGetBool(ret);
+}
+
+/*
+ * Change is checked against the row filter, if any.
+ *
+ * If it returns true, the change is replicated, otherwise, it is not.
+ */
+static bool
+pgoutput_row_filter(Relation relation, HeapTuple oldtuple, HeapTuple newtuple, RelationSyncEntry *entry)
+{
+	EState	   *estate;
+	ExprContext *ecxt;
+	ListCell   *lc;
+	bool		result = true;
+
+	/* Bail out if there is no row filter */
+	if (entry->qual == NIL)
+		return true;
+
+	elog(DEBUG3, "table \"%s.%s\" has row filter",
+		 get_namespace_name(get_rel_namespace(RelationGetRelid(relation))),
+		 get_rel_name(relation->rd_id));
+
+	PushActiveSnapshot(GetTransactionSnapshot());
+
+	estate = create_estate_for_relation(relation);
+
+	if (entry->scantuple == NULL)
+		elog(DEBUG1, "entry->scantuple is null");
+
+	/* Prepare context per tuple */
+	ecxt = GetPerTupleExprContext(estate);
+	ecxt->ecxt_scantuple = entry->scantuple;
+
+	ExecStoreHeapTuple(newtuple ? newtuple : oldtuple, ecxt->ecxt_scantuple, false);
+
+	/*
+	 * If the subscription has multiple publications and the same table has a
+	 * different row filter in these publications, all row filters must be
+	 * matched in order to replicate this change.
+	 */
+	foreach(lc, entry->exprstate)
+	{
+		ExprState  *exprstate = (ExprState *) lfirst(lc);
+
+		/* Evaluates row filter */
+		result = pgoutput_row_filter_exec_expr(exprstate, ecxt);
+
+		elog(DEBUG3, "row filter %smatched", result ? "" : "not ");
+
+		/* If the tuple does not match one of the row filters, bail out */
+		if (!result)
+			break;
+	}
+
+	/* Cleanup allocated resources */
+	ResetExprContext(ecxt);
+	FreeExecutorState(estate);
+	PopActiveSnapshot();
+
+	return result;
+}
+
 /*
  * Sends the decoded DML over wire.
  *
@@ -547,7 +714,7 @@ pgoutput_change(LogicalDecodingContext *ctx, ReorderBufferTXN *txn,
 	if (in_streaming)
 		xid = change->txn->xid;

 	/* First check the table filter */
 	switch (change->action)
@@ -571,8 +738,6 @@ pgoutput_change(LogicalDecodingContext *ctx, ReorderBufferTXN *txn,
 	/* Avoid leaking memory by using and resetting our own context */
 	old = MemoryContextSwitchTo(data->context);
 
-	maybe_send_schema(ctx, txn, change, relation, relentry);
-
 	/* Send the data */
 	switch (change->action)
 	{
@@ -580,6 +745,16 @@ pgoutput_change(LogicalDecodingContext *ctx, ReorderBufferTXN *txn,
 			{
 				HeapTuple	tuple = &change->data.tp.newtuple->tuple;
 
+				if (!pgoutput_row_filter(relation, NULL, tuple, relentry))
@@ -603,6 +778,12 @@ pgoutput_change(LogicalDecodingContext *ctx, ReorderBufferTXN *txn,
 
+				/* Check row filter. */
+				if (!pgoutput_row_filter(relation, oldtuple, newtuple, relentry))
+					break;
+
+				maybe_send_schema(ctx, txn, change, relation, relentry);
+
 				/* Switch relation if publishing via root. */
 				if (relentry->publish_as_relid != RelationGetRelid(relation))
 				{
@@ -631,6 +812,12 @@ pgoutput_change(LogicalDecodingContext *ctx, ReorderBufferTXN *txn,
 			{
 				HeapTuple	oldtuple = &change->data.tp.oldtuple->tuple;
 
+				/* Check row filter. */
+				if (!pgoutput_row_filter(relation, oldtuple, NULL, relentry))
+					break;
+
+				maybe_send_schema(ctx, txn, change, relation, relentry);
+
 				/* Switch relation if publishing via root. */
 				if (relentry->publish_as_relid != RelationGetRelid(relation))
 				{
@@ -694,7 +881,7 @@ pgoutput_truncate(LogicalDecodingContext *ctx, ReorderBufferTXN *txn,
 		if (!is_publishable_relation(relation))
 			continue;

 		if (!relentry->pubactions.pubtruncate)
 			continue;
@@ -1005,9 +1192,10 @@ set_schema_sent_in_streamed_txn(RelationSyncEntry *entry, TransactionId xid)
  * when publishing.
  */
 static RelationSyncEntry *
-get_rel_sync_entry(PGOutputData *data, Oid relid)
+get_rel_sync_entry(PGOutputData *data, Relation relation)
 {
 	RelationSyncEntry *entry;
+	Oid			relid = RelationGetRelid(relation);
 	bool		am_partition = get_rel_relispartition(relid);
 	char		relkind = get_rel_relkind(relid);
 	bool		found;
@@ -1030,6 +1218,9 @@ get_rel_sync_entry(PGOutputData *data, Oid relid)
 		entry->replicate_valid = false;
 		entry->pubactions.pubinsert = entry->pubactions.pubupdate =
 			entry->pubactions.pubdelete = entry->pubactions.pubtruncate = false;
+		entry->qual = NIL;
+		entry->scantuple = NULL;
+		entry->exprstate = NIL;
 		entry->publish_as_relid = InvalidOid;
 		entry->map = NULL;		/* will be set by maybe_send_schema() if
 								 * needed */
@@ -1041,6 +1232,7 @@ get_rel_sync_entry(PGOutputData *data, Oid relid)
 		List	   *pubids = GetRelationPublications(relid);
 		ListCell   *lc;
 		Oid			publish_as_relid = relid;
+		TupleDesc	tupdesc;
 
 		/* Reload publications if needed before use. */
 		if (!publications_valid)
@@ -1054,6 +1246,23 @@ get_rel_sync_entry(PGOutputData *data, Oid relid)
 			publications_valid = true;
 		}
 
+		/* Release tuple table slot */
+		if (entry->scantuple != NULL)
+		{
+			ExecDropSingleTupleTableSlot(entry->scantuple);
+			entry->scantuple = NULL;
+
+			elog(DEBUG1, "get_rel_sync_entry: free entry->scantuple");
+		}
+
+		/* create a tuple table slot for row filter */
+		tupdesc = RelationGetDescr(relation);
+		oldctx = MemoryContextSwitchTo(CacheMemoryContext);
+		entry->scantuple = MakeSingleTupleTableSlot(tupdesc, &TTSOpsHeapTuple);
+		MemoryContextSwitchTo(oldctx);
+
+		elog(DEBUG1, "get_rel_sync_entry: allocate entry->scantuple");
+
 		/*
 		 * Build publication cache. We can't use one provided by relcache as
 		 * relcache considers all publications given relation is in, but here
@@ -1063,6 +1272,9 @@ get_rel_sync_entry(PGOutputData *data, Oid relid)
 		{
 			Publication *pub = lfirst(lc);
 			bool		publish = false;
+			HeapTuple	rftuple;
+			Datum		rfdatum;
+			bool		rfisnull;
 
 			if (pub->alltables)
 			{
@@ -1122,9 +1334,34 @@ get_rel_sync_entry(PGOutputData *data, Oid relid)
 				entry->pubactions.pubtruncate |= pub->pubactions.pubtruncate;
 			}
 
+					exprstate = pgoutput_row_filter_init_expr(rfnode);
+					entry->exprstate = lappend(entry->exprstate, exprstate);
 
 		list_free(pubids);
@@ -1242,6 +1479,7 @@ rel_sync_cache_publication_cb(Datum arg, int cacheid, uint32 hashvalue)
 {
 	HASH_SEQ_STATUS status;
 	RelationSyncEntry *entry;
+	MemoryContext oldctx;
 
 	/*
 	 * We can get here if the plugin was used in SQL interface as the
@@ -1251,6 +1489,8 @@ rel_sync_cache_publication_cb(Datum arg, int cacheid, uint32 hashvalue)
 	if (RelationSyncCache == NULL)
 		return;
 
@@ -1268,5 +1508,15 @@ rel_sync_cache_publication_cb(Datum arg, int cacheid, uint32 hashvalue)
+		if (entry->exprstate != NIL)
+			list_free_deep(entry->exprstate);
+		entry->exprstate = NIL;

 	ntups = PQntuples(res);

 
 
 	/*
 
 /*
 
index f332bad4d4..2703b9c3fe 100644

 extern bool is_publishable_relation(Relation rel);
-extern ObjectAddress publication_add_relation(Oid pubid, Relation targetrel,
+extern ObjectAddress publication_add_relation(Oid pubid, PublicationRelationInfo *pri,
 											  bool if_not_exists);

 /* ----------------

 
 	/*

 

diff --git a/src/test/regress/expected/publication.out b/src/test/regress/expected/publication.out
index 63d6ab7a4e..444f8344bc 100644
--- a/src/test/regress/expected/publication.out
+++ b/src/test/regress/expected/publication.out
@@ -156,6 +156,77 @@ Tables:
 
 DROP TABLE testpub_parted1;
 DROP PUBLICATION testpub_forparted, testpub_forparted1;
+CREATE TABLE testpub_rf_tbl1 (a integer, b text);
+CREATE TABLE testpub_rf_tbl2 (c text, d integer);
+CREATE TABLE testpub_rf_tbl3 (e integer);
+CREATE TABLE testpub_rf_tbl4 (g text);
+SET client_min_messages = 'ERROR';
+CREATE PUBLICATION testpub5 FOR TABLE testpub_rf_tbl1, testpub_rf_tbl2 WHERE (c <> 'test' AND d < 5);
+RESET client_min_messages;
+\dRp+ testpub5
+                                    Publication testpub5
+          Owner           | All tables | Inserts | Updates | Deletes | Truncates | Via root 
+--------------------------+------------+---------+---------+---------+-----------+----------
+ regress_publication_user | f          | t       | t       | t       | t         | f
+Tables:
+    "public.testpub_rf_tbl1"
+    "public.testpub_rf_tbl2" WHERE (((c <> 'test'::text) AND (d < 5)))
+
+ALTER PUBLICATION testpub5 ADD TABLE testpub_rf_tbl3 WHERE (e > 1000 AND e < 2000);
+\dRp+ testpub5
+                                    Publication testpub5
+          Owner           | All tables | Inserts | Updates | Deletes | Truncates | Via root 
+--------------------------+------------+---------+---------+---------+-----------+----------
+ regress_publication_user | f          | t       | t       | t       | t         | f
+Tables:
+    "public.testpub_rf_tbl1"
+    "public.testpub_rf_tbl2" WHERE (((c <> 'test'::text) AND (d < 5)))
+    "public.testpub_rf_tbl3" WHERE (((e > 1000) AND (e < 2000)))
+
+ALTER PUBLICATION testpub5 DROP TABLE testpub_rf_tbl2;
+\dRp+ testpub5
+                                    Publication testpub5
+          Owner           | All tables | Inserts | Updates | Deletes | Truncates | Via root 
+--------------------------+------------+---------+---------+---------+-----------+----------
+ regress_publication_user | f          | t       | t       | t       | t         | f
+Tables:
+    "public.testpub_rf_tbl1"
+    "public.testpub_rf_tbl3" WHERE (((e > 1000) AND (e < 2000)))
+
+-- remove testpub_rf_tbl1 and add testpub_rf_tbl3 again (another WHERE expression)
+ALTER PUBLICATION testpub5 SET TABLE testpub_rf_tbl3 WHERE (e > 300 AND e < 500);
+\dRp+ testpub5
+                                    Publication testpub5
+          Owner           | All tables | Inserts | Updates | Deletes | Truncates | Via root 
+--------------------------+------------+---------+---------+---------+-----------+----------
+ regress_publication_user | f          | t       | t       | t       | t         | f
+Tables:
+    "public.testpub_rf_tbl3" WHERE (((e > 300) AND (e < 500)))
+
+-- fail - functions disallowed
+ALTER PUBLICATION testpub5 ADD TABLE testpub_rf_tbl4 WHERE (length(g) < 6);
+ERROR:  functions are not allowed in publication WHERE expressions
+LINE 1: ...ICATION testpub5 ADD TABLE testpub_rf_tbl4 WHERE (length(g) ...
+                                                             ^
+-- fail - user-defined operators disallowed
+CREATE FUNCTION testpub_rf_func(integer, integer) RETURNS boolean AS $$ SELECT hashint4($1) > $2 $$ LANGUAGE SQL;
+CREATE OPERATOR =#> (PROCEDURE = testpub_rf_func, LEFTARG = integer, RIGHTARG = integer);
+CREATE PUBLICATION testpub6 FOR TABLE testpub_rf_tbl3 WHERE (e =#> 27);
+ERROR:  user-defined operators are not allowed in publication WHERE expressions
+LINE 1: ...ICATION testpub6 FOR TABLE testpub_rf_tbl3 WHERE (e =#> 27);
+                                                               ^
+-- fail - WHERE not allowed in DROP
+ALTER PUBLICATION testpub5 DROP TABLE testpub_rf_tbl3 WHERE (e < 27);
+ERROR:  syntax error at or near "WHERE"
+LINE 1: ...R PUBLICATION testpub5 DROP TABLE testpub_rf_tbl3 WHERE (e <...
+                                                             ^
+DROP TABLE testpub_rf_tbl1;
+DROP TABLE testpub_rf_tbl2;
+DROP TABLE testpub_rf_tbl3;
+DROP TABLE testpub_rf_tbl4;
+DROP PUBLICATION testpub5;
+DROP OPERATOR =#>(integer, integer);
+DROP FUNCTION testpub_rf_func(integer, integer);
 -- fail - view
 CREATE PUBLICATION testpub_fortbl FOR TABLE testpub_view;
 ERROR:  "testpub_view" is not a table
diff --git a/src/test/regress/sql/publication.sql b/src/test/regress/sql/publication.sql
index d844075368..b1606cce7e 100644
--- a/src/test/regress/sql/publication.sql
+++ b/src/test/regress/sql/publication.sql
@@ -93,6 +93,38 @@ ALTER PUBLICATION testpub_forparted SET (publish_via_partition_root = true);
 DROP TABLE testpub_parted1;
 DROP PUBLICATION testpub_forparted, testpub_forparted1;
 
+CREATE TABLE testpub_rf_tbl1 (a integer, b text);
+CREATE TABLE testpub_rf_tbl2 (c text, d integer);
+CREATE TABLE testpub_rf_tbl3 (e integer);
+CREATE TABLE testpub_rf_tbl4 (g text);
+SET client_min_messages = 'ERROR';
+CREATE PUBLICATION testpub5 FOR TABLE testpub_rf_tbl1, testpub_rf_tbl2 WHERE (c <> 'test' AND d < 5);
+RESET client_min_messages;
+\dRp+ testpub5
+ALTER PUBLICATION testpub5 ADD TABLE testpub_rf_tbl3 WHERE (e > 1000 AND e < 2000);
+\dRp+ testpub5
+ALTER PUBLICATION testpub5 DROP TABLE testpub_rf_tbl2;
+\dRp+ testpub5
+-- remove testpub_rf_tbl1 and add testpub_rf_tbl3 again (another WHERE expression)
+ALTER PUBLICATION testpub5 SET TABLE testpub_rf_tbl3 WHERE (e > 300 AND e < 500);
+\dRp+ testpub5
+-- fail - functions disallowed
+ALTER PUBLICATION testpub5 ADD TABLE testpub_rf_tbl4 WHERE (length(g) < 6);
+-- fail - user-defined operators disallowed
+CREATE FUNCTION testpub_rf_func(integer, integer) RETURNS boolean AS $$ SELECT hashint4($1) > $2 $$ LANGUAGE SQL;
+CREATE OPERATOR =#> (PROCEDURE = testpub_rf_func, LEFTARG = integer, RIGHTARG = integer);
+CREATE PUBLICATION testpub6 FOR TABLE testpub_rf_tbl3 WHERE (e =#> 27);
+-- fail - WHERE not allowed in DROP
+ALTER PUBLICATION testpub5 DROP TABLE testpub_rf_tbl3 WHERE (e < 27);
+
+DROP TABLE testpub_rf_tbl1;
+DROP TABLE testpub_rf_tbl2;
+DROP TABLE testpub_rf_tbl3;
+DROP TABLE testpub_rf_tbl4;
+DROP PUBLICATION testpub5;
+DROP OPERATOR =#>(integer, integer);
+DROP FUNCTION testpub_rf_func(integer, integer);
+
 -- fail - view
 CREATE PUBLICATION testpub_fortbl FOR TABLE testpub_view;
 SET client_min_messages = 'ERROR';
diff --git a/src/test/subscription/t/021_row_filter.pl b/src/test/subscription/t/021_row_filter.pl
new file mode 100644
index 0000000000..0f6d2f0128
--- /dev/null
+++ b/src/test/subscription/t/021_row_filter.pl
@@ -0,0 +1,298 @@
+# Test logical replication behavior with row filtering
+use strict;
+use warnings;
+use PostgresNode;
+use TestLib;
+use Test::More tests => 7;
+
+# create publisher node
+my $node_publisher = get_new_node('publisher');
+$node_publisher->init(allows_streaming => 'logical');
+$node_publisher->start;
+
+# create subscriber node
+my $node_subscriber = get_new_node('subscriber');
+$node_subscriber->init(allows_streaming => 'logical');
+$node_subscriber->start;
+
+# setup structure on publisher
+$node_publisher->safe_psql('postgres',
+	"CREATE TABLE tab_rowfilter_1 (a int primary key, b text)");
+$node_publisher->safe_psql('postgres',
+	"CREATE TABLE tab_rowfilter_2 (c int primary key)");
+$node_publisher->safe_psql('postgres',
+	"CREATE TABLE tab_rowfilter_3 (a int primary key, b boolean)");
+$node_publisher->safe_psql('postgres',
+	"CREATE TABLE tab_rowfilter_partitioned (a int primary key, b integer) PARTITION BY RANGE(a)"
+);
+$node_publisher->safe_psql('postgres',
+	"CREATE TABLE tab_rowfilter_less_10k (LIKE tab_rowfilter_partitioned)");
+$node_publisher->safe_psql('postgres',
+	"ALTER TABLE tab_rowfilter_partitioned ATTACH PARTITION tab_rowfilter_less_10k FOR VALUES FROM (MINVALUE) TO (10000)"
+);
+$node_publisher->safe_psql('postgres',
+	"CREATE TABLE tab_rowfilter_greater_10k (LIKE tab_rowfilter_partitioned)"
+);
+$node_publisher->safe_psql('postgres',
+	"ALTER TABLE tab_rowfilter_partitioned ATTACH PARTITION tab_rowfilter_greater_10k FOR VALUES FROM (10000) TO (MAXVALUE)"
+);
+
+# setup structure on subscriber
+$node_subscriber->safe_psql('postgres',
+	"CREATE TABLE tab_rowfilter_1 (a int primary key, b text)");
+$node_subscriber->safe_psql('postgres',
+	"CREATE TABLE tab_rowfilter_2 (c int primary key)");
+$node_subscriber->safe_psql('postgres',
+	"CREATE TABLE tab_rowfilter_3 (a int primary key, b boolean)");
+$node_subscriber->safe_psql('postgres',
+	"CREATE TABLE tab_rowfilter_partitioned (a int primary key, b integer) PARTITION BY RANGE(a)"
+);
+$node_subscriber->safe_psql('postgres',
+	"CREATE TABLE tab_rowfilter_less_10k (LIKE tab_rowfilter_partitioned)");
+$node_subscriber->safe_psql('postgres',
+	"ALTER TABLE tab_rowfilter_partitioned ATTACH PARTITION tab_rowfilter_less_10k FOR VALUES FROM (MINVALUE) TO (10000)"
+);
+$node_subscriber->safe_psql('postgres',
+	"CREATE TABLE tab_rowfilter_greater_10k (LIKE tab_rowfilter_partitioned)"
+);
+$node_subscriber->safe_psql('postgres',
+	"ALTER TABLE tab_rowfilter_partitioned ATTACH PARTITION tab_rowfilter_greater_10k FOR VALUES FROM (10000) TO (MAXVALUE)"
+);
+
+# setup logical replication
+$node_publisher->safe_psql('postgres',
+	"CREATE PUBLICATION tap_pub_1 FOR TABLE tab_rowfilter_1 WHERE (a > 1000 AND b <> 'filtered')"
+);
+
+$node_publisher->safe_psql('postgres',
+	"ALTER PUBLICATION tap_pub_1 ADD TABLE tab_rowfilter_2 WHERE (c % 7 = 0)"
+);
+
+$node_publisher->safe_psql('postgres',
+	"ALTER PUBLICATION tap_pub_1 SET TABLE tab_rowfilter_1 WHERE (a > 1000 AND b <> 'filtered'), tab_rowfilter_2 WHERE (c % 2 = 0), tab_rowfilter_3"
+);
+
+$node_publisher->safe_psql('postgres',
+	"CREATE PUBLICATION tap_pub_2 FOR TABLE tab_rowfilter_2 WHERE (c % 3 = 0)"
+);
+
+$node_publisher->safe_psql('postgres',
+	"CREATE PUBLICATION tap_pub_3 FOR TABLE tab_rowfilter_partitioned WHERE (a < 5000)"
+);
+$node_publisher->safe_psql('postgres',
+	"ALTER PUBLICATION tap_pub_3 ADD TABLE tab_rowfilter_less_10k WHERE (a < 6000)"
+);
+$node_publisher->safe_psql('postgres',
+	"CREATE PUBLICATION tap_pub_not_used FOR TABLE tab_rowfilter_1 WHERE (a < 0)"
+);
+
+#
+# The following INSERTs are executed before the CREATE SUBSCRIPTION, so these
+# SQL commands are for testing the initial data copy using logical replication.
+#
+$node_publisher->safe_psql('postgres',
+	"INSERT INTO tab_rowfilter_1 (a, b) VALUES (1, 'not replicated')");
+$node_publisher->safe_psql('postgres',
+	"INSERT INTO tab_rowfilter_1 (a, b) VALUES (1500, 'filtered')");
+$node_publisher->safe_psql('postgres',
+	"INSERT INTO tab_rowfilter_1 (a, b) VALUES (1980, 'not filtered')");
+$node_publisher->safe_psql('postgres',
+	"INSERT INTO tab_rowfilter_1 (a, b) SELECT x, 'test ' || x FROM generate_series(990,1002) x"
+);
+$node_publisher->safe_psql('postgres',
+	"INSERT INTO tab_rowfilter_2 (c) SELECT generate_series(1, 20)");
+$node_publisher->safe_psql('postgres',
+	"INSERT INTO tab_rowfilter_3 (a, b) SELECT x, (x % 3 = 0) FROM generate_series(1, 10) x");
+
+# insert data into partitioned table and directly on the partition
+$node_publisher->safe_psql('postgres',
+	"INSERT INTO tab_rowfilter_partitioned (a, b) VALUES(1, 100),(7000, 101),(15000, 102),(5500, 300)");
+$node_publisher->safe_psql('postgres',
+	"INSERT INTO tab_rowfilter_less_10k (a, b) VALUES(2, 200),(6005, 201)");
+$node_publisher->safe_psql('postgres',
+	"INSERT INTO tab_rowfilter_greater_10k (a, b) VALUES(16000, 103)");
+
+my $publisher_connstr = $node_publisher->connstr . ' dbname=postgres';
+my $appname           = 'tap_sub';
+$node_subscriber->safe_psql('postgres',
+	"CREATE SUBSCRIPTION tap_sub CONNECTION '$publisher_connstr application_name=$appname' PUBLICATION tap_pub_1, tap_pub_2, tap_pub_3"
+);
+
+$node_publisher->wait_for_catchup($appname);
+
+# wait for initial table synchronization to finish
+my $synced_query =
+  "SELECT count(1) = 0 FROM pg_subscription_rel WHERE srsubstate NOT IN ('r', 's');";
+$node_subscriber->poll_query_until('postgres', $synced_query)
+  or die "Timed out while waiting for subscriber to synchronize data";
+
+# Check expected replicated rows for tab_rowfilter_1
+# tap_pub_1 filter is: (a > 1000 AND b <> 'filtered')
+# - INSERT (1, 'not replicated')   NO, because a is not > 1000
+# - INSERT (1500, 'filtered')      NO, because b == 'filtered'
+# - INSERT (1980, 'not filtered')  YES
+# - generate_series(990,1002)      YES, only for 1001,1002 because a > 1000
+#
+my $result =
+  $node_subscriber->safe_psql('postgres',
+	"SELECT a, b FROM tab_rowfilter_1 ORDER BY 1, 2");
+is( $result, qq(1001|test 1001
+1002|test 1002
+1980|not filtered), 'check initial data copy from table tab_rowfilter_1');
+
+# Check expected replicated rows for tab_rowfilter_2
+# tap_pub_1 filter is: (c % 2 = 0)
+# tap_pub_2 filter is: (c % 3 = 0)
+# When there are multiple publications for the same table, all filter
+# expressions should succeed. In this case, rows are replicated if c value is
+# divided by 2 AND 3 (6, 12, 18).
+#
+$result =
+  $node_subscriber->safe_psql('postgres',
+	"SELECT count(c), min(c), max(c) FROM tab_rowfilter_2");
+is($result, qq(3|6|18), 'check initial data copy from table tab_rowfilter_2');
+
+# Check expected replicated rows for tab_rowfilter_3
+# There is no filter. 10 rows are inserted, so 10 rows are replicated.
+$result =
+  $node_subscriber->safe_psql('postgres',
+	"SELECT count(a) FROM tab_rowfilter_3");
+is($result, qq(10), 'check initial data copy from table tab_rowfilter_3');
+
+# Check expected replicated rows for partitions
+# publication option publish_via_partition_root is false so use the row filter
+# from a partition
+# tab_rowfilter_partitioned filter: (a < 5000)
+# tab_rowfilter_less_10k filter:    (a < 6000)
+# tab_rowfilter_greater_10k filter: no filter
+#
+# INSERT into tab_rowfilter_partitioned:
+# - INSERT (1,100)       YES, because 1 < 6000
+# - INSERT (7000, 101)   NO,  because 7000 is not < 6000
+# - INSERT (15000, 102)  YES, because tab_rowfilter_greater_10k has no filter
+# - INSERT (5500, 300)   YES, because 5500 < 6000
+#
+# INSERT directly into tab_rowfilter_less_10k:
+# - INSERT (2, 200)      YES, because 2 < 6000
+# - INSERT (6005, 201)   NO, because 6005 is not < 6000
+#
+# INSERT directly into tab_rowfilter_greater_10k:
+# - INSERT (16000, 103)  YES, because tab_rowfilter_greater_10k has no filter
+#
+$result =
+  $node_subscriber->safe_psql('postgres',
+	"SELECT a, b FROM tab_rowfilter_less_10k ORDER BY 1, 2");
+is($result, qq(1|100
+2|200
+5500|300), 'check initial data copy from partition tab_rowfilter_less_10k');
+
+$result =
+  $node_subscriber->safe_psql('postgres',
+	"SELECT a, b FROM tab_rowfilter_greater_10k ORDER BY 1, 2");
+is($result, qq(15000|102
+16000|103), 'check initial data copy from partition tab_rowfilter_greater_10k');
+
+# The following commands are executed after CREATE SUBSCRIPTION, so these SQL
+# commands are for testing normal logical replication behavior.
+#
+# test row filter (INSERT, UPDATE, DELETE)
+$node_publisher->safe_psql('postgres',
+	"INSERT INTO tab_rowfilter_1 (a, b) VALUES (800, 'test 800')");
+$node_publisher->safe_psql('postgres',
+	"INSERT INTO tab_rowfilter_1 (a, b) VALUES (1600, 'test 1600')");
+$node_publisher->safe_psql('postgres',
+	"INSERT INTO tab_rowfilter_1 (a, b) VALUES (1601, 'test 1601')");
+$node_publisher->safe_psql('postgres',
+	"INSERT INTO tab_rowfilter_1 (a, b) VALUES (1700, 'test 1700')");
+$node_publisher->safe_psql('postgres',
+	"UPDATE tab_rowfilter_1 SET b = NULL WHERE a = 1600");
+$node_publisher->safe_psql('postgres',
+	"UPDATE tab_rowfilter_1 SET b = 'test 1601 updated' WHERE a = 1601");
+$node_publisher->safe_psql('postgres',
+	"DELETE FROM tab_rowfilter_1 WHERE a = 1700");
+
+$node_publisher->wait_for_catchup($appname);
+
+# Check expected replicated rows for tab_rowfilter_1
+# tap_pub_1 filter is: (a > 1000 AND b <> 'filtered')
+#
+# - 1001, 1002, 1980 already exist from initial data copy
+# - INSERT (800, 'test 800')   NO, because 800 is not > 1000
+# - INSERT (1600, 'test 1600') YES, because 1600 > 1000 and 'test 1600' <> 'filtered'
+# - INSERT (1601, 'test 1601') YES, because 1601 > 1000 and 'test 1601' <> 'filtered'
+# - INSERT (1700, 'test 1700') YES, because 1700 > 1000 and 'test 1700' <> 'filtered'
+# - UPDATE (1600, NULL)        NO, row filter evaluates to false because NULL is not <> 'filtered'
+# - UPDATE (1601, 'test 1601 updated') YES, because 1601 > 1000 and 'test 1601 updated' <> 'filtered'
+# - DELETE (1700)              NO, row filter contains column b that is not part of
+# the PK or REPLICA IDENTITY and old tuple contains b = NULL, hence, row filter
+# evaluates to false
+#
+$result =
+  $node_subscriber->safe_psql('postgres',
+	"SELECT a, b FROM tab_rowfilter_1 ORDER BY 1, 2");
+is($result, qq(1001|test 1001
+1002|test 1002
+1600|test 1600
+1601|test 1601 updated
+1700|test 1700
+1980|not filtered), 'check replicated rows to table tab_rowfilter_1');
+
+# Publish using root partitioned table
+# Use a different partitioned table layout (exercise publish_via_partition_root)
+$node_publisher->safe_psql('postgres',
+	"ALTER PUBLICATION tap_pub_3 SET (publish_via_partition_root = true)");
+$node_subscriber->safe_psql('postgres',
+	"TRUNCATE TABLE tab_rowfilter_partitioned");
+$node_subscriber->safe_psql('postgres',
+	"ALTER SUBSCRIPTION tap_sub REFRESH PUBLICATION WITH (copy_data = true)");
+$node_publisher->safe_psql('postgres',
+	"INSERT INTO tab_rowfilter_partitioned (a, b) VALUES(4000, 400),(4001, 401),(4002, 402)");
+$node_publisher->safe_psql('postgres',
+	"INSERT INTO tab_rowfilter_less_10k (a, b) VALUES(4500, 450)");
+$node_publisher->safe_psql('postgres',
+	"INSERT INTO tab_rowfilter_less_10k (a, b) VALUES(5600, 123)");
+$node_publisher->safe_psql('postgres',
+	"INSERT INTO tab_rowfilter_greater_10k (a, b) VALUES(14000, 1950)");
+$node_publisher->safe_psql('postgres',
+	"UPDATE tab_rowfilter_less_10k SET b = 30 WHERE a = 4001");
+$node_publisher->safe_psql('postgres',
+	"DELETE FROM tab_rowfilter_less_10k WHERE a = 4002");
+
+$node_publisher->wait_for_catchup($appname);
+
+# Check expected replicated rows for partitions
+# publication option publish_via_partition_root is true so use the row filter
+# from the root partitioned table
+# tab_rowfilter_partitioned filter: (a < 5000)
+# tab_rowfilter_less_10k filter:    (a < 6000)
+# tab_rowfilter_greater_10k filter: no filter
+#
+# After TRUNCATE, REFRESH PUBLICATION, the initial data copy will apply the
+# partitioned table row filter.
+# - INSERT (1, 100)      YES, 1 < 5000
+# - INSERT (7000, 101)   NO, 7000 is not < 5000
+# - INSERT (15000, 102)  NO, 15000 is not < 5000
+# - INSERT (5500, 300)   NO, 5500 is not < 5000
+# - INSERT (2, 200)      YES, 2 < 5000
+# - INSERT (6005, 201)   NO, 6005 is not < 5000
+# - INSERT (16000, 103)  NO, 16000 is not < 5000
+#
+# Execute SQL commands after initial data copy for testing the logical
+# replication behavior.
+# - INSERT (4000, 400)    YES, 4000 < 5000
+# - INSERT (4001, 401)    YES, 4001 < 5000
+# - INSERT (4002, 402)    YES, 4002 < 5000
+# - INSERT (4500, 450)    YES, 4500 < 5000
+# - INSERT (5600, 123)    NO, 5600 is not < 5000
+# - INSERT (14000, 1950)  NO, 16000 is not < 5000
+$result =
+  $node_subscriber->safe_psql('postgres',
+	"SELECT a, b FROM tab_rowfilter_partitioned ORDER BY 1, 2");
+is( $result, qq(1|100
+2|200
+4000|400
+4001|30
+4500|450), 'check publish_via_partition_root behavior');
+
+$node_subscriber->stop('fast');
+$node_publisher->stop('fast');
-- 
2.20.1

From b0d60791f06908d2dc118ff7f5dc669c91062913 Mon Sep 17 00:00:00 2001
From: Euler Taveira <[email-masked]>
Date: Sun, 31 Jan 2021 20:48:43 -0300
Subject: [PATCH v18 2/2] Measure row filter overhead

---
 src/backend/replication/pgoutput/pgoutput.c | 16 +++++++++-------
 1 file changed, 9 insertions(+), 7 deletions(-)

diff --git a/src/backend/replication/pgoutput/pgoutput.c b/src/backend/replication/pgoutput/pgoutput.c
index 08c018a300..5700a3306b 100644
--- a/src/backend/replication/pgoutput/pgoutput.c
+++ b/src/backend/replication/pgoutput/pgoutput.c
@@ -638,6 +638,8 @@ pgoutput_row_filter(Relation relation, HeapTuple oldtuple, HeapTuple newtuple, R
 	ExprContext *ecxt;
 	ListCell   *lc;
 	bool		result = true;
+	instr_time	start_time;
+	instr_time	end_time;
 
 	/* Bail out if there is no row filter */
 	if (entry->qual == NIL)
@@ -647,13 +649,12 @@ pgoutput_row_filter(Relation relation, HeapTuple oldtuple, HeapTuple newtuple, R
 		 get_namespace_name(get_rel_namespace(RelationGetRelid(relation))),
 		 get_rel_name(relation->rd_id));

 
-	if (entry->scantuple == NULL)
-		elog(DEBUG1, "entry->scantuple is null");
-
 	/* Prepare context per tuple */
 	ecxt = GetPerTupleExprContext(estate);
 	ecxt->ecxt_scantuple = entry->scantuple;
@@ -684,6 +685,11 @@ pgoutput_row_filter(Relation relation, HeapTuple oldtuple, HeapTuple newtuple, R
 	FreeExecutorState(estate);
 	PopActiveSnapshot();

@@ -1251,8 +1257,6 @@ get_rel_sync_entry(PGOutputData *data, Relation relation)
 		{
 			ExecDropSingleTupleTableSlot(entry->scantuple);
 			entry->scantuple = NULL;
-
-			elog(DEBUG1, "get_rel_sync_entry: free entry->scantuple");
 		}
 
 		/* create a tuple table slot for row filter */
@@ -1261,8 +1265,6 @@ get_rel_sync_entry(PGOutputData *data, Relation relation)
 		entry->scantuple = MakeSingleTupleTableSlot(tupdesc, &TTSOpsHeapTuple);
 		MemoryContextSwitchTo(oldctx);
 
-		elog(DEBUG1, "get_rel_sync_entry: allocate entry->scantuple");
-
 		/*
 		 * Build publication cache. We can't use one provided by relcache as
 		 * relcache considers all publications given relation is in, but here
-- 
2.20.1

---

From: "Euler Taveira" <[email-masked]>
Date: Sun, 11 Jul 2021 16:48:26 -0300

> (There are some missing words and things which could be better expressed)
I incorporated all your wording suggestions.

> err = _("subqueries are not allowed in publication WHERE expressions");
The same expression "cannot use subquery in ..." is used in the other switch
cases. If you think this message can be improved, I suggest that you submit a
separate patch to change all sentences.

> err = _("set-returning functions are not allowed in publication WHERE
This is a different function. I just followed the same wording from similar
sentences around it.

> (3) The current code still allows arbitrary code execution, e.g. via a
I fixed it in v18.

>     parser_errposition(pstate, location)));
I'm still working on a way to accept built-in functions but while we don't have
it, let's forbid custom operators too.

> You need to update the patch comment accordingly.
I forgot to mention it in the patch I sent a few minutes ago. I'm not sure we
need to mention every error condition (specially one that will be rarely used).

Fixed.

In summary, v18 contains

* Peter Smith's review
* Greg Nancarrow's review
* cache ExprState
* cache TupleTableSlot
* forbid custom operators
* various fixes

---

From: Tomas Vondra <[email-masked]>
Date: Mon, 12 Jul 2021 01:09:21 +0200

Hi,

I took a look at this patch, which seems to be in CF since 2018. I have 
only some basic comments and observations at this point:

1) alter_publication.sgml

I think "expression is executed" sounds a bit strange, perhaps 
"evaluated" would be better?

2) create_publication.sgml

Why is the patch changing publish_via_partition_root docs? That seems 
like a rather unrelated bit.

    The <literal>WHERE</literal> clause should probably contain only
    columns that are part of the primary key or be covered by
    <literal>REPLICA ...

I'm not sure what exactly is this trying to say. What does "should 
probably ..." mean in practice for the users? Does that mean something 
bad will happen for other columns, or what? I'm sure this wording will 
be quite confusing for users.

It may also be unclear whether the condition is evaluated on the old or 
new row, so perhaps add an example illustrating that & more detailed 
comment, or something. E.g. what will happen with

	UPDATE departments SET active = false WHERE active;

3) publication_add_relation

Does this need to build the parse state even for whereClause == NULL?

4) AlterPublicationTables

I wonder if this new reworked code might have issues with subscriptions 
containing many tables, but I haven't tried.

5) OpenTableList

I really dislike that the list can have two different node types 
(Relation and PublicationTable). In principle we don't actually need the 
extra flag, we can simply check the node type directly by IsA() and act 
based on that. However, I think it'd be better to just use a single node 
type from all places.

I don't see why not to set whereClause every time, I don't think the 
extra if saves anything, it's just a bit more complex.

5) CloseTableList

The comment about node types seems pointless, this function has no flag 
and the element type does not matter.

6) parse_agg.c

    ... are not allowed in publication WHERE expressions

I think all similar cases use "WHERE conditions" instead.

7) transformExprRecurse

The check at the beginning seems rather awkward / misplaced - it's way 
too specific for this location (there are no other p_expr_kind 
references in this function). Wouldn't transformFuncCall (or maybe 
ParseFuncOrColumn) be a more appropriate place?

Initially I was wondering why not to allow function calls in WHERE 
conditions, but I see that was discussed in the past as problematic. But 
that reminds me that I don't see any docs describing what expressions 
are allowed in WHERE conditions - maybe we should explicitly list what 
expressions are allowed?

8) pgoutput.c

I have not reviewed this in detail yet, but there seems to be something 
wrong because `make check-world` fails in subscription/010_truncate.pl 
after hitting an assert  (backtrace attached) during "START_REPLICATION 
SLOT" in get_rel_sync_entry in this code:

     /* Release tuple table slot */
     if (entry->scantuple != NULL)
     {
         ExecDropSingleTupleTableSlot(entry->scantuple);
         entry->scantuple = NULL;
     }

So there seems to be something wrong with how the slot is created.

regards

-- 
Tomas Vondra
EnterpriseDB: http://www.enterprisedb.com
The Enterprise PostgreSQL Company

---

From: "Euler Taveira" <[email-masked]>
Date: Sun, 11 Jul 2021 20:30:24 -0300

> with cache (v18)
cfbot seems to be unhappy with v18 on some of the hosts. Cirrus/FreeBSD failed
in the test 010_truncate. It also failed in a Cirrus/Linux box. I failed to
reproduce in my local FreeBSD box. Since it passes appveyor and Cirrus/macos,
it could probably be a transient issue.

$ uname -a
FreeBSD freebsd12 12.2-RELEASE FreeBSD 12.2-RELEASE r366954 GENERIC  amd64
$ PROVE_TESTS="t/010_truncate.pl" gmake check
gmake -C ../../../src/backend generated-headers
gmake[1]: Entering directory '/usr/home/euler/pglr-row-filter-v17/src/backend'
gmake -C catalog distprep generated-header-symlinks
gmake[2]: Entering directory '/usr/home/euler/pglr-row-filter-v17/src/backend/catalog'
gmake[2]: Nothing to be done for 'distprep'.
gmake[2]: Nothing to be done for 'generated-header-symlinks'.
gmake[2]: Leaving directory '/usr/home/euler/pglr-row-filter-v17/src/backend/catalog'
gmake -C utils distprep generated-header-symlinks
gmake[2]: Entering directory '/usr/home/euler/pglr-row-filter-v17/src/backend/utils'
gmake[2]: Nothing to be done for 'distprep'.
gmake[2]: Nothing to be done for 'generated-header-symlinks'.
gmake[2]: Leaving directory '/usr/home/euler/pglr-row-filter-v17/src/backend/utils'
gmake[1]: Leaving directory '/usr/home/euler/pglr-row-filter-v17/src/backend'
rm -rf '/home/euler/pglr-row-filter-v17'/tmp_install
/bin/sh ../../../config/install-sh -c -d '/home/euler/pglr-row-filter-v17'/tmp_install/log
gmake -C '../../..' DESTDIR='/home/euler/pglr-row-filter-v17'/tmp_install install >'/home/euler/pglr-row-filter-v17'/tmp_install/log/install.log 2>&1
gmake -j1  checkprep >>'/home/euler/pglr-row-filter-v17'/tmp_install/log/install.log 2>&1
rm -rf '/usr/home/euler/pglr-row-filter-v17/src/test/subscription'/tmp_check
/bin/sh ../../../config/install-sh -c -d '/usr/home/euler/pglr-row-filter-v17/src/test/subscription'/tmp_check
cd . && TESTDIR='/usr/home/euler/pglr-row-filter-v17/src/test/subscription' PATH="/home/euler/pglr-row-filter-v17/tmp_install/home/euler/pgrf18/bin:$PATH" LD_LIBRARY_PATH="/home/euler/pglr-row-filter-v17/tmp_install/home/euler/pgrf18/lib" LD_LIBRARY_PATH_RPATH=1 PGPORT='69999' PG_REGRESS='/usr/home/euler/pglr-row-filter-v17/src/test/subscription/../../../src/test/regress/pg_regress' /usr/local/bin/prove -I ../../../src/test/perl/ -I .  t/010_truncate.pl
t/010_truncate.pl .. ok    
All tests successful.
Files=1, Tests=14,  5 wallclock secs ( 0.02 usr  0.00 sys +  1.09 cusr  0.99 csys =  2.10 CPU)
Result: PASS

---

From: Alvaro Herrera <[email-masked]>
Date: Sun, 11 Jul 2021 21:48:57 -0400

Hi

Andres complained about the safety of doing general expression
evaluation in pgoutput; that was first in 

https://postgr.es/m/[email-masked]
where he described a possible approach to handle it by restricting
expressions to have limited shape; and later in
http://postgr.es/m/[email-masked]

I was just scanning the patch trying to see if some sort of protection
had been added for this, but I couldn't find anything.  (Some functions
are under-commented, though).  So, is it there already, and if so what
is it?  And if it isn't, then I think it should definitely be put there
in some form.

Thanks

-- 
Álvaro Herrera              Valdivia, Chile  —  https://www.EnterpriseDB.com/

---

From: Greg Nancarrow <[email-masked]>
Date: Mon, 12 Jul 2021 12:15:04 +1000

> it could probably be a transient issue.

I don't think it's a transient issue.
I also get a test failure in subscription/010_truncate.pl when I run
"make check-world" with the v18 patches applied.
The problem can be avoided with the following change (to match what
was originally in my v17-0005 performance-improvement patch):

diff --git a/src/backend/replication/pgoutput/pgoutput.c
b/src/backend/replication/pgoutput/pgoutput.c
index 08c018a300..800bae400b 100644
--- a/src/backend/replication/pgoutput/pgoutput.c
+++ b/src/backend/replication/pgoutput/pgoutput.c
@@ -1256,8 +1256,8 @@ get_rel_sync_entry(PGOutputData *data, Relation relation)
         }

         /* create a tuple table slot for row filter */
-        tupdesc = RelationGetDescr(relation);
         oldctx = MemoryContextSwitchTo(CacheMemoryContext);
+        tupdesc = CreateTupleDescCopy(RelationGetDescr(relation));
         entry->scantuple = MakeSingleTupleTableSlot(tupdesc, &TTSOpsHeapTuple);
         MemoryContextSwitchTo(oldctx);

This creates a TupleDesc copy in CacheMemoryContext that is not
refcounted, so it side-steps the problem.
At this stage I am not sure why the original v18 patch code doesn't
work correctly for the TupleDesc refcounting here.
The TupleDesc refcount is zero when it's time to dealloc the tuple
slot (thus causing that Assert to fire), yet when the slot was
created, the TupleDesc refcount was incremented.- so it seems
something else has already decremented the refcount by the time it
comes to deallocate the slot. Perhaps there's an order-of-cleanup or
MemoryContext issue here or some buggy code somewhere, not sure yet.

Regards,
Greg Nancarrow
Fujitsu Australia

---

From: Amit Kapila <[email-masked]>
Date: Mon, 12 Jul 2021 10:16:06 +0530

On Mon, Jul 12, 2021 at 7:19 AM Alvaro Herrera <[email-masked]> wrote:
> is it?

I think the patch is trying to prohibit arbitrary expressions in the
WHERE clause via
transformWhereClause(..EXPR_KIND_PUBLICATION_WHERE..). You can notice
that at various places the expressions are prohibited via
EXPR_KIND_PUBLICATION_WHERE. I am not sure that the checks are correct
and sufficient but I think there is some attempt to do it. For
example, the below sort of ad-hoc check for func_call doesn't seem to
be good idea.

+ /* Functions are not allowed in publication WHERE clauses */
+ if (pstate->p_expr_kind == EXPR_KIND_PUBLICATION_WHERE &&
nodeTag(expr) == T_FuncCall)
+ ereport(ERROR,
+ (errcode(ERRCODE_FEATURE_NOT_SUPPORTED),
+ errmsg("functions are not allowed in publication WHERE expressions"),
+ parser_errposition(pstate, exprLocation(expr))));

Now, the other idea I had in mind was to traverse the WHERE clause
expression in publication_add_relation and identify if it contains
anything other than the ANDed list of 'foo.bar op constant'
expressions. OTOH, for index where clause expressions or policy check
expressions, we use a technique similar to what we have in the patch
to prohibit certain kinds of expressions.

Do you have any preference on how this should be addressed?

-- 
With Regards,
Amit Kapila.

---

From: Tomas Vondra <[email-masked]>
Date: Mon, 12 Jul 2021 11:31:23 +0200

> + parser_errposition(pstate, exprLocation(expr))));

Yes, I mentioned this bit of code in my review, although I was mostly 
wondering if this is the wrong place to make this check.

> Do you have any preference on how this should be addressed?

I don't think this is sufficient, because who knows where "op" comes 
from? It might be from an extension, in which case the problem pointed 
out by Petr Jelinek [1] would apply. OTOH I suppose we could allow 
expressions like (Var op Var), i.e. "a < b" or something like that. And 
then why not allow (a+b < c-10) and similar "more complex" expressions, 
as long as all the operators are built-in?

In terms of implementation, I think there are two basic options - either 
we can define a new "expression" type in gram.y, which would be a subset 
of a_expr etc. Or we can do it as some sort of expression walker, kinda 
like what the transform* functions do now.

regards

[1] 
https://www.postgresql.org/message-id/92e5587d-28b8-5849-2374-5ca3863256f1%402ndquadrant.com

---

From: Amit Kapila <[email-masked]>
Date: Mon, 12 Jul 2021 15:05:10 +0530

> I did another measure using as baseline the previous patch (v16).

Good improvement but I think it is better to measure the performance
by using synchronous_replication by setting the subscriber as
standby_synchronous_names, which will provide the overall saving of
time. We can probably see when the timings when no rows are filtered,
when 10% rows are filtered when 30% are filtered and so on.

I think the way caching has been done in the patch is a bit
inefficient. Basically, it always invalidates and rebuilds the
expressions even though some unrelated operation has happened on
publication. For example, say publication has initially table t1 with
rowfilter r1 for which we have cached the state. Now you altered
publication and added table t2, it will invalidate the entire state of
t1 as well. I think we can avoid that if we invalidate the rowfilter
related state only on relcache invalidation i.e in
rel_sync_cache_relation_cb and save it the very first time we prepare
the expression. In that case, we don't need to do it in advance when
preparing relsyncentry, this will have the additional advantage that
we won't spend cycles on preparing state unless it is required (for
truncate we won't require row_filtering, so it won't be prepared).

Few other things, I have noticed:
1.
I am seeing tupledesc leak by following below steps:
ERROR:  tupdesc reference 00000000008D7D18 is not owned by resource
owner TopTransaction
CONTEXT:  slot "tap_sub", output plugin "pgoutput", in the change
callback, associated LSN 0/170BD50

Publisher
CREATE TABLE tab_rowfilter_1 (a int primary key, b text);
CREATE PUBLICATION tap_pub_1 FOR TABLE tab_rowfilter_1 WHERE (a > 1000
AND b <> 'filtered');

Subscriber
CREATE TABLE tab_rowfilter_1 (a int primary key, b text);
CREATE SUBSCRIPTION tap_sub
         CONNECTION 'host=localhost port=5432 dbname=postgres'
        PUBLICATION tap_pub_1;

Publisher
INSERT INTO tab_rowfilter_1 (a, b) VALUES (1980, 'not filtered');
Alter table tab_rowfilter_1 drop column b cascade;
INSERT INTO tab_rowfilter_1 (a) VALUES (1982);

2.
postgres=# Alter table tab_rowfilter_1 alter column b set data type varchar;
ERROR:  unexpected object depending on column: publication of table
tab_rowfilter_1 in publication tap_pub_1

I think for this you need to change ATExecAlterColumnType to handle
the publication case.

-- 
With Regards,
Amit Kapila.

---

From: Tomas Vondra <[email-masked]>
Date: Mon, 12 Jul 2021 13:44:42 +0200

While looking at the other logrep patch [1] (column filtering) I noticed
Alvaro's comment regarding a new parsenode (PublicationTable) not having
read/out/equal/copy funcs. I'd bet the same thing applies here, so
perhaps see if the patch needs the same fix.

[1]
https://www.postgresql.org/message-id/202107062342.eq6htmp2wgp2%40alvherre.pgsql

---

From: "Euler Taveira" <[email-masked]>
Date: Mon, 12 Jul 2021 16:53:08 -0300

> perhaps see if the patch needs the same fix.
Good catch! I completely forgot about _copyPublicationTable() and
_equalPublicationTable().

---

From: Peter Smith <[email-masked]>
Date: Tue, 13 Jul 2021 13:25:48 +1000

Hi Euler,

Thanks for addressing my previous review comments.

I have reviewed the latest v18 patch. Below are some more review
comments and patches.

(The patches 0003,0004 are just examples of what is mentioned in my
comments; The patches 0001,0002 are there only to try to keep cfbot
green).

//////////

1. Commit comment - wording

"When a publication is defined or modified, rows that don't satisfy a
WHERE clause may be
optionally filtered out."

=>

I think this means to say: "Rows that don't satisfy an optional WHERE
clause will be filtered out."

------

2. Commit comment - wording

"The row filter is per table, which allows different row filters to be
defined for different tables."

=>

I think all that is the same as just saying: "The row filter is per table."

------

3. PG docs - independent improvement

You wrote (ref [1] point 3):

"I agree it can be confusing. BTW, CREATE PUBLICATION does not mention that the
root partitioned table is used. We should improve that sentence too."

I agree, but that PG docs improvement is independent of your RowFilter
patch; please make another thread for that idea.

------

4. doc/src/sgml/ref/create_publication.sgml - independent improvement

@@ -131,9 +135,9 @@ CREATE PUBLICATION <replaceable
class="parameter">name</replaceable>

I think that Tomas wrote (ref [2] point 2) that this change seems
unrelated to your RowFilter patch.

I agree; I liked the change, but IMO you need to propose this one in
another thread too.

------

5. doc/src/sgml/ref/create_subscription.sgml - wording

@@ -102,7 +102,16 @@ CREATE SUBSCRIPTION <replaceable
class="parameter">subscription_name</replaceabl
          <para>

I felt that the sentence: "If any table in the publications has a
<literal>WHERE</literal> clause, data synchronization does not use it
if the subscriber is a <productname>PostgreSQL</productname> version
before 15."

Could be expressed more simply like: "If the subscriber is a
<productname>PostgreSQL</productname> version before 15 then any row
filtering is ignored."

------

6. src/backend/commands/publicationcmds.c - wrong function comment

 /*

=>

The 2nd parameter does not exist in v18, so that comment about
pub_drop_table seems to be a cut/paste error from the OpenTableList.

------

src/backend/replication/logical/tablesync.c - bug ?

=>

I did not understand the above call to make_copy_attnamelist. The
result seems unused before it is overwritten later in this same
function (??)

------

7. src/backend/replication/logical/tablesync.c  -
fetch_remote_table_info enhancement

=>

I think a small improvement is possible in this SQL.

If we change that to "SELECT DISTINCT pg_get_expr(prqual, prrelid)"...
then it avoids the copy SQL from having multiple WHERE clauses which
are all identical. This could happen when subscribed to multiple
publications which had the same filter for the same table.

I attached a tmp POC patch for this change and it works as expected.
For example, I subscribe to 3 publications, but 2 of them have the
same filter for the table.

BEFORE
COPY (SELECT key, value, data FROM public.test WHERE (key > 0) AND
(key > 1000) AND (key > 1000)) TO STDOUT

AFTER
COPY (SELECT key, value, data FROM public.test WHERE (key > 0) AND
(key > 1000) ) TO STDOUT

------

8. src/backend/replication/pgoutput/pgoutput.c - qual member is redundant

  bool replicate_valid;

=>

Now that the exprstate is introduced I think that the other member
"qual" is redundant, so it can be removed.

FYI - I attached a tmp patch with all the qual references deleted and
everything is fine.

------

9. src/backend/replication/pgoutput/pgoutput.c - comment typo?

+ /*

=>

typo: it/that ?

I think it ought to say "This is the same code as ExecPrepareExpr()
but that is not used because"...

------

10. src/backend/replication/pgoutput/pgoutput.c - redundant debug logging?

The above debug logging is really only a repeat (with different
wording) of the same information already being logged inside the
pgoutput_row_filter_exec_expr function isn't it? Consider removing the
redundant logging.

e.g. This is already getting logged by pgoutput_row_filter_exec_expr:

    elog(DEBUG3, "row filter evaluates to %s (isnull: %s)",
         DatumGetBool(ret) ? "true" : "false",
         isnull ? "true" : "false");

------
[1] https://www.postgresql.org/message-id/532a18d8-ce90-4444-8570-8a9fcf09f329%40www.fastmail.com
[2] https://www.postgresql.org/message-id/849ee491-bba3-c0ae-cc25-4fce1c03f105%40enterprisedb.com
[3] https://www.postgresql.org/message-id/532a18d8-ce90-4444-8570-8a9fcf09f329%40www.fastmail.com

Kind Regards,
Peter Smith.
Fujitsu Australia

---

From: Amit Kapila <[email-masked]>
Date: Tue, 13 Jul 2021 10:24:39 +0530

> as long as all the operators are built-in?

Yeah, and the patch already disallows the user-defined operators in
filters. I think ideally if the operator doesn't refer to UDFs, we can
allow to directly use such an OP in the filter as we can add a
dependency for the same.

> like what the transform* functions do now.

I think it is better to use some form of walker here rather than
extending the grammar for this. However, the question is do we need
some special kind of expression walker here or can we handle all
required cases via transformWhereClause() call as the patch is trying
to do. AFAIU, the main things we want to prohibit in the filter are:
(a) it doesn't refer to any relation other than catalog in where
clause, (b) it doesn't use UDFs in any way (in expressions, in
user-defined operators, user-defined types, etc.), (c) the columns
referred to in the filter should be part of PK or Replica Identity.
Now, if all such things can be detected by the approach patch has
taken then why do we need a special kind of expression walker? OTOH,
if we can't detect some of this then probably we can use a special
walker.

I think in the long run one idea to allow UDFs is probably by
explicitly allowing users to specify whether the function is
publication predicate safe and if so, then we can allow such functions
in the filter clause.

-- 
With Regards,
Amit Kapila.

---

From: Peter Smith <[email-masked]>
Date: Tue, 13 Jul 2021 16:59:08 +1000

Hi Euler,

Greg noticed that your patch set was missing any implementation of the
psql tab auto-complete for the new row filter WHERE syntax.

So I have added a POC patch for this missing feature.

Unfortunately, there is an existing HEAD problem overlapping with this
exact same code. I reported this already in another thread [1].

So there are 2 patches attached here:
0001 - Fixes the other reported problem (I hope this may be pushed soon)
0002 - Adds the tab-completion code for your row filter WHERE's

------
[1] https://www.postgresql.org/message-id/CAHut+Ps-vkmnWAShWSRVCB3gx8aM=[email-masked]

Kind Regards,
Peter Smith.
Fujitsu Australia

---

From: Amit Kapila <[email-masked]>
Date: Tue, 13 Jul 2021 16:27:34 +0530

> publication predicate safe and if so, then we can allow such functions

Another idea here could be to read the publication-related catalog
with the latest snapshot instead of a historic snapshot. If we do that
then if the user faces problems as described by Petr [1] due to
missing dependencies via UDFs then she can Alter the Publication to
remove/change the filter clause and after that, we would be able to
recognize the updated filter clause and the system will be able to
move forward.

I might be missing something but reading publication catalogs with
non-historic snapshots shouldn't create problems as we use the
historic snapshots are required to decode WAL.

I think the problem described by Petr[1] is also possible today if the
user drops the publication and there is a corresponding subscription,
basically, the system will stuck with error: "ERROR:  publication
"mypub" does not exist. I think allowing to use non-historic snapshots
just for publications will resolve that problem as well.

[1] - https://www.postgresql.org/message-id/92e5587d-28b8-5849-2374-5ca3863256f1%402ndquadrant.com

-- 
With Regards,
Amit Kapila.

---

From: Jeff Davis <[email-masked]>
Date: Tue, 13 Jul 2021 08:44:43 -0700

On Tue, 2021-07-13 at 10:24 +0530, Amit Kapila wrote:
> clause,

Right, because the walsender is using a historical snapshot.

> (b) it doesn't use UDFs in any way (in expressions, in
> user-defined operators, user-defined types, etc.),

Is this a reasonable requirement? Postgres has a long history of
allowing UDFs nearly everywhere that a built-in is allowed. It feels
wrong to make built-ins special for this feature.

> (c) the columns

Why?

Also:

* Andres also mentioned that the function should not leak memory.
* One use case for this feature is when sharding a table, so the
expression should allow things like "hashint8(x) between ...". I'd
really like to see this problem solved, as well.

> publication predicate safe and if so, then we can allow such

This sounds like a better direction. We probably need some kind of
catalog information here to say what functions/operators are "safe" for
this kind of purpose. There are a couple questions:

1. Should this notion of safety be specific to this feature, or should
we try to generalize it so that other areas of the system might benefit
as well?

2. Should this marking be superuser-only, or user-specified?

3. Should it be related to the IMMUTABLE/STABLE/VOLATILE designation,
or completely separate?

Regards,
	Jeff Davis

---

From: Tomas Vondra <[email-masked]>
Date: Tue, 13 Jul 2021 21:07:38 +0200

> wrong to make built-ins special for this feature.

Well, we can either prohibit UDF or introduce a massive foot-gun.

The problem with functions in general (let's ignore SQL functions) is 
that they're black boxes, so we don't know what's inside. And if the 
function gets broken after an object gets dropped, the replication is 
broken and the only way to fix it is to recover the subscription.

And this is not hypothetical issue, we've seen this repeatedly :-(

So as much as I'd like to see support for UDFs here, I think it's better 
to disallow them - at least for now. And maybe relax that restriction 
later, if possible.

>> referred to in the filter should be part of PK or Replica Identity.

I'm not sure either.

> really like to see this problem solved, as well.

I think built-in functions should be fine, because generally don't get 
dropped etc. (And if you drop built-in function, well - sorry.)

Not sure about the memory leaks - I suppose we'd free memory for each 
row, so this shouldn't be an issue I guess ...

> this kind of purpose. There are a couple questions:

Not sure. It's true it's a bit like volatile/stable/immutable categories 
where we can't guarantee those labels are correct, and it's up to the 
user to keep the pieces if they pick the wrong category.

But we can achieve the same goal by introducing a simple GUC called 
dangerous_allow_udf_in_decoding, I think.

regards

---

From: Tomas Vondra <[email-masked]>
Date: Tue, 13 Jul 2021 21:15:48 +0200

> historic snapshots are required to decode WAL.

IMHO the best option for v1 is to just restrict the filters to 
known-safe expressions. That is, just built-in operators, no UDFs etc. 
Yes, it's not great, but both alternative proposals (allowing UDFs or 
using current snapshot) are problematic for various reasons.

Even with those restrictions the row filtering seems quite useful, and 
we can relax those restrictions later if we find acceptable compromise 
and/or decide it's worth the risk. Seems better than having to introduce 
new restrictions later.

> [1] - https://www.postgresql.org/message-id/92e5587d-28b8-5849-2374-5ca3863256f1%402ndquadrant.com

That seems like a completely different problem, TBH. For example the 
slot is dropped too, which means the WAL is likely gone etc.

regards

---

From: Alvaro Herrera <[email-masked]>
Date: Tue, 13 Jul 2021 15:21:17 -0400

> Not sure about the memory leaks - I suppose we'd free memory for each row,
> so this shouldn't be an issue I guess ...

I'm not sure we need to be terribly strict about expression evaluation
not leaking any memory here.   I'd rather have a memory context that can
be reset per row.

-- 
Álvaro Herrera         PostgreSQL Developer  —  https://www.EnterpriseDB.com/

---

From: "Euler Taveira" <[email-masked]>
Date: Tue, 13 Jul 2021 17:37:55 -0300

> I have reviewed the latest v18 patch. Below are some more review
Peter, thanks for quickly check the new patch. I'm attaching a new patch (v19)
that addresses (a) this new review, (b) Tomas' review and (c) Greg's review. I
also included the copy/equal node support for the new node (PublicationTable)
mentioned by Tomas in another email.

8<

> clause will be filtered out."
Agreed.

> I think all that is the same as just saying: "The row filter is per table."
Agreed.

> patch; please make another thread for that idea.
I will. And I will also include the next item that I removed from the patch.

> I agree; I liked the change, but IMO you need to propose this one in
Reverted.

8<

> <productname>PostgreSQL</productname> version before 15 then any row
Agreed.

8<

> pub_drop_table seems to be a cut/paste error from the OpenTableList.
Oops. Removed.

> result seems unused before it is overwritten later in this same
Good catch. This seems to be a leftover from an ancient version.

> publications which had the same filter for the same table.
Good catch!

> "qual" is redundant, so it can be removed.
I was thinking about it for the next patch. Removed.

8<

Fixed.

> pgoutput_row_filter_exec_expr function isn't it? Consider removing the
Agreed. Removed.

--
Euler Taveira
EDB   https://www.enterprisedb.com/
From edc9ea832bacb663b47375b2132fe3f9d7482f78 Mon Sep 17 00:00:00 2001
From: Euler Taveira <[email-masked]>
Date: Mon, 18 Jan 2021 12:07:51 -0300
Subject: [PATCH v19] Row filter for logical replication

This feature adds row filter for publication tables. When a publication
is defined or modified, rows that don't satisfy an optional WHERE clause
will be filtered out. This allows a database or set of tables to be
partially replicated. The row filter is per table. A new row filter can
be added simply by specifying a WHERE clause after the table name. The
WHERE clause must be enclosed by parentheses.

If your publication contains a partitioned table, the publication parameter
publish_via_partition_root determines if it uses the partition row filter (if
the parameter is false, the default) or the root partitioned table row filter.
---
 doc/src/sgml/catalogs.sgml                  |   8 +
 doc/src/sgml/ref/alter_publication.sgml     |  11 +-
 doc/src/sgml/ref/create_publication.sgml    |  33 ++-
 doc/src/sgml/ref/create_subscription.sgml   |  10 +-
 src/backend/catalog/pg_publication.c        |  47 ++-
 src/backend/commands/publicationcmds.c      | 110 +++++---
 src/backend/nodes/copyfuncs.c               |  14 +
 src/backend/nodes/equalfuncs.c              |  12 +
 src/backend/parser/gram.y                   |  24 +-
 src/backend/parser/parse_agg.c              |  10 +
 src/backend/parser/parse_expr.c             |  13 +
 src/backend/parser/parse_func.c             |   3 +
 src/backend/parser/parse_oper.c             |   7 +
 src/backend/replication/logical/tablesync.c |  95 ++++++-
 src/backend/replication/pgoutput/pgoutput.c | 255 ++++++++++++++++-
 src/bin/pg_dump/pg_dump.c                   |  24 +-
 src/bin/pg_dump/pg_dump.h                   |   1 +
 src/bin/psql/describe.c                     |  15 +-
 src/include/catalog/pg_publication.h        |   9 +-
 src/include/catalog/pg_publication_rel.h    |   6 +
 src/include/nodes/nodes.h                   |   1 +
 src/include/nodes/parsenodes.h              |  11 +-
 src/include/parser/parse_node.h             |   1 +
 src/test/regress/expected/publication.out   |  71 +++++
 src/test/regress/sql/publication.sql        |  32 +++
 src/test/subscription/t/021_row_filter.pl   | 298 ++++++++++++++++++++
 26 files changed, 1047 insertions(+), 74 deletions(-)
 create mode 100644 src/test/subscription/t/021_row_filter.pl

index faa114b2c6..4bb4314458 100644
 
  <refsynopsisdiv>
 <synopsis>
+      is evaluated with the role used for the replication connection.
index ff82fbca55..35006d9ffc 100644
 
      <para>
@@ -182,6 +186,21 @@ CREATE PUBLICATION <replaceable class="parameter">name</replaceable>
    disallowed on those tables.
   </para>
 
+  <para>
+   The <literal>WHERE</literal> clause should contain only columns that are
+   part of the primary key or be covered  by <literal>REPLICA
+   IDENTITY</literal> otherwise, <command>DELETE</command> operations will not
+   be replicated. That's because old row is used and it only contains primary
+   key or columns that are part of the <literal>REPLICA IDENTITY</literal>; the
+   remaining columns are <literal>NULL</literal>. For <command>INSERT</command>
+   and <command>UPDATE</command> operations, any column might be used in the
+   <literal>WHERE</literal> clause. New row is used and it contains all
+   columns. A <literal>NULL</literal> value causes the expression to evaluate
+   to false; avoid using columns without not-null constraints in the
+   <literal>WHERE</literal> clause. The <literal>WHERE</literal> clause does
+   not allow functions and user-defined operators.
+  </para>
+
   <para>
    For an <command>INSERT ... ON CONFLICT</command> command, the publication will
    publish the operation that actually results from the command.  So depending
@@ -197,6 +216,11 @@ CREATE PUBLICATION <replaceable class="parameter">name</replaceable>
   <para>
    <acronym>DDL</acronym> operations are not published.
   </para>
+
+  <para>
+   The <literal>WHERE</literal> clause expression is executed with the role used
+   for the replication connection.
+  </para>
  </refsect1>
 
  <refsect1>
@@ -209,6 +233,13 @@ CREATE PUBLICATION mypublication FOR TABLE users, departments;
 </programlisting>
   </para>
 
+  <para>
index e812beee37..cf9424845d 100644
@@ -102,7 +102,15 @@ CREATE SUBSCRIPTION <replaceable class="parameter">subscription_name</replaceabl
+          to be copied. If the subscriber is a
+          <productname>PostgreSQL</productname> version before 15 then any row
+          filtering is ignored.
index 86e415af89..b084cea2aa 100644
+	Node	   *whereclause = NULL;

+	relid = RelationGetRelid(targetrel);
+
 	/*
 	 * Check for duplicates. Note that this does not really prevent
 	 * duplicates, it's here just to provide nicer error message in common
@@ -177,6 +186,26 @@ publication_add_relation(Oid pubid, Relation targetrel,

+	if (pri->whereClause != NULL)
+	{
@@ -189,6 +218,12 @@ publication_add_relation(Oid pubid, Relation targetrel,

 	/* Insert tuple into catalog. */
@@ -205,6 +240,14 @@ publication_add_relation(Oid pubid, Relation targetrel,
 	ObjectAddressSet(referenced, RelationRelationId, relid);
 	recordDependencyOn(&myself, &referenced, DEPENDENCY_AUTO);

diff --git a/src/backend/commands/publicationcmds.c b/src/backend/commands/publicationcmds.c
index 95c253c8e0..659f448c97 100644
--- a/src/backend/commands/publicationcmds.c
+++ b/src/backend/commands/publicationcmds.c
@@ -385,31 +385,24 @@ AlterPublicationTables(AlterPublicationStmt *stmt, Relation rel,
 		List	   *delrels = NIL;
 		ListCell   *oldlc;

-			foreach(newlc, rels)
-			{
 
 		/* And drop them. */
@@ -499,7 +492,8 @@ RemovePublicationRelById(Oid proid)
 }
 
 /*
- * Open relations specified by a RangeVar list.
+ * Open relations specified by a RangeVar list (PublicationTable or Relation).
+ *
  * The returned tables are locked in ShareUpdateExclusiveLock mode in order to
  * add them to a publication.
  */
@@ -509,16 +503,41 @@ OpenTableList(List *tables)
 	List	   *relids = NIL;
 	List	   *rels = NIL;
 	ListCell   *lc;
+	PublicationRelationInfo *pri;
 
 	/*
 	 * Open, share-lock, and check all the explicitly-specified relations
 	 */
 	foreach(lc, tables)
 	{
-		RangeVar   *rv = castNode(RangeVar, lfirst(lc));
-		bool		recurse = rv->inh;
+		PublicationTable *t = NULL;
+		RangeVar   *rv;
+		bool		recurse;
 		Relation	rel;
 		Oid			myrelid;
+		bool		whereclause;
+
+		/*
+		 * ALTER PUBLICATION ... ADD TABLE provides a PublicationTable List
+		 * (Relation, Where clause). ALTER PUBLICATION ... DROP TABLE provides
+		 * a Relation List. Check the List element to be used.
+		 */
+		if (IsA(lfirst(lc), PublicationTable))
+			whereclause = true;
+		else
+			whereclause = false;
+
+		if (whereclause)
+		{
+			t = lfirst(lc);
+			rv = castNode(RangeVar, t->relation);
+		}
+		else
+		{
+			rv = castNode(RangeVar, lfirst(lc));
+		}
+
+		recurse = rv->inh;
 
@@ -538,8 +557,14 @@ OpenTableList(List *tables)
 
 		/*
@@ -572,7 +597,15 @@ OpenTableList(List *tables)
 
@@ -593,10 +626,12 @@ CloseTableList(List *rels)
 
 	foreach(lc, rels)
 	{

 /*
@@ -612,15 +647,15 @@ PublicationAddTables(Oid pubid, List *rels, bool if_not_exists,
 
 	foreach(lc, rels)
 	{

-		obj = publication_add_relation(pubid, rel, if_not_exists);
+		obj = publication_add_relation(pubid, pri, if_not_exists);
 		if (stmt)
 		{
 			EventTriggerCollectSimpleCommand(obj, InvalidObjectAddress,
@@ -644,11 +679,10 @@ PublicationDropTables(Oid pubid, List *rels, bool missing_ok)
 
 	foreach(lc, rels)
 	{
 
@@ -658,7 +692,7 @@ PublicationDropTables(Oid pubid, List *rels, bool missing_ok)
 
 		ObjectAddressSet(obj, PublicationRelRelationId, prid);
diff --git a/src/backend/nodes/copyfuncs.c b/src/backend/nodes/copyfuncs.c
index 6fef067957..e04797ba58 100644
--- a/src/backend/nodes/copyfuncs.c
+++ b/src/backend/nodes/copyfuncs.c
@@ -4840,6 +4840,17 @@ _copyAlterPublicationStmt(const AlterPublicationStmt *from)
 	return newnode;
 }
 
+static PublicationTable *
+_copyPublicationTable(const PublicationTable *from)
+{
+	PublicationTable *newnode = makeNode(PublicationTable);
+
+	COPY_NODE_FIELD(relation);
+	COPY_NODE_FIELD(whereClause);
+
+	return newnode;
+}
+
 static CreateSubscriptionStmt *
 _copyCreateSubscriptionStmt(const CreateSubscriptionStmt *from)
 {
@@ -5704,6 +5715,9 @@ copyObjectImpl(const void *from)
 		case T_AlterPublicationStmt:
 			retval = _copyAlterPublicationStmt(from);
 			break;
+		case T_PublicationTable:
+			retval = _copyPublicationTable(from);
+			break;
 		case T_CreateSubscriptionStmt:
 			retval = _copyCreateSubscriptionStmt(from);
 			break;
diff --git a/src/backend/nodes/equalfuncs.c b/src/backend/nodes/equalfuncs.c
index b9cc7b199c..06b0adc2b0 100644
--- a/src/backend/nodes/equalfuncs.c
+++ b/src/backend/nodes/equalfuncs.c
@@ -2314,6 +2314,15 @@ _equalAlterPublicationStmt(const AlterPublicationStmt *a,
 	return true;
 }
 
+static bool
+_equalPublicationTable(const PublicationTable *a, const PublicationTable *b)
+{
+	COMPARE_NODE_FIELD(relation);
+	COMPARE_NODE_FIELD(whereClause);
+
+	return true;
+}
+
 static bool
 _equalCreateSubscriptionStmt(const CreateSubscriptionStmt *a,
 							 const CreateSubscriptionStmt *b)
@@ -3700,6 +3709,9 @@ equal(const void *a, const void *b)
 		case T_AlterPublicationStmt:
 			retval = _equalAlterPublicationStmt(a, b);
 			break;
+		case T_PublicationTable:
+			retval = _equalPublicationTable(a, b);
+			break;
 		case T_CreateSubscriptionStmt:
 			retval = _equalCreateSubscriptionStmt(a, b);
 			break;
diff --git a/src/backend/parser/gram.y b/src/backend/parser/gram.y
index eb24195438..d82ea003db 100644
--- a/src/backend/parser/gram.y
+++ b/src/backend/parser/gram.y
@@ -426,14 +426,14 @@ static Node *makeRecursiveViewSelect(char *relname, List *aliases, Node *query);
 				transform_element_list transform_type_list
 				TriggerTransitions TriggerReferencing
 				vacuum_relation_list opt_vacuum_relation_list
-				drop_option_list
+				drop_option_list publication_table_list

 
 publication_for_tables:
 
+publication_table_list:
 
 			break;

 			/*

 
 			/*
 
 			/*
 
 			/*
 
index 682c107e74..a37a6feff1 100644
 
 /*
 
 	lrel->nspname = nspname;
 	lrel->relname = relname;
@@ -799,6 +803,59 @@ fetch_remote_table_info(char *nspname, char *relname,
 	lrel->natts = natt;
 
+	 * Get relation qual. DISTINCT avoids the same expression of a table in
+	 * multiple publications from being included multiple times in the final
+	 * expression.
+						 "SELECT DISTINCT pg_get_expr(prqual, prrelid) "
 
@@ -812,6 +869,7 @@ copy_table(Relation rel)
 {
 	LogicalRepRelMapEntry *relmapentry;
 	LogicalRepRelation lrel;
+	List	   *qual = NIL;
 	WalRcvExecResult *res;
 	StringInfoData cmd;
 	CopyFromState cstate;
@@ -820,7 +878,7 @@ copy_table(Relation rel)

 	/* Put the relation into relmap. */
 	logicalrep_relmap_update(&lrel);
@@ -831,14 +889,18 @@ copy_table(Relation rel)
 
@@ -847,8 +909,29 @@ copy_table(Relation rel)
index abd5217ab1..7fbf9f4e31 100644

@@ -99,6 +108,8 @@ typedef struct RelationSyncEntry
 
 	bool		replicate_valid;
 
 	/*
 	 * OID of the relation to publish changes as.  For a partition, this may
@@ -122,7 +133,7 @@ static HTAB *RelationSyncCache = NULL;
 
@@ -131,6 +142,13 @@ static void set_schema_sent_in_streamed_txn(RelationSyncEntry *entry,
 
@@ -520,6 +538,149 @@ send_relation_and_attrs(Relation relation, TransactionId xid,
 
+/*
+	 * ExecPrepareExpr() but that is not used because it doesn't use an EState.
+	if (entry->exprstate == NIL)
@@ -547,7 +708,7 @@ pgoutput_change(LogicalDecodingContext *ctx, ReorderBufferTXN *txn,
 	if (in_streaming)
 		xid = change->txn->xid;

 	/* First check the table filter */
 	switch (change->action)
@@ -571,8 +732,6 @@ pgoutput_change(LogicalDecodingContext *ctx, ReorderBufferTXN *txn,
 	/* Avoid leaking memory by using and resetting our own context */
 	old = MemoryContextSwitchTo(data->context);
 
-	maybe_send_schema(ctx, txn, change, relation, relentry);
-
 	/* Send the data */
 	switch (change->action)
 	{
@@ -580,6 +739,16 @@ pgoutput_change(LogicalDecodingContext *ctx, ReorderBufferTXN *txn,
 			{
 				HeapTuple	tuple = &change->data.tp.newtuple->tuple;
 
@@ -603,6 +772,12 @@ pgoutput_change(LogicalDecodingContext *ctx, ReorderBufferTXN *txn,
 
@@ -631,6 +806,12 @@ pgoutput_change(LogicalDecodingContext *ctx, ReorderBufferTXN *txn,
 
@@ -694,7 +875,7 @@ pgoutput_truncate(LogicalDecodingContext *ctx, ReorderBufferTXN *txn,

@@ -1005,9 +1186,10 @@ set_schema_sent_in_streamed_txn(RelationSyncEntry *entry, TransactionId xid)
@@ -1030,6 +1212,8 @@ get_rel_sync_entry(PGOutputData *data, Oid relid)
@@ -1041,6 +1225,7 @@ get_rel_sync_entry(PGOutputData *data, Oid relid)
+		TupleDesc	tupdesc = RelationGetDescr(relation);
 
 		/* Reload publications if needed before use. */
 		if (!publications_valid)
@@ -1054,6 +1239,22 @@ get_rel_sync_entry(PGOutputData *data, Oid relid)
 			publications_valid = true;
 		}
 
+		/* Release tuple table slot */
+		if (entry->scantuple != NULL)
+		{
+			ExecDropSingleTupleTableSlot(entry->scantuple);
+			entry->scantuple = NULL;
+		}
+
+		/*
+		 * Create a tuple table slot for row filter. TupleDesc must live as
+		 * long as the cache remains.
+		 */
+		oldctx = MemoryContextSwitchTo(CacheMemoryContext);
+		tupdesc = CreateTupleDescCopy(tupdesc);
+		entry->scantuple = MakeSingleTupleTableSlot(tupdesc, &TTSOpsHeapTuple);
+		MemoryContextSwitchTo(oldctx);
+
 		/*
 		 * Build publication cache. We can't use one provided by relcache as
 		 * relcache considers all publications given relation is in, but here
@@ -1063,6 +1264,9 @@ get_rel_sync_entry(PGOutputData *data, Oid relid)
 		{
 			Publication *pub = lfirst(lc);
 			bool		publish = false;
+			HeapTuple	rftuple;
+			Datum		rfdatum;
+			bool		rfisnull;
 
 			if (pub->alltables)
 			{
@@ -1122,9 +1326,33 @@ get_rel_sync_entry(PGOutputData *data, Oid relid)
 				entry->pubactions.pubtruncate |= pub->pubactions.pubtruncate;
 			}

 		list_free(pubids);
@@ -1242,6 +1470,7 @@ rel_sync_cache_publication_cb(Datum arg, int cacheid, uint32 hashvalue)
 {
 	HASH_SEQ_STATUS status;
 	RelationSyncEntry *entry;
+	MemoryContext oldctx;
 
 	/*
 	 * We can get here if the plugin was used in SQL interface as the
@@ -1251,6 +1480,8 @@ rel_sync_cache_publication_cb(Datum arg, int cacheid, uint32 hashvalue)
 	if (RelationSyncCache == NULL)
 		return;
 
@@ -1268,5 +1499,11 @@ rel_sync_cache_publication_cb(Datum arg, int cacheid, uint32 hashvalue)

 	ntups = PQntuples(res);

 
 
 	/*
 
 /*

 

 /* ----------------

 
 	/*

---

From: "Euler Taveira" <[email-masked]>
Date: Tue, 13 Jul 2021 17:39:27 -0300

> only some basic comments and observations at this point:
Tomas, thanks for reviewing this patch again.

Fixed.

> like a rather unrelated bit.
Removed. I will submit a separate patch for this.

> be quite confusing for users.
Reading again it seems "probably" is confusing. Let's remove it.

> UPDATE departments SET active = false WHERE active;
Yeah. I avoided to mention this internal detail about old/new row but it seems
better to be clear. How about the following paragraph?

  <para>
   The <literal>WHERE</literal> clause should contain only columns that are
   part of the primary key or be covered  by <literal>REPLICA
   IDENTITY</literal> otherwise, <command>DELETE</command> operations will not
   be replicated. That's because old row is used and it only contains primary
   key or columns that are part of the <literal>REPLICA IDENTITY</literal>; the
   remaining columns are <literal>NULL</literal>. For <command>INSERT</command>
   and <command>UPDATE</command> operations, any column might be used in the
   <literal>WHERE</literal> clause. New row is used and it contains all
   columns. A <literal>NULL</literal> value causes the expression to evaluate
   to false; avoid using columns without not-null constraints in the
   <literal>WHERE</literal> clause. The <literal>WHERE</literal> clause does
   not allow functions and user-defined operators.
  </para>

> Does this need to build the parse state even for whereClause == NULL?
No. Fixed.

> containing many tables, but I haven't tried.
This piece of code is already complicated. Amit complained about it too [1].
Are you envisioning any specific issue (other than open thousands of relations,
do some stuff, and close them all)? IMO the open/close relation should be
postponed for as long as possible.

> based on that. However, I think it'd be better to just use a single node 
Amit complained about having a runtime test for ALTER PUBLICATION ... DROP
TABLE in case user provides a WHERE clause [2]. I did that way (runtime test)
because it simplified the code. I would tend to avoid moving grammar task into
a runtime, that's why I agreed to change it. I didn't like the multi-node
argument handling for OpenTableList() (mainly because of the extra argument in
the function signature) but with your suggestion (IsA()) maybe it is
acceptable. What do you think? I included IsA() in v19.

> extra if saves anything, it's just a bit more complex.
See runtime test in [2].

Fixed.

> I think all similar cases use "WHERE conditions" instead.
No. Policy, index, statistics, partition, column generation use expressions.
COPY and trigger use conditions. It is also referred as expression in the
synopsis.

> ParseFuncOrColumn) be a more appropriate place?
Probably. I have to try the multiple possibilities to make sure it forbids all
cases.

> are allowed in WHERE conditions - maybe we should explicitly list what 
I started to investigate how to safely allow built-in functions. There is a
long discussion about using functions in a logical decoding context. As I said
during the last CF for v14, I prefer this to be a separate feature. I realized
that I mentioned that functions and user-defined operators are not allowed in
the commit message but forgot to mention it in the documentation.

> SLOT" in get_rel_sync_entry in this code:
That's because I didn't copy the TupleDesc in CacheMemoryContext. Greg pointed
it too in a previous email [3]. The new patch (v19) includes a fix for it.

[1] https://www.postgresql.org/message-id/CA%2BHiwqG3Jz-cRS%3D4gqXmZDjDAi%3D%3D19GvrFCCqAawwHcOCEn4fQ%40mail.gmail.com
[2] https://www.postgresql.org/message-id/CAA4eK1Lu7oPHm2j%3DnLeqZLVoro76E0EWvH%2B5wmGG39iJNBzUog%40mail.gmail.com
[3] https://www.postgresql.org/message-id/CAJcOf-d70xg1O2jX1CrUeXaj%2BnMas3%2BNyJwYjbRsK6ZctH%2Bx5Q%40mail.gmail.com

---

From: "Euler Taveira" <[email-masked]>
Date: Tue, 13 Jul 2021 18:02:43 -0300

8<

> >> referred to in the filter should be part of PK or Replica Identity.
This patch uses the old row for DELETE operations and new row for INSERT and
UPDATE operations. Since we usually don't use REPLICA IDENTITY FULL, all
columns in an old row that are not part of the PK or REPLICA IDENTITY are NULL.
The row filter evaluates NULL to false. Documentation says

  <para>

---

From: Alvaro Herrera <[email-masked]>
Date: Tue, 13 Jul 2021 17:06:36 -0400

> +   not allow functions and user-defined operators.

There's a couple of points in this paragraph ..

1. if you use REPLICA IDENTITY FULL, then the expressions would work
even if they use any other column with DELETE.  Maybe it would be
reasonable to test for this in the code and raise an error if the
expression requires a column that's not part of the replica identity.
(But that could be relaxed if the publication does not publish
updates/deletes.)

2. For UPDATE, does the expression apply to the old tuple or to the new
tuple?  You say it's the new tuple, but from the user point of view I
think it would make more sense that it would apply to the old tuple.
(Of course, if you're thinking that the R.I. is the PK and the PK is
never changed, then you don't really care which one it is, but I bet
that some people would not like that assumption.)

I think it is sensible that it's the old tuple that is matched, not the
new; consider what happens if you change the PK in the update and the
replica already has that tuple.  If you match on the new tuple and it
doesn't match the expression (so you filter out the update), but the old
tuple does match the expression, then the replica will retain the
mismatching tuple forever.

3. You say that a NULL value in any of those columns causes the
expression to become false and thus the tuple is not published.  This
seems pretty unfriendly, but maybe it would be useful to have examples
of the behavior.  Does ExecInitCheck() handle things in the other way,
and if so does using a similar trick give more useful behavior?

<para>
 The WHERE clause may only contain references to columns that are part
 of the table's replica identity.
 If <>DELETE</> or <>UPDATE</> operations are published, this
 restriction can be bypassed by making the replica identity be the whole
 row with <command>ALTER TABLE .. SET REPLICA IDENTITY FULL</command>.
 The <literal>WHERE</literal> clause does not allow functions or
 user-defined operators.
</para>

-- 
Álvaro Herrera              Valdivia, Chile  —  https://www.EnterpriseDB.com/
"The Gord often wonders why people threaten never to come back after they've
been told never to return" (www.actsofgord.com)

---

From: "Euler Taveira" <[email-masked]>
Date: Tue, 13 Jul 2021 21:57:44 -0300

> (But that could be relaxed if the publication does not publish
I thought about it but came to the conclusion that it doesn't worth it.  Even
with REPLICA IDENTITY FULL expression evaluates to false if the column allows
NULL values. Besides that REPLICA IDENTITY is changed via another DDL (ALTER
TABLE) and you have to make sure you don't allow changing REPLICA IDENTITY
because some row filter uses the column you want to remove from it.

> that some people would not like that assumption.)
New tuple. The main reason is that new tuple is always there for UPDATEs.
Hence, row filter might succeed even if the row filter contains a column that
is not part of PK or REPLICA IDENTITY. pglogical also chooses to use new tuple
when it is available (e.g. for INSERT and UPDATE). If you don't like this
approach we can (a) create a new publication option to choose between old tuple
and new tuple for UPDATEs or (b) qualify columns using a special reference
(such as NEW.id or OLD.foo). Both options can provide flexibility but (a) is
simpler.

> and if so does using a similar trick give more useful behavior?
ExecInitCheck() is designed for CHECK constraints and SQL standard requires
taht NULL constraint conditions are not treated as errors. This feature uses a
WHERE clause and behaves like it. I mean, a NULL result does not return the
row. See ExecQual().

---

From: Amit Kapila <[email-masked]>
Date: Wed, 14 Jul 2021 11:09:08 +0530

> (But that could be relaxed if the publication does not publish

+1.

> because some row filter uses the column you want to remove from it.

Yeah, that is required but is it not feasible to do so?

> New tuple. The main reason is that new tuple is always there for UPDATEs.

I am not sure if that is a very good reason to use a new tuple.

-- 
With Regards,
Amit Kapila.

---

From: Amit Kapila <[email-masked]>
Date: Wed, 14 Jul 2021 11:16:46 +0530

> > > really like to see this problem solved, as well.
..
> not leaking any memory here.   I'd rather have a memory context that can

I also think that should be sufficient here and if I am reading
correctly patch already evaluates the expression in per-tuple context
and reset it for each tuple. Jeff, do you or Andres have something
else in mind?

-- 
With Regards,
Amit Kapila.

---

From: Amit Kapila <[email-masked]>
Date: Wed, 14 Jul 2021 14:59:26 +0530

> dropped etc. (And if you drop built-in function, well - sorry.)

I am not sure if all built-in functions are also safe. I think we
can't allow volatile functions (ex. setval) that can update the
database which doesn't seem to be allowed in the historic snapshot.
Similarly, it might not be okay to invoke stable functions that access
the database as those might expect current snapshot. I think immutable
functions should be okay but that brings us to Jeff's question of can
we tie the marking of functions that can be used here with
IMMUTABLE/STABLE/VOLATILE designation? The UDFs might have a higher
risk that something used in those functions can be dropped but I guess
we can address that by using the current snapshot to access the
publication catalog.

> dangerous_allow_udf_in_decoding, I think.

One guc for all UDFs sounds dangerous.

-- 
With Regards,
Amit Kapila.

---

From: Amit Kapila <[email-masked]>
Date: Wed, 14 Jul 2021 15:08:15 +0530

> slot is dropped too, which means the WAL is likely gone etc.

I think if we can use WAL archive (if available) and re-create the
slot, the system should move but recreating the publication won't
allow the system to move.

-- 
With Regards,
Amit Kapila.

---

From: Tomas Vondra <[email-masked]>
Date: Wed, 14 Jul 2021 12:28:16 +0200

> I am not sure if that is a very good reason to use a new tuple.

True. Perhaps we should look at other places with similar concept of 
WHERE conditions and old/new rows, and try to be consistent with those?

I can think of:

1) updatable views with CHECK option

2) row-level security

3) triggers

Is there some reasonable rule which of the old/new tuples (or both) to 
use for the WHERE condition? Or maybe it'd be handy to allow referencing 
OLD/NEW as in triggers?

regards

---

From: Dilip Kumar <[email-masked]>
Date: Wed, 14 Jul 2021 16:30:30 +0530

> use for the WHERE condition? Or maybe it'd be handy to allow referencing

I think for insert we are only allowing those rows to replicate which
are matching filter conditions, so if we updating any row then also we
should maintain that sanity right? That means at least on the NEW rows
we should apply the filter, IMHO.  Said that, now if there is any row
inserted which were satisfying the filter and replicated, if we update
it with the new value which is not satisfying the filter then it will
not be replicated,  I think that makes sense because if an insert is
not sending any row to a replica which is not satisfying the filter
then why update has to do that, right?

-- 
Regards,
Dilip Kumar

---

From: Greg Nancarrow <[email-masked]>
Date: Wed, 14 Jul 2021 21:21:36 +1000

> mentioned by Tomas in another email.

Some minor v19 patch review points you might consider for your next
patch version:
(I'm still considering the other issues raised about WHERE clauses and
filtering)

(1) src/backend/commands/publicationcmds.c
OpenTableList

Some suggested abbreviations:

BEFORE:
if (IsA(lfirst(lc), PublicationTable))
   whereclause = true;
else
   whereclause = false;

AFTER:
whereclause = IsA(lfirst(lc), PublicationTable);

BEFORE:
if (whereclause)
   pri->whereClause = t->whereClause;
else
   pri->whereClause = NULL;

AFTER:
pri->whereClause = whereclause? t->whereClause : NULL;

(2) src/backend/parser/parse_expr.c

I think that the check below:

/* Functions are not allowed in publication WHERE clauses */
if (pstate->p_expr_kind == EXPR_KIND_PUBLICATION_WHERE &&
nodeTag(expr) == T_FuncCall)
    ereport(ERROR,
        (errcode(ERRCODE_FEATURE_NOT_SUPPORTED),
        errmsg("functions are not allowed in publication WHERE expressions"),
        parser_errposition(pstate, exprLocation(expr))));

should be moved down into the "T_FuncCall" case of the switch
statement below it, so that "if (pstate->p_expr_kind ==
EXPR_KIND_PUBLICATION_WHERE" doesn't get checked every call to
transformExprRecurse() regardless of the expression Node type.

(3) Save a nanosecond when entry->exprstate is already NIL:

BEFORE:
if (entry->exprstate != NIL)
   list_free_deep(entry->exprstate);
entry->exprstate = NIL;

AFTER:
if (entry->exprstate != NIL)
{
   list_free_deep(entry->exprstate);
   entry->exprstate = NIL;
}

Regards,
Greg Nancarrow
Fujitsu Australia

---

From: Amit Kapila <[email-masked]>
Date: Wed, 14 Jul 2021 18:20:16 +0530

> use for the WHERE condition? Or maybe it'd be handy to allow referencing

I think apart from the above, it might be good if we can find what
some other databases does in this regard?

-- 
With Regards,
Amit Kapila.

---

From: Alvaro Herrera <[email-masked]>
Date: Wed, 14 Jul 2021 10:01:05 -0400

> then why update has to do that, right?

Right, that's a good aspect to think about.

I think the guiding principle for which tuple to use for the filter is
what is most useful to the potential user of the feature, rather than
what is the easiest to implement.

-- 
Álvaro Herrera              Valdivia, Chile  —  https://www.EnterpriseDB.com/
"La libertad es como el dinero; el que no la sabe emplear la pierde" (Álvarez)

---

From: Tomas Vondra <[email-masked]>
Date: Wed, 14 Jul 2021 16:16:56 +0200

> Right, that's a good aspect to think about.

I agree, that seems like a reasonable approach.

The way I'm thinking about this is that for INSERT and DELETE it's clear 
which row version should be used (because there's just one). And for 
UPDATE we could see that as DELETE + INSERT, and apply the same rule to 
each action.

On the other hand, I can imagine cases where it'd be useful to send the 
UPDATE when the old row matches the condition and new row does not.

> what is the easiest to implement.

+1

---

From: Tomas Vondra <[email-masked]>
Date: Wed, 14 Jul 2021 16:33:49 +0200

> some other databases does in this regard?

Yeah, that might tell us what the users would like to do with it. I did 
some quick search, but haven't found much :-( The one thing I found is 
that Debezium [1] allows accessing both the "old" and "new" rows through 
value.before and value.after, and use both for filtering.

I haven't found much about how this works in other databases, sadly.

Perhaps the best way forward is to stick to the approach that INSERT 
uses new, DELETE uses old and UPDATE works as DELETE+INSERT (probably), 
and leave anything fancier (like being able to reference both versions 
of the row) for a future patch.

[1] 
https://wanna-joke.com/wp-content/uploads/2015/01/german-translation-comics-science.jpg

regards

---

From: Dilip Kumar <[email-masked]>
Date: Wed, 14 Jul 2021 20:18:34 +0530

On Wed, Jul 14, 2021 at 8:04 PM Tomas Vondra
<[email-masked]> wrote:

> of the row) for a future patch.

If UPDATE works as DELETE+ INSERT, does that mean both the OLD row and
the NEW row should satisfy the filter, then only it will be sent?
That means if we insert a row that is not satisfying the condition
(which is not sent to the subscriber) and later if we update that row
and change the values such that the modified value matches the filter
then we will not send it because only the NEW row is satisfying the
condition but OLD row doesn't.  I am just trying to understand your
idea.  Or you are saying that in this case, we will not send anything
for the OLD row as it was not satisfying the condition but the
modified row will be sent as an INSERT operation because this is
satisfying the condition?

---

From: Alvaro Herrera <[email-masked]>
Date: Wed, 14 Jul 2021 10:52:25 -0400

On 2021-Jul-14, Tomas Vondra wrote:

> which row version should be used (because there's just one). And for UPDATE
> we could see that as DELETE + INSERT, and apply the same rule to each
> action.
> 

In any case, it seems to me that the condition expression should be
scanned to see which columns are used in Vars (pull_varattnos?), and
verify if those columns are in the REPLICA IDENTITY; and if they are
not, raise an error.  Most of the time the REPLICA IDENTITY is going to
be the primary key; but if the user wants to use other columns in the
expression, we can HINT that they can set REPLICA IDENTITY FULL.

-- 
Álvaro Herrera              Valdivia, Chile  —  https://www.EnterpriseDB.com/
<Schwern> It does it in a really, really complicated way
<crab> why does it need to be complicated?
<Schwern> Because it's MakeMaker.

---

From: Tomas Vondra <[email-masked]>
Date: Wed, 14 Jul 2021 16:57:24 +0200

> satisfying the condition?

Good questions. I'm not sure, I probably have not thought it through.

So yeah, I think we should probably stick to the principle that what we 
send needs to match the filter condition, which applied to this case 
would mean we should be looking at the new row version.

The more elaborate scenarios can be added later by a patch allowing to 
explicitly reference the old/new row versions.

regards

---

From: Tomas Vondra <[email-masked]>
Date: Wed, 14 Jul 2021 17:08:34 +0200

> expression, we can HINT that they can set REPLICA IDENTITY FULL.

Yeah, but AFAIK that's needed only when replicating DELETEs, so perhaps 
we could ignore this for subscriptions without DELETE.

The other question is when to check/enforce this. I guess we'll have to 
do that during decoding, not just when the publication is being created, 
because the user can do ALTER TABLE later.

regards

---

From: Alvaro Herrera <[email-masked]>
Date: Wed, 14 Jul 2021 11:13:16 -0400

> Yeah, but AFAIK that's needed only when replicating DELETEs, so perhaps we
> could ignore this for subscriptions without DELETE.

Yeah, I said that too in my older reply :-)

> The other question is when to check/enforce this. I guess we'll have to do
> that during decoding, not just when the publication is being created,

... if you're saying the user can change the replica identity after we
have some publications with filters defined, then I think we should
verify during ALTER TABLE and not allow the change if there's a
publication that requires it.  I mean, during decoding we should be able
to simply assume that the tuple is correct for what we need at that
point.

---

From: "Euler Taveira" <[email-masked]>
Date: Wed, 14 Jul 2021 13:30:33 -0300

> satisfying the condition?
That's a fair argument for the default UPDATE behavior. It seems we have a
consensus that UPDATE operation will use old row. If there is no objections, I
will change it in the next version.

We can certainly discuss the possibilities for UPDATE operations. It can choose
which row to use: old, new or both (using an additional publication argument or
OLD and NEW placeholders to reference old and new rows are feasible ideas).

---

From: "Euler Taveira" <[email-masked]>
Date: Wed, 14 Jul 2021 14:25:20 -0300

> we could ignore this for subscriptions without DELETE.
... and UPDATE. It seems we have a consensus to use old row in the row filter
for UPDATEs. I think you meant publication.

> because the user can do ALTER TABLE later.
I'm afraid this check during decoding has a considerable cost. If we want to
enforce this condition, I suggest that we add it to CREATE PUBLICATION, ALTER
PUBLICATION ... ADD|SET TABLE and ALTER TABLE ... REPLICA IDENTITY. Data are
being constantly modified; schema is not.

---

From: "Euler Taveira" <[email-masked]>
Date: Wed, 14 Jul 2021 15:30:15 -0300

> Some minor v19 patch review points you might consider for your next
Greg, thanks for another review. I agree with all of these changes. It will be
in the next patch.

---

From: Amit Kapila <[email-masked]>
Date: Thu, 15 Jul 2021 07:32:25 +0530

+1.

-- 
With Regards,
Amit Kapila.

---

From: Amit Kapila <[email-masked]>
Date: Thu, 15 Jul 2021 07:36:52 +0530

> for UPDATEs. I think you meant publication.

If I read correctly people are suggesting to use a new row for updates
but I still suggest completing the analysis (or at least spend some
more time) Tomas and I requested in the few emails above and then
conclude on this point.

-- 
With Regards,
Amit Kapila.

---

From: Dilip Kumar <[email-masked]>
Date: Thu, 15 Jul 2021 09:37:53 +0530

> If I read correctly people are suggesting to use a new row for updates

Right

+1

---

From: Greg Nancarrow <[email-masked]>
Date: Thu, 15 Jul 2021 21:50:46 +1000

> some other databases does in this regard?

I did a bit of investigation in the case of Oracle Database and SQL Server.
(purely from my interpretation of available documentation; I did not
actually use the replication software)

For Oracle (GoldenGate), it appears that it provides the ability for
filters to reference both OLD and NEW rows in replication of UPDATEs:
"For update operations, it can be advantageous to retrieve the before
values of source columns: the values before the update occurred. These
values are stored in the trail and can be used in filters and column
mappings"
It provides @BEFORE and @AFTER functions for this.

For SQL Server, the available replication models seem quite different
to that in PostgreSQL, and not all seem to support row filtering.
For "snapshot replication", it seems that it effectively supports
filtering rows on the NEW values.
It seems that the snapshot is taken at a transactional boundary and
rows included according to any filtering, and is then replicated.
So to include the result of a particular UPDATE in the replication,
the replication row filtering would effectively be done on the result
(NEW) rows.
Another type of replication that supports row filtering is "merge
replication", which again seems to be effectively based on NEW rows:
"For merge replication to process a row, the data in the row must
satisfy the row filter, and it must have changed since the last
synchronization"
It's not clear to me if there is ANY way to filter on the OLD row
values by using some option.

If anybody has experience with the replication software for these
other databases and I've interpreted the documentation for these
incorrectly, please let me know.

Regards,
Greg Nancarrow
Fujitsu Australia

---

From: Peter Smith <[email-masked]>
Date: Fri, 16 Jul 2021 12:35:36 +1000

> Greg, thanks for another review. I agree with all of these changes. It will be

Hi, here are a couple more minor review comments for the V19 patch.

(The 2nd one overlaps a bit with one that Greg previously gave).

//////

1. doc/src/sgml/ref/create_publication.sgml

=>

typo: "and user-defined operators." --> "or user-defined operators."

------

2. src/backend/commands/publicationcmds.c - OpenTableList IsA logic

IIUC the tables list can only consist of one kind of list element.

Since there is no expected/permitted "mixture" of kinds then there is
no need to check the IsA within the loop like v19 is doing; instead
you can check only the list head element. If you want to, then you
could Assert that every list element has a consistent kind as the
initial kind, but maybe that is overkill too?

PSA a small tmp patch to demonstrate what this comment is about.

------
Kind Regards,
Peter Smith.
Fujitsu Australia

---

From: Amit Kapila <[email-masked]>
Date: Fri, 16 Jul 2021 08:56:53 +0530

> then why update has to do that, right?

There is another theory in this regard which is what if the old row
(created by the previous insert) is not sent to the subscriber as that
didn't match the filter but after the update, we decide to send it
because the updated row (new row) matches the filter condition. In
this case, I think it will generate an update conflict on the
subscriber as the old row won't be present. As of now, we just skip
the update but in the future, we might have some conflict handling
there. If this is true then even if the new row matches the filter,
there is no guarantee that it will be applied on the subscriber-side
unless the old row also matches the filter. Sure, there could be a
case where the user might have changed the filter between insert and
update but maybe we can have a separate way to deal with such cases if
required like providing some provision where the user can specify
whether it would like to match old/new row in updates?

-- 
With Regards,
Amit Kapila.

---

From: Dilip Kumar <[email-masked]>
Date: Fri, 16 Jul 2021 10:10:56 +0530

On Fri, Jul 16, 2021 at 8:57 AM Amit Kapila <[email-masked]> wrote:
> unless the old row also matches the filter.

Yeah, it's a valid point.

 Sure, there could be a
> whether it would like to match old/new row in updates?

Yeah, I think the best way is that users should get an option whether
they want to apply the filter on the old row or on the new row, or
both, in fact, they should be able to apply the different filters on
old and new rows.  I have one more thought in mind: currently, we are
providing a filter for the publication table, doesn't it make sense to
provide filters for operations of the publication table?  I mean the
different filters for Insert, delete, and the old row of update and
the new row of the update.

---

From: Amit Kapila <[email-masked]>
Date: Fri, 16 Jul 2021 11:19:53 +0530

On Fri, Jul 16, 2021 at 10:11 AM Dilip Kumar <[email-masked]> wrote:
> old and new rows.

I am not so sure about different filters for old and new rows but it
makes sense to by default apply the filter to both old and new rows.
Then also provide a way for user to specify if the filter can be
specified to just old or new row.

>  I have one more thought in mind: currently, we are

Hmm, I think this sounds a bit of a stretch but if there is any field
use case then we can consider this in the future.

-- 
With Regards,
Amit Kapila.

---

From: Greg Nancarrow <[email-masked]>
Date: Fri, 16 Jul 2021 17:37:05 +1000

> specified to just old or new row.

I'm having some doubts and concerns about what is being suggested.

My current thought and opinion is that the row filter should
(initially, or at least by default) specify the condition of the row
data at the publication boundary (i.e. what is actually sent to and
received by the subscriber). That means for UPDATE, I think that the
filter should operate on the new value.
This has the clear advantage of knowing (from the WHERE expression)
what restrictions are placed on the data that is actually published
and what subscribers will actually receive. So it's more predictable.
If we filter on OLD rows, then we would need to know exactly what is
updated by the UPDATE in order to know what is actually published (for
example, the UPDATE could modify the columns being checked in the
publication WHERE expression).
I'm not saying that's wrong, or a bad idea, but it's more complicated
and potentially confusing. Maybe there could be an option for it.
Also, even if we allowed OLD/NEW to be specified in the WHERE
expression, OLD wouldn't make sense for INSERT and NEW wouldn't make
sense for DELETE, so one WHERE expression with OLD/NEW references
wouldn't seem valid to cover all operations INSERT, UPDATE and DELETE.
I think that was what Dilip was essentially referring to, with his
suggestion of using different filters for different operations (though
I think that may be going too far for the initial implementation).

Regards,
Greg Nancarrow
Fujitsu Australia

---

From: Tomas Vondra <[email-masked]>
Date: Fri, 16 Jul 2021 10:45:59 +0200

On 7/16/21 5:26 AM, Amit Kapila wrote:
> there.

Right.

> If this is true then even if the new row matches the filter,
> unless the old row also matches the filter. Sure, there could be a > case where the user might have changed the filter between insert and

I think the best we can do for now is to document this. AFAICS it can't 
be solved without a conflict resolution that would turn the UPDATE to 
INSERT. And that would require REPLICA IDENTITY FULL, otherwise the 
UPDATE would not have data for all the columns.

regards

---

From: Alvaro Herrera <[email-masked]>
Date: Fri, 16 Jul 2021 17:35:33 -0400

> I'm having some doubts and concerns about what is being suggested.

Yeah.  I think the idea that some updates fail to reach the replica,
leaving the downstream database in a different state than it would be if
those updates had reached it, is unsettling.  It makes me wish we raised
an error at UPDATE time if both rows would not pass the filter test in
the same way -- that is, if the old row passes the filter, then the new
row must be a pass as well.

Maybe a second option is to have replication change any UPDATE into
either an INSERT or a DELETE, if the old or the new row do not pass the
filter, respectively.  That way, the databases would remain consistent.

-- 
Álvaro Herrera           39°49'30"S 73°17'W  —  https://www.EnterpriseDB.com/
"You're _really_ hosed if the person doing the hiring doesn't understand
relational systems: you end up with a whole raft of programmers, none of
whom has had a Date with the clue stick."              (Andrew Sullivan)

---

From: Amit Kapila <[email-masked]>
Date: Mon, 19 Jul 2021 15:12:14 +0530

> row must be a pass as well.

Hmm, do you mean to say that raise an error in walsender while
decoding if old or new doesn't match filter clause? How would
walsender come out of that error? Even, if seeing the error user
changed the filter clause for publication, I think it would still see
the old ones due to historical snapshot and keep on getting the same
error. One idea could be that we use the current snapshot to read the
publications catalog table, then the user would probably change the
filter or do something to move forward from this error. The other
options could be:

a. Just log it and move to the next row
b. send to stats collector some info about this which can be displayed
in a view and then move ahead
c. just skip it like any other row that doesn't match the filter clause.

I am not sure if there is any use of sending a row if one of the
old/new rows doesn't match the filter. Because if the old row doesn't
match but the new one matches the criteria, we will anyway just throw
such a row on the subscriber instead of applying it. OTOH, if old
matches but new doesn't match then it probably doesn't fit the analogy
that new rows should behave similarly to Inserts. I am of opinion that
we should do either (a) or (c) when one of the old or new rows doesn't
match the filter clause.

> filter, respectively.  That way, the databases would remain consistent.

I guess such things should be handled via conflict resolution on the
subscriber side.

-- 
With Regards,
Amit Kapila.

---

From: Dilip Kumar <[email-masked]>
Date: Mon, 19 Jul 2021 16:30:47 +0530

On Mon, Jul 19, 2021 at 3:12 PM Amit Kapila <[email-masked]> wrote:
> such a row on the subscriber instead of applying it.

But at some time that will be true even if we skip the row based on
(a) or (c) right.  Suppose the OLD row was not satisfying the
condition but the NEW row is satisfying the condition, now even if we
skip this operation then in the next operation on the same row even if
both OLD and NEW rows are satisfying the filter the operation will
just be dropped by the subscriber right? because we did not send the
previous row when it first updated to value which were satisfying the
condition.  So basically, any row is inserted which did not satisfy
the condition first then post that no matter how many updates we do to
that row either it will be skipped by the publisher because the OLD
row was not satisfying the condition or it will be skipped by the
subscriber as there was no matching row.

> > filter, respectively.  That way, the databases would remain consistent.

Yeah, I think this is the best way to keep the data consistent.

---

From: Amit Kapila <[email-masked]>
Date: Mon, 19 Jul 2021 17:00:00 +0530

> value.before and value.after, and use both for filtering.

Okay, but does it apply a filter to both rows for an Update event?

> https://wanna-joke.com/wp-content/uploads/2015/01/german-translation-comics-science.jpg

This link doesn't provide Debezium information, seems like a typo.

-- 
With Regards,
Amit Kapila.

---

From: Tomas Vondra <[email-masked]>
Date: Mon, 19 Jul 2021 15:31:56 +0200

> subscriber as there was no matching row.

I have a feeling it's getting overly complicated, to the extent that
it'll be hard to explain to users and reason about. I don't think
there's a "perfect" solution for cases when the filter expression gives
different answers for old/new row - it'll always be surprising for some
users :-(

So maybe the best thing is to stick to the simple approach already used
e.g. by pglogical, which simply user the new row when available (insert,
update) and old one for deletes.

I think that behaves more or less sensibly and it's easy to explain.

All the other things (e.g. turning UPDATE to INSERT, advanced conflict
resolution etc.) will require a lot of other stuff, and I see them as
improvements of this simple approach.

> Yeah, I think this is the best way to keep the data consistent.

It'd also require REPLICA IDENTITY FULL, which seems like it'd add a
rather significant overhead.

regards

---

From: Tomas Vondra <[email-masked]>
Date: Mon, 19 Jul 2021 15:34:48 +0200

> This link doesn't provide Debezium information, seems like a typo.

Uh, yeah - I copied a different link. I meant to send this one:

https://debezium.io/documentation/reference/configuration/filtering.html

regards

---

From: Greg Nancarrow <[email-masked]>
Date: Tue, 20 Jul 2021 10:12:02 +1000

> improvements of this simple approach.

+1
My thoughts on this are very similar.

Regards,
Greg Nancarrow
Fujitsu Australia

---

From: "[email-masked]" <[email-masked]>
Date: Tue, 20 Jul 2021 00:30:14 +0000

Hi,

I am interested in this feature and took a quick a look at the patch.
Here are a few comments.

(1)

We'd better use appendStringInfoString(&cmd, q);

(2)

Is it better to invoke eval_const_expressions or canonicalize_qual here to
simplify the expression ?

(3)

we'd better use appendPQExpBufferStr instead of appendPQExpBuffer here.

(4)

It might looks clearer to use IsA(expr, FuncCall) here.

Best regards,
Houzj

---

From: Alvaro Herrera <[email-masked]>
Date: Mon, 19 Jul 2021 22:23:07 -0400

> update) and old one for deletes.

OK, no objection to that plan.

-- 
Álvaro Herrera           39°49'30"S 73°17'W  —  https://www.EnterpriseDB.com/
"No es bueno caminar con un hombre muerto"

---

From: Amit Kapila <[email-masked]>
Date: Tue, 20 Jul 2021 09:54:58 +0530

> Yeah, I think this is the best way to keep the data consistent.

Today, while studying the behavior of this particular operation in
other databases, I found that IBM's InfoSphere Data Replication does
exactly this. See [1]. I think there is a merit if want to follow this
idea.

[1] - https://www.ibm.com/docs/en/idr/11.4.0?topic=rows-search-conditions

-- 
With Regards,
Amit Kapila.

---

From: Amit Kapila <[email-masked]>
Date: Tue, 20 Jul 2021 10:53:30 +0530

> different answers for old/new row - it'll always be surprising for some

It is possible but OTOH, the three replication solutions (Debezium,
Oracle, IBM's InfoSphere Data Replication) which have this feature
seems to filter based on both old and new rows in one or another way.
Also, I am not sure if the simple approach of just filter based on the
new row is very clear because it can also confuse users in a way that
even if all the new rows matches the filters, they don't see anything
on the subscriber and in fact, that can cause a lot of network
overhead without any gain.

> I think that behaves more or less sensibly and it's easy to explain.

Okay, if nothing better comes up, then we can fall back to this option.

> resolution etc.) will require a lot of other stuff,
>

I have not evaluated this yet but I think spending some time thinking
about turning Update to Insert/Delete (yesterday's suggestion by
Alvaro) might be worth especially as that seems to be followed by some
other replication solution as well.

>and I see them as

Why? I think it would just need similar restrictions as we are
planning for Delete operation such that filter columns must be either
present in primary or replica identity columns.

-- 
With Regards,
Amit Kapila.

---

From: Greg Nancarrow <[email-masked]>
Date: Tue, 20 Jul 2021 16:08:01 +1000

> exactly this. See [1]. I think there is a merit if want to follow this

So in this model (after initial sync of rows according to the filter),
for UPDATE, the OLD row is checked against the WHERE clause, to know
if the row had been previously published. If it hadn't, and the NEW
row satisfies the WHERE clause, then it needs to be published as an
INSERT. If it had been previously published, but the NEW row doesn't
satisfy the WHERE condition, then it needs to be published as a
DELETE. Otherwise, if both OLD and NEW rows satisfy the WHERE clause,
it needs to be published as an UPDATE.
At least, that seems to be the model when the WHERE clause refers to
the NEW (updated) values, as used in most of their samples (i.e. in
that database "the current log record", indicated by a ":" prefix on
the column name).
I think that allowing the OLD values ("old log record") to be
referenced in the WHERE clause, as that model does, could be
potentially confusing.

Regards,
Greg Nancarrow
Fujitsu Australia

---

From: Amit Kapila <[email-masked]>
Date: Tue, 20 Jul 2021 13:59:28 +0530

> it needs to be published as an UPDATE.

Yeah, this is what I also understood.

> referenced in the WHERE clause, as that model does, could be

I think in terms of referring to old and new rows, we already have
terminology which we used at various other similar places. See Create
Rule docs [1]. For where clause, it says "Within condition and
command, the special table names NEW and OLD can be used to refer to
values in the referenced table. NEW is valid in ON INSERT and ON
UPDATE rules to refer to the new row being inserted or updated. OLD is
valid in ON UPDATE and ON DELETE rules to refer to the existing row
being updated or deleted.". We need similar things for the WHERE
clause in publication if we want special syntax to refer to old and
new rows.

I think if we use some existing way to refer to old/new values then it
shouldn't be confusing to users.

[1] - https://www.postgresql.org/docs/devel/sql-createrule.html

-- 
With Regards,
Amit Kapila.

---

From: Amit Kapila <[email-masked]>
Date: Tue, 20 Jul 2021 14:30:07 +0530

> exactly this. See [1]. I think there is a merit if want to follow this

As per my initial analysis, there shouldn't be much difficulty in
implementing this behavior. We need to change the filter API
(pgoutput_row_filter) such that it tells us whether the filter is
satisfied by the old row, new row or both and then the caller should
be able to make a decision based on that. I think that should be
sufficient to turn update to insert/delete when required. I might be
missing something here but this doesn't appear to require any drastic
changes in the patch.

-- 
With Regards,
Amit Kapila.

---

From: Amit Kapila <[email-masked]>
Date: Tue, 20 Jul 2021 14:32:06 +0530

On Wed, Jul 14, 2021 at 2:08 AM Euler Taveira <[email-masked]> wrote:
> Peter, thanks for quickly check the new patch. I'm attaching a new patch (v19).
>

The latest patch doesn't apply cleanly. Can you please rebase it and
see if you can address some simpler comments till we reach a consensus
on some of the remaining points?

-- 
With Regards,
Amit Kapila.

---

From: Tomas Vondra <[email-masked]>
Date: Tue, 20 Jul 2021 11:09:56 +0200

> overhead without any gain.

True. My point is that it's easier to explain than when using some
combination of old/new row, and theapproach "replicate if the filter
matches both rows" proposed in this thread would be confusing too.

If the subscriber database can be modified, we kinda already have this
issue already - the row can be deleted, and all UPDATEs will be lost.
Yes, for read-only replicas that won't happen, but I think we're moving
to use cases more advanced than that.

I think there are only two ways to *guarantee* this does not happen:

* prohibit updates of columns referenced in row filters

* some sort of conflict resolution, turning UPDATE to INSERT etc.

> other replication solution as well.

I think that requires quite a bit of infrastructure, and I'd bet we'll
need to handle other types of conflicts too. I don't have a clear
opinion if that's required to get this patch working - I'd try getting
the simplest implementation with reasonable behavior, with those more
advanced things as future enhancements.

> present in primary or replica identity columns.

How else would you turn UPDATE to INSERT? For UPDATE we only send the
identity columns and modified columns, and the decision happens on the
subscriber. So we need to send everything if there's a risk we'll need
those columns. But it's early I only had one coffee, so I may be missing
something glaringly obvious.

regards

---

From: Amit Kapila <[email-masked]>
Date: Tue, 20 Jul 2021 15:12:04 +0530

On Tue, Jul 20, 2021 at 2:39 PM Tomas Vondra
<[email-masked]> wrote:
> need to handle other types of conflicts too.
>

Hmm, I don't see why we need any additional infrastructure here if we
do this at the publisher. I think this could be done without many
changes to the patch as explained in one of my previous emails [1].

> I don't have a clear
> subscriber.

Hmm, we log the entire new tuple and replica identity columns for the
old tuple in WAL for Update. And, we are going to use a new tuple for
Insert, so we have everything we need.

[1] - https://www.postgresql.org/message-id/CAA4eK1%2BAXEd5bO-qPp6L9Ptckk09nbWvP8V7q5UW4hg%2BkHjXwQ%40mail.gmail.com

-- 
With Regards,
Amit Kapila.

---

From: Dilip Kumar <[email-masked]>
Date: Tue, 20 Jul 2021 15:18:53 +0530

> Insert, so we have everything we need.

But for making that decision we need to apply the filter on the old
rows as well right.  So if we want to apply the filter on the old rows
then either the filter should only be on the replica identity key or
we need to use REPLICA IDENTITY FULL.  I think that is what Tomas
wants to point out.

---

From: Amit Kapila <[email-masked]>
Date: Tue, 20 Jul 2021 15:32:53 +0530

> we need to use REPLICA IDENTITY FULL.  I think that is what Tomas

I have already mentioned that for Updates the filter needs criteria
similar to Deletes. This is exactly the requirement for Delete as
well.

-- 
With Regards,
Amit Kapila.

---

From: Tomas Vondra <[email-masked]>
Date: Tue, 20 Jul 2021 12:13:13 +0200

> changes to the patch as explained in one of my previous emails [1].

Oh, I see. I've been thinking about doing the "usual" conflict
resolution on the subscriber side. I'm not sure about doing this on the
publisher ...

> Insert, so we have everything we need.

Do we log the TOAST-ed values that were not updated?

regards

---

From: Dilip Kumar <[email-masked]>
Date: Tue, 20 Jul 2021 16:33:13 +0530

> Do we log the TOAST-ed values that were not updated?

No, we don't, I have submitted a patch sometime back to fix that [1]

[1] https://commitfest.postgresql.org/33/3162/

---

From: Greg Nancarrow <[email-masked]>
Date: Tue, 20 Jul 2021 21:42:50 +1000

> clause in publication if we want special syntax to refer to old and

I have no doubt we COULD allow references to OLD and NEW in the WHERE
clause, but do we actually want to?
This is what I thought could cause confusion, when mixed with the
model that I previously described.
It's not entirely clear to me exactly how it works, when the WHERE
clause is applied to the OLD and NEW rows, when the WHERE condition
itself can refer to OLD and/or NEW (coupled with the fact that NEW
doesn't make sense for DELETE and OLD doesn't make sense for INSERT).
Combine that with the fact that a publication can have multiple tables
each with their own WHERE clause, and tables can be dropped/(re)added
to the publication with a different WHERE clause, and it starts to get
a little complicated working out exactly what the result should be.

Regards,
Greg Nancarrow
Fujitsu Australia

---

From: Amit Kapila <[email-masked]>
Date: Tue, 20 Jul 2021 17:37:58 +0530

> doesn't make sense for DELETE and OLD doesn't make sense for INSERT).

It is not new, the same is true when they are used in RULES and
probably in other places where we use them.

-- 
With Regards,
Amit Kapila.

---

From: Amit Kapila <[email-masked]>
Date: Thu, 22 Jul 2021 17:14:58 +0530

> No, we don't, I have submitted a patch sometime back to fix that [1]

That patch seems to log WAL for key unchanged columns. What about if
unchanged non-key columns? Do they get logged as part of the new tuple
or is there some other way we can get those? If not, then we need to
probably think of restricting filter clause in some way.

-- 
With Regards,
Amit Kapila.

---

From: Dilip Kumar <[email-masked]>
Date: Thu, 22 Jul 2021 20:05:53 +0530

> probably think of restricting filter clause in some way.

But what sort of restrictions? I mean we can not put based on data
type right that will be too restrictive, other option is only to allow
replica identity keys columns in the filter condition?

---

From: Amit Kapila <[email-masked]>
Date: Fri, 23 Jul 2021 08:29:04 +0530

On Thu, Jul 22, 2021 at 8:06 PM Dilip Kumar <[email-masked]> wrote:
> type right that will be too restrictive,
>

Yeah, data type restriction sounds too restrictive and unless the data
is toasted, the data will be anyway available. I think such kind of
restriction should be the last resort but let's try to see if we can
do something better.

> other option is only to allow

Yes, that is what I had in mind because if key column(s) is changed
then we will have data for both old and new tuples. But if it is not
changed then we will have it probably for the old tuple unless we
decide to fix the bug you mentioned in a different way in which case
we might either need to log it for the purpose of this feature (but
that will be any way for HEAD) or need to come up with some other
solution here. I think we can't even fetch such columns data during
decoding because we have catalog-only historic snapshots here. Do you
have any better ideas?

-- 
With Regards,
Amit Kapila.

---

From: Amit Kapila <[email-masked]>
Date: Fri, 23 Jul 2021 08:35:55 +0530

> decoding because we have catalog-only historic snapshots here. Do you

BTW, I wonder how pglogical can handle this because if these unchanged
toasted values are not logged in WAL for the new tuple then how the
comparison for such columns will work? Either they are forcing WAL in
some way or don't allow WHERE clause on such columns or maybe they
have dealt with it in some other way unless they are unaware of this
problem.

-- 
With Regards,
Amit Kapila.

---

From: Rahila Syed <[email-masked]>
Date: Fri, 23 Jul 2021 14:26:56 +0530

> have dealt with it in some other way unless they are unaware of this
The column comparison for row filtering happens before the unchanged toast
columns are filtered. Unchanged toast columns are filtered just before
writing the tuple
to output stream. I think this is the case both for pglogical and the
proposed patch.
So, I can't see why the not logging of unchanged toast columns would be a
problem
for row filtering. Am I missing something?

Thank you,
Rahila Syed

---

From: Amit Kapila <[email-masked]>
Date: Fri, 23 Jul 2021 15:46:23 +0530

On Fri, Jul 23, 2021 at 2:27 PM Rahila Syed <[email-masked]> wrote:
> columns are filtered. Unchanged toast columns are filtered just before writing the tuple
> to output stream.
>

To perform filtering, you need to use the tuple from WAL and that
tuple doesn't seem to have unchanged toast values, so how can we do
filtering? I think it is a good idea to test this once.

-- 
With Regards,
Amit Kapila.

---

From: "[email-masked]" <[email-masked]>
Date: Tue, 27 Jul 2021 00:51:41 +0000

On July 23, 2021 6:16 PM Amit Kapila <[email-masked]> wrote:
> > The column comparison for row filtering happens before the unchanged
> > toast columns are filtered. Unchanged toast columns are filtered just
> > before writing the tuple to output stream.
> >
> 
> To perform filtering, you need to use the tuple from WAL and that tuple doesn't
> seem to have unchanged toast values, so how can we do filtering? I think it is a
> good idea to test this once.

I agreed.

Currently, both unchanged toasted key column and unchanged toasted non-key
column is not logged. So, we cannot get the toasted value directly for these
columns when doing row filtering.

I tested the current patch for toasted data and found a problem: In the current
patch, it will try to fetch the toast data from toast table when doing row
filtering[1]. But, it's unsafe to do that in walsender. We can see it use
HISTORIC snapshot in heap_fetch_toast_slice() and also the comments of
init_toast_snapshot() have said "Detoasting *must* happen in the same
transaction that originally fetched the toast pointer.". The toast data could
have been changed when doing row filtering. For exmaple, I tested the following
steps and get an error.

1) UPDATE a nonkey column in publisher.
2) Use debugger to block the walsender process in function
   pgoutput_row_filter_exec_expr().
3) Open another psql to connect the publisher, and drop the table which updated
   in 1).
4) Unblock the debugger in 2), and then I can see the following error:
---
ERROR:  could not read block 0 in file "base/13675/16391"
---

[1]
(1)------publisher------
CREATE TABLE toasted_key (
    id serial,
    toasted_key text PRIMARY KEY,
    toasted_col1 text,
    toasted_col2 text
);
select repeat('9999999999', 200) as tvalue \gset
CREATE PUBLICATION pub FOR TABLE toasted_key WHERE (toasted_col2 = :'tvalue');
ALTER TABLE toasted_key REPLICA IDENTITY USING INDEX toasted_key_pkey;
ALTER TABLE toasted_key ALTER COLUMN toasted_key SET STORAGE EXTERNAL;
ALTER TABLE toasted_key ALTER COLUMN toasted_col1 SET STORAGE EXTERNAL;
ALTER TABLE toasted_key ALTER COLUMN toasted_col2 SET STORAGE EXTERNAL;
INSERT INTO toasted_key(toasted_key, toasted_col1, toasted_col2) VALUES(repeat('1234567890', 200), repeat('9876543210', 200), repeat('9999999999', 200));

(2)------subscriber------
CREATE TABLE toasted_key (
    id serial,
    toasted_key text PRIMARY KEY,
    toasted_col1 text,
    toasted_col2 text
);

CREATE SUBSCRIPTION sub CONNECTION 'dbname=postgres port=10000' PUBLICATION pub;

(3)------publisher------
UPDATE toasted_key SET toasted_col1 = repeat('1111113113', 200);

Based on the above steps, the row filter will ge through the following path
and fetch toast data in walsender.
------
pgoutput_row_filter_exec_expr
    ...
    texteq
        ...
        text *targ1 = DatumGetTextPP(arg1);
            pg_detoast_datum_packed
                detoast_attr
------

---

From: Dilip Kumar <[email-masked]>
Date: Tue, 27 Jul 2021 09:56:33 +0530

> ERROR:  could not read block 0 in file "base/13675/16391"

Yeah, that's a big problem, seems like the expression evaluation
machinery directly going and detoasting the externally stored data
using some random snapshot.  Ideally, in walsender we can never
attempt to detoast the data because there is no guarantee that those
data are preserved.  Somehow before going to the expression evaluation
machinery, I think we will have to deform that tuple and need to do
something for the externally stored data otherwise it will be very
difficult to control that inside the expression evaluation.

---

From: Peter Smith <[email-masked]>
Date: Tue, 27 Jul 2021 17:52:00 +1000

FYI - v19 --> v20

(Only very minimal changes. Nothing functional)

Changes:

* The v19 patch was broken due to changes of commit [1] so I have
rebased so the cfbot is happy.

* I also renamed the TAP test 021_row_filter.pl ==> 023_row_filter.pl
because commit [2] already added another TAP test numbered 021.

------
[1] https://github.com/postgres/postgres/commit/2b00db4fb0c7f02f000276bfadaab65a14059168
[2] https://github.com/postgres/postgres/commit/a8fd13cab0ba815e9925dc9676e6309f699b5f72

Kind Regards,
Peter Smith.
Fujitsu Australia

---

From: Peter Smith <[email-masked]>
Date: Tue, 3 Aug 2021 13:22:24 +1000

FYI - v20 --> v21

(Only very minimal changes)

* I noticed that the v20 TAP test (023_row_filter.pl) began failing
due to a recent commit [1], so I have rebased it to keep the cfbot
happy.

------
[1] https://github.com/postgres/postgres/commit/201a76183e2056c2217129e12d68c25ec9c559c8

Kind Regards,
Peter Smith.
Fujitsu Australia

---

From: Amit Kapila <[email-masked]>
Date: Tue, 3 Aug 2021 16:25:44 +0530

> difficult to control that inside the expression evaluation.

True, I think it would be possible after we fix the issue reported in
another thread [1] where we will log the key values as part of
old_tuple_key for toast tuples even if they are not changed. We can
have a restriction that in the WHERE clause that user can specify only
Key columns for Updates similar to Deletes. Then, we have the data
required for filter columns basically if the toasted key values are
changed, then they will be anyway part of the old and new tuple and if
they are not changed then they will be part of the old tuple. I have
not checked the implementation part of it but theoretically, it seems
possible. If my understanding is correct then it becomes necessary to
solve the other bug [1] to solve this part of the problem for this
patch. The other possibility is to disallow columns (datatypes) that
can lead to toasted data (at least for Updates) which doesn't sound
like a good idea to me. Do you have any other ideas for this problem?

[1] - https://www.postgresql.org/message-id/OS0PR01MB611342D0A92D4F4BF26C0F47FB229%40OS0PR01MB6113.jpnprd01.prod.outlook.com

-- 
With Regards,
Amit Kapila.

---

From: Peter Smith <[email-masked]>
Date: Thu, 5 Aug 2021 16:46:50 +1000

v21 --> v22

(This small change is only to keep the patch up-to-date with HEAD)

Changes:

* A recent commit [1] added a new TAP subscription test file 023, so
now this patch's test file (previously "023_row_filter.pl") has been
bumped to "024_row_filter.pl".

------
[1] https://github.com/postgres/postgres/commit/63cf61cdeb7b0450dcf3b2f719c553177bac85a2

Kind Regards,
Peter Smith.
Fujitsu Australia

---

From: Peter Smith <[email-masked]>
Date: Sun, 8 Aug 2021 16:04:40 +1000

v22 --> v23

Changes:

* A rebase was needed (due to commit [1]) to keep the patch working with cfbot.

------
[1] https://github.com/postgres/postgres/commit/93d573d86571d148e2d14415166ec6981d34ea9d

Kind Regards,
Peter Smith.
Fujitsu Australia

---

From: Dilip Kumar <[email-masked]>
Date: Mon, 9 Aug 2021 18:12:52 +0530

On Tue, Aug 3, 2021 at 4:25 PM Amit Kapila <[email-masked]> wrote:
> they are not changed then they will be part of the old tuple.

Right.

 I have
> possible.

Yeah, It would be possible to because at least after fixing [1] we
would have the required column data.  The only thing I am worried
about is while applying the filter on the new tuple the toasted
unchanged key data will not be a part of the new tuple.  So we can not
directly call the expression evaluation machinary, basically, somehow
we need to deform the new tuple and then replace the data from the old
tuple before passing it to expression evaluation.  Anyways this is an
implementation part so we can look into that while implementing.

 If my understanding is correct then it becomes necessary to
> patch.

Right.

The other possibility is to disallow columns (datatypes) that
> like a good idea to me.

Yeah, that will be a big limitation, then we won't be able to allow
expression on any varlena types.

 Do you have any other ideas for this problem?

As of now no other better idea to suggest.

---

From: Peter Smith <[email-masked]>
Date: Tue, 24 Aug 2021 17:46:37 +1000

> truncate we won't require row_filtering, so it won't be prepared).

I have used debug logging to confirm that what Amit wrote [1] is
correct; the row-filter ExprState of *every* table's row_filter will
be invalidated (and so subsequently gets rebuilt) when the user
changes the PUBLICATION tables. This was a side-effect of the
rel_sync_cache_publication_cb which is freeing the cached ExprState
and setting the entry->replicate_valid = false; for *every* entry.

So yes, the ExprCache is getting rebuilt for some situations where it
is not strictly necessary to do so.

But...

1. Although the ExprState cache is effective, in practice the
performance improvement was not very much. My previous results [2]
showed only about 2sec saving for 100K calls to the
pgoutput_row_filter function. So I think eliminating just one or two
unnecessary calls in the get_rel_sync_entry is going to make zero
observable difference.

2. IMO it is safe to expect that the ALTER PUBLICATION is a rare
operation relative to the number of times that pgoutput_row_filter
will be called (the pgoutput_row_filter is quite a "hot" function
since it is called for every INSERT/UPDATE/DELETE). It will be orders
of magnitude difference 1:1000, 1:100000 etc.

~~

Anyway, I have implemented the suggested cache change because I agree
it is probably theoretically superior, even if in practice there is
almost no difference.

PSA 2 new patches (v24*)

Summary:

1. Now the rfnode_list row-filter cache is built 1 time only in
function get_rel_sync_entry.

2. Now the ExprState list cache is lazy-built 1 time only when first
needed in function pgoutput_row_filter

3. Now those caches are invalidated in function
rel_sync_cache_relation_cb; Invalidation of one relation's caches will
no longer cause the other relations' row-filter caches to be re-built.

------

I also ran performance tests to compare the old/new ExprState caching.
These tests are inserting 1 million rows using different percentages
of row filtering.

Please refer to the attached result data/results.

The main takeaway points from the test results are:

1. Using row-filter ExprState caching is slightly better than having
no ExprState caching.

2. The old/new style ExprState caches have approximately the same
performance. Essentially the *only* runtime difference with the
old/new cache is the added condition in the pgouput_row_filter to
check if the ExprState cache needs to be lazy-built or not. Over a
million rows maybe this extra condition accounts for a tiny difference
or maybe the small before/after differences can be attributed just to
natural runtime variations.

------
[1] https://www.postgresql.org/message-id/CAA4eK1%2BxQb06NGs6Y7OzwMtKYYixEqR8tdWV5THAVE4SAqNrDg%40mail.gmail.com
[2] https://www.postgresql.org/message-id/CAHut%2BPs3GgPKUJ2npfY4bQdxAmYW%2ByQin%2BhQuBsMYvX%3DkBqEpA%40mail.gmail.com

Kind Regards,
Peter Smith.
Fujitsu Australia

---

From: "Euler Taveira" <[email-masked]>
Date: Tue, 24 Aug 2021 21:21:57 -0300

> is not strictly necessary to do so.
I'm afraid we are overenginnering this feature. We already have a cache
mechanism that was suggested (that shows a small improvement). As you said the
gain for this new improvement is zero or minimal (it depends on your logical
replication setup/maintenance).

> it is probably theoretically superior, even if in practice there is
I didn't inspect your patch carefully but it seems you add another List to
control this new cache mechanism. I don't like it. IMO if we can use the data
structures that we have now, let's implement your idea; otherwise, -1 for this
new micro optimization.

[By the way, it took some time to extract what you changed. Since we're trading
patches, I personally appreciate if you can send a patch on the top of the
current one. I have some changes too and it is time consuming incorporating
changes in the main patch.]