Received: from malur.postgresql.org ([217.196.149.56]) by arkaria.postgresql.org with esmtps (TLS1.3:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.92) (envelope-from ) id 1n5nhn-0006BV-G5 for pgsql-hackers@arkaria.postgresql.org; Fri, 07 Jan 2022 11:46:47 +0000 Received: from localhost ([127.0.0.1] helo=malur.postgresql.org) by malur.postgresql.org with esmtp (Exim 4.92) (envelope-from ) id 1n5nhm-0000Ci-74 for pgsql-hackers@arkaria.postgresql.org; Fri, 07 Jan 2022 11:46:46 +0000 Received: from makus.postgresql.org ([2001:4800:3e1:1::229]) by malur.postgresql.org with esmtps (TLS1.3:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.92) (envelope-from ) id 1n5nhl-0000CZ-Tc for pgsql-hackers@lists.postgresql.org; Fri, 07 Jan 2022 11:46:45 +0000 Received: from wout3-smtp.messagingengine.com ([64.147.123.19]) by makus.postgresql.org with esmtps (TLS1.3:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.92) (envelope-from ) id 1n5nhi-000587-Kl for pgsql-hackers@postgresql.org; Fri, 07 Jan 2022 11:46:44 +0000 Received: from compute1.internal (compute1.nyi.internal [10.202.2.41]) by mailout.west.internal (Postfix) with ESMTP id 5AA0F3202F43; Fri, 7 Jan 2022 06:46:40 -0500 (EST) Received: from mailfrontend1 ([10.202.2.162]) by compute1.internal (MEProxy); Fri, 07 Jan 2022 06:46:41 -0500 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d= messagingengine.com; h=cc:content-transfer-encoding:content-type :date:from:in-reply-to:message-id:mime-version:references :subject:to:x-me-proxy:x-me-proxy:x-me-sender:x-me-sender :x-sasl-enc; s=fm1; bh=5iM+HvQ75fOYnmBIa3GjEgXunmmGN2wM3Xgpm8/qE 0o=; b=fG9+orCMOvSTx9v0qxwJrCYXwFkWQS9BaS/gOuqGTMvdbUP4zXg3tPkq4 moP72YIjSDWw/QVD0tUlCw25LPqf12MzKTiLHPWVSFjJ78ZE0+l6+mADEzUWqncT wd1BNWtzDcp62ZoOjWLowoTMzOiJABdGlCqAZvmkfCuquikw+bKSO3LzCLcbiXx1 y8Sx1pDOuR3+Oz31fx13ip6MXl01Wlss8pP4YlFLBLZk0u02UfKMLkl6zQCQu1XO xlxMnYpIHkDq9j+3YNn/Fx6LPAqn2qpSk02RmGOsZEl88RXpeYJjwypRN7y7/cc0 GpF46P58TZDMOM6rBt3MJRM7HbPXA== X-ME-Sender: X-ME-Received: X-ME-Proxy-Cause: gggruggvucftvghtrhhoucdtuddrgedvuddrudegvddgtdekucetufdoteggodetrfdotf fvucfrrhhofhhilhgvmecuhfgrshhtofgrihhlpdfqfgfvpdfurfetoffkrfgpnffqhgen uceurghilhhouhhtmecufedttdenucesvcftvggtihhpihgvnhhtshculddquddttddmne cujfgurhepkfffgggfuffvfhfhjggtgfesthejredttdefjeenucfhrhhomheprfgvthgv rhcugfhishgvnhhtrhgruhhtuceophgvthgvrhdrvghishgvnhhtrhgruhhtsegvnhhtvg hrphhrihhsvggusgdrtghomheqnecuggftrfgrthhtvghrnhepfeejgeehteeuhfevvedu leeufedtjeetiefftedvudfhtdeifefgueettdevgefgnecuvehluhhsthgvrhfuihiivg eptdenucfrrghrrghmpehmrghilhhfrhhomhepphgvthgvrhdrvghishgvnhhtrhgruhht segvnhhtvghrphhrihhsvggusgdrtghomh X-ME-Proxy: Received: by mail.messagingengine.com (Postfix) with ESMTPA; Fri, 7 Jan 2022 06:46:37 -0500 (EST) Message-ID: <2d6c8b74-bdef-767b-bdb6-29705985ed9c@enterprisedb.com> Date: Fri, 7 Jan 2022 12:46:35 +0100 MIME-Version: 1.0 User-Agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10.15; rv:91.0) Gecko/20100101 Thunderbird/91.4.1 Subject: Re: Column Filtering in Logical Replication Content-Language: en-US To: Alvaro Herrera , Justin Pryzby Cc: Tomas Vondra , Amit Kapila , "houzj.fnst@fujitsu.com" , Rahila Syed , Peter Smith , pgsql-hackers@postgresql.org References: <202201031431.akwhcx6xrusu@alvherre.pgsql> From: Peter Eisentraut In-Reply-To: <202201031431.akwhcx6xrusu@alvherre.pgsql> Content-Type: text/plain; charset=UTF-8; format=flowed Content-Transfer-Encoding: 7bit List-Id: List-Help: List-Subscribe: List-Post: List-Owner: List-Archive: Archived-At: Precedence: bulk I think this is getting pretty good now. I like the overall behavior now. Some details: There are still a few references to "filter", but I see most of the patch now uses column list or something. Maybe do another cleanup pass before finalizing the patch. doc/src/sgml/catalogs.sgml needs to be updated. doc/src/sgml/ref/alter_publication.sgml: "allows to change" -> "allows changing" src/backend/catalog/pg_publication.c: publication_translate_columns(): I find the style of having a couple of output arguments plus a return value that is actually another output value confusing. (It would be different if the return value was some kind of success value.) Let's make it all output arguments. About the XXX question there: I would make the column numbers always sorted. I don't have a strong reason for this, but otherwise we might get version differences, unstable dumps etc. It doesn't seem complicated to keep this a bit cleaner. I think publication_translate_columns() also needs to prohibit generated columns. We already exclude those implicitly throughout the logical replication code, but if a user explicitly set one here, things would probably break. src/backend/commands/tablecmds.c: ATExecReplicaIdentity(): Regarding the question of how to handle REPLICA_IDENTITY_NOTHING: I see two ways to do this. Right now, the approach is that the user can set the replica identity freely, and we decide later based on that what we can replicate (e.g., no updates). For this patch, that would mean we don't restrict what columns can be in the column list, but we check what actions we can replicate based on the column list. The alternative is that we require the column list to include the replica identity, as the patch is currently doing, which would mean that REPLICA_IDENTITY_NOTHING can be allowed since it's essentially a set of zero columns. I find the current behavior a bit weird on reflection. If a user wants to replicate on some columns and only INSERTs, that should be allowed regardless of what the replica identity columns are. src/backend/replication/pgoutput/pgoutput.c: In get_rel_sync_entry(), why did you remove the block - if (entry->pubactions.pubinsert && entry->pubactions.pubupdate && - entry->pubactions.pubdelete && entry->pubactions.pubtruncate) - break; Maybe this is intentional, but it's not clear to me.