Received: from malur.postgresql.org ([217.196.149.56]) by arkaria.postgresql.org with esmtps (TLS1.3:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.92) (envelope-from ) id 1mvfsZ-00055M-74 for pgsql-hackers@arkaria.postgresql.org; Fri, 10 Dec 2021 13:24:03 +0000 Received: from localhost ([127.0.0.1] helo=malur.postgresql.org) by malur.postgresql.org with esmtp (Exim 4.92) (envelope-from ) id 1mvfsX-0006Kp-WD for pgsql-hackers@arkaria.postgresql.org; Fri, 10 Dec 2021 13:24:02 +0000 Received: from magus.postgresql.org ([2a02:c0:301:0:ffff::29]) by malur.postgresql.org with esmtps (TLS1.3:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.92) (envelope-from ) id 1mvfsX-0006Kf-Mw for pgsql-hackers@lists.postgresql.org; Fri, 10 Dec 2021 13:24:01 +0000 Received: from out3-smtp.messagingengine.com ([66.111.4.27]) by magus.postgresql.org with esmtps (TLS1.3:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.92) (envelope-from ) id 1mvfsU-00048W-9S for pgsql-hackers@postgresql.org; Fri, 10 Dec 2021 13:24:01 +0000 Received: from compute1.internal (compute1.nyi.internal [10.202.2.41]) by mailout.nyi.internal (Postfix) with ESMTP id 1C5BC5C02F4; Fri, 10 Dec 2021 08:23:56 -0500 (EST) Received: from mailfrontend1 ([10.202.2.162]) by compute1.internal (MEProxy); Fri, 10 Dec 2021 08:23:56 -0500 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d= messagingengine.com; h=cc:content-transfer-encoding:content-type :date:from:in-reply-to:message-id:mime-version:subject:to :x-me-proxy:x-me-proxy:x-me-sender:x-me-sender:x-sasl-enc; s= fm1; bh=rltlcR8DGdoB9/TrXNEinahRx1GOPDeFVnAP9f+W1sg=; b=njQ+pEi2 BYIiRljAX5IH+UJcWdenf8LNA3bff6kgIJMl9cMnrXwFwakeli+CgHw2EvYzGxu6 uJbSmhkDzGuP/Z6cw9sbk2zPMilsUF6idAZgDf+83ubWm5fmWMvoDKT4Z6I8BuRj 9Zlmq7tQz3oNPSL155bjUw9cqvr0RQLxHR4XC6R/1Vhy0zC4kYn+iRs7soPup/mn 2ALaOKKQOHNy6jdu5bYxINa3J8sElgBsIH3sqvFJ+b+1FT50MqYGkF20tTomQIWt dFpX1yjKEK5i0vQuD+OSnjWll+KwjclC7JVctoBV4323HboY89pBQiLNHoub520m SKWNcs7aH8FY0g== X-ME-Sender: X-ME-Received: X-ME-Proxy-Cause: gggruggvucftvghtrhhoucdtuddrgedvuddrkedvgdehfecutefuodetggdotefrodftvf curfhrohhfihhlvgemucfhrghsthforghilhdpqfgfvfdpuffrtefokffrpgfnqfghnecu uegrihhlohhuthemuceftddtnecusecvtfgvtghiphhivghnthhsucdlqddutddtmdenuc fjughrpeffhffvuffkgggtugfgjgesthekredttddtjeenucfhrhhomheptehlvhgrrhho ucfjvghrrhgvrhgruceorghlvhhhvghrrhgvsegrlhhvhhdrnhhoqdhiphdrohhrgheqne cuggftrfgrthhtvghrnhepudejveejffejveduudetvdekieeuiedtvdeileefkeevjedu feeguedvjefhueelnecuffhomhgrihhnpegvnhhtvghrphhrihhsvggusgdrtghomhenuc evlhhushhtvghrufhiiigvpedtnecurfgrrhgrmhepmhgrihhlfhhrohhmpegrlhhvhhgv rhhrvgesrghlvhhhrdhnohdqihhprdhorhhg X-ME-Proxy: Received: by mail.messagingengine.com (Postfix) with ESMTPA; Fri, 10 Dec 2021 08:23:55 -0500 (EST) Received: by perhan.alvh.no-ip.org (Postfix, from userid 1000) id 0FF382A075D; Fri, 10 Dec 2021 10:23:51 -0300 (-03) Date: Fri, 10 Dec 2021 10:23:51 -0300 From: Alvaro Herrera To: Peter Eisentraut Cc: PostgreSQL-development Subject: Re: Column Filtering in Logical Replication Message-ID: <202112101323.gdbduksd5w5z@alvherre.pgsql> MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Disposition: inline Content-Transfer-Encoding: 8bit In-Reply-To: <15691075-5172-ad49-e7ae-89b82c1f0cfc@enterprisedb.com> List-Id: List-Help: List-Subscribe: List-Post: List-Owner: List-Archive: Archived-At: Precedence: bulk On 2021-Dec-10, Peter Eisentraut wrote: > I looked through this a bit. You had said that you are still going to > integrate past review comments, so I didn't look to deeply before you get to > that. Thanks for doing this! As it happens I've spent the last couple of days working on some of these details. > There was no documentation, so I wrote a bit (patch 0001). It only touches > the CREATE PUBLICATION and ALTER PUBLICATION pages at the moment. There was > no mention in the Logical Replication chapter that warranted updating. > Perhaps we should revisit that chapter at the end of the release cycle. Thanks. I hadn't looked at the docs yet, so I'll definitely take this. > DDL tests should be done in src/test/regress/sql/publication.sql rather than > through TAP tests, to keep it simpler. Yeah, I noticed this too but hadn't done it yet. > Note the FIXME marker that it does not recognize if the > listed columns don't exist. I had fixed this already, so I suppose it should be okay. > I removed a now redundant test from the TAP > test file. The other error condition test in the TAP test file > ('publication relation test_part removed') I didn't understand: test_part > was added with columns (a, b), so why would dropping column b remove the > whole entry? Maybe I missed something, or this could be explained better. There was some discussion about it earlier in the thread and I was also against this proposed behavior. > I was curious what happens when you have different publications with > different column lists, so I wrote a test for that (patch 0003). It turns > out it works, so there is nothing to do, but perhaps the test is useful to > keep. Great, thanks. Yes, I think it will be. > On the implementation side, I think the added catalog column > pg_publication_rel.prattrs should be an int2 array, not a text array. I already rewrote it to use a int2vector column in pg_publication_rel. This interacted badly with the previous behavior on dropping columns, which I have to revisit, but otherwise it seems much better. (Particularly since we don't need to care about quoting names and such.) > Finally, I suggest not naming this feature "column filter". I think this > name arose because of the analogy with the "row filter" feature also being > developed. But a filter is normally a dynamic data-driven action, which > this is not. Golden Gate calls it in their documentation "Selecting > Columns", or we could just call it "column list". Hmm, I hadn't thought of renaming the feature, but I have to admit that I was confused because of the name, so I agree with choosing some other name. I'll integrate your changes and post the whole thing later. -- Álvaro Herrera 39°49'30"S 73°17'W — https://www.EnterpriseDB.com/ Si no sabes adonde vas, es muy probable que acabes en otra parte.