Received: from malur.postgresql.org ([217.196.149.56]) by arkaria.postgresql.org with esmtps (TLS1.3:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.92) (envelope-from ) id 1kSVL3-0007Zr-NQ for pgsql-bugs@arkaria.postgresql.org; Wed, 14 Oct 2020 01:12:22 +0000 Received: from localhost ([127.0.0.1] helo=malur.postgresql.org) by malur.postgresql.org with esmtp (Exim 4.92) (envelope-from ) id 1kSVL0-0006gN-ND for pgsql-bugs@arkaria.postgresql.org; Wed, 14 Oct 2020 01:12:18 +0000 Received: from magus.postgresql.org ([2a02:c0:301:0:ffff::29]) by malur.postgresql.org with esmtps (TLS1.3:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.92) (envelope-from ) id 1kSVL0-0006gG-E8 for pgsql-bugs@lists.postgresql.org; Wed, 14 Oct 2020 01:12:18 +0000 Received: from wout4-smtp.messagingengine.com ([64.147.123.20]) by magus.postgresql.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.92) (envelope-from ) id 1kSVKx-0002Q6-5z for pgsql-bugs@lists.postgresql.org; Wed, 14 Oct 2020 01:12:18 +0000 Received: from compute2.internal (compute2.nyi.internal [10.202.2.42]) by mailout.west.internal (Postfix) with ESMTP id 170FB1099; Tue, 13 Oct 2020 21:12:11 -0400 (EDT) Received: from mailfrontend2 ([10.202.2.163]) by compute2.internal (MEProxy); Tue, 13 Oct 2020 21:12:11 -0400 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d= messagingengine.com; h=cc:content-type:date:from:in-reply-to :message-id:mime-version:subject:to:x-me-proxy:x-me-proxy :x-me-sender:x-me-sender:x-sasl-enc; s=fm1; bh=ChqKP3g6elaAwqZf4 CBlZxVUCA2omzD9i4qxpyZeI28=; b=VBFJNaXfgyQx4FZlHWO8RSng7haxdm/5l F1h5gp8T6/oQkhGZr/1NobUb9CW71zHE1kongyzVaD840glgrTk2hr+VY5/lCVj/ +drTCz4q9bPMrR0t9/u8R085UZUgrJcX5oTFWj3R+BbqgEIjUmJ6O+56kQ8tiy9n L8aubxL6K7jveBKx5+x95Q5hMWUqNJZZ/y+1rBiBgUVOLdych89d1NwDH1H9VYts rXJUdTIdoMJ1veQrSSsmbFrFSZXl/J9HYCeqC83eT15XIF2AIZzKNmV//lamhjvV GiPJiT0klgH6GKpZDDsXO0n70DHzCZ1dM8l6c4IRDwfHEJE97NXwg== X-ME-Sender: X-ME-Proxy-Cause: gggruggvucftvghtrhhoucdtuddrgedujedriedtgdeggecutefuodetggdotefrodftvf curfhrohhfihhlvgemucfhrghsthforghilhdpqfgfvfdpuffrtefokffrpgfnqfghnecu uegrihhlohhuthemuceftddtnecusecvtfgvtghiphhivghnthhsucdlqddutddtmdenuc fjughrpeffhffvuffkgggtuggjfgesmhdtreertdervdenucfhrhhomheptehlvhgrrhho ucfjvghrrhgvrhgruceorghlvhhhvghrrhgvsegrlhhvhhdrnhhoqdhiphdrohhrgheqne cuggftrfgrthhtvghrnhepieeghfelueduheegtdekjeehueffvddujefgteduieekjefh udefgeefiedvjeetnecukfhppeduledtrdelhedrudelrddugeeknecuvehluhhsthgvrh fuihiivgeptdenucfrrghrrghmpehmrghilhhfrhhomheprghlvhhhvghrrhgvsegrlhhv hhdrnhhoqdhiphdrohhrgh X-ME-Proxy: Received: from perhan.alvh.no-ip.org (unknown [190.95.19.148]) by mail.messagingengine.com (Postfix) with ESMTPA id DE08D306467E; Tue, 13 Oct 2020 21:12:09 -0400 (EDT) Received: by perhan.alvh.no-ip.org (Postfix, from userid 1000) id 6A3C52A14F1; Tue, 13 Oct 2020 22:12:07 -0300 (-03) Date: Tue, 13 Oct 2020 22:12:07 -0300 From: Alvaro Herrera To: Petr Jelinek Cc: Tom Lane , Henry Hinze , pgsql-bugs@lists.postgresql.org, Peter Eisentraut Subject: Re: BUG #16643: PG13 - Logical replication - initial startup never finishes and gets stuck in startup loop Message-ID: <20201014011207.GA18985@alvherre.pgsql> MIME-Version: 1.0 Content-Type: multipart/mixed; boundary="ReaqsoxgOBHFXBhH" Content-Disposition: inline In-Reply-To: <6decb65b-aa4f-2b9a-1b53-7246cebe481a@2ndquadrant.com> User-Agent: Mutt/1.10.1 (2018-07-13) List-Id: List-Help: List-Subscribe: List-Post: List-Owner: List-Archive: Precedence: bulk --ReaqsoxgOBHFXBhH Content-Type: text/plain; charset=us-ascii Content-Disposition: inline On 2020-Oct-12, Petr Jelinek wrote: > It's not only about size of the tables, it's mainly that there is no write > activity so the main apply is not moving past the LSN at which table sync > has started at. With bigger table you get at least something written > (running xacts, autovacuum, or whatever) that moves lsn forward eventually. I see -- yeah, okay. > > However, and this is one reason why I'd welcome Petr/Peter thoughts on > > this, I don't really understand what happens in LogicalRepApplyLoop > > afterwards with a tablesync worker; are we actually doing anything > > useful there, considering that the actual data copy seems to have > > occurred in the CopyFrom() call in copy_table? In other words, by the > > time we return control to ApplyWorkerMain with a slot name, isn't the > > work all done, and the only thing we need is to synchronize protocol and > > close the connection? > > There are 2 possible states at that point, either tablesync is ahead (when > main apply lags or nothing is happening on publication side) or it's behind > the main apply. When tablesync is ahead we are indeed done and just need to > update the state of the table (which is what the code you removed did, but > LogicalRepApplyLoop should do it as well, just a bit later). When it's > behind we need to do catchup for that table only which still happens in the > tablesync worker. See the explanation at the beginning of tablesync.c, it > probably needs some small adjustments after the changes in your first patch. ... Ooh, things start to make some sense now. So how about the attached? There are some not really related cleanups. (Changes to protocol.sgml are still pending.) If I understand correcly, the early exit in tablesync.c is not saving *a lot* of time (we don't actually skip replaying any WAL), even if it's saving execution of a bunch of code. So I stand by my position that removing the code is better because it's clearer about what is actually happening. --ReaqsoxgOBHFXBhH Content-Type: text/x-diff; charset=us-ascii Content-Disposition: attachment; filename="0001-Restore-logical-replication-dupe-command-tags.patch"