Received: from malur.postgresql.org ([217.196.149.56]) by arkaria.postgresql.org with esmtps (TLS1.3:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.92) (envelope-from ) id 1kRKCv-0000dZ-Q2 for pgsql-bugs@arkaria.postgresql.org; Sat, 10 Oct 2020 19:07:06 +0000 Received: from localhost ([127.0.0.1] helo=malur.postgresql.org) by malur.postgresql.org with esmtp (Exim 4.92) (envelope-from ) id 1kRKCu-0005nr-Mx for pgsql-bugs@arkaria.postgresql.org; Sat, 10 Oct 2020 19:07:04 +0000 Received: from magus.postgresql.org ([2a02:c0:301:0:ffff::29]) by malur.postgresql.org with esmtps (TLS1.3:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.92) (envelope-from ) id 1kRKCt-0005nk-O7 for pgsql-bugs@lists.postgresql.org; Sat, 10 Oct 2020 19:07:04 +0000 Received: from [64.147.123.20] (helo=wout4-smtp.messagingengine.com) by magus.postgresql.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.92) (envelope-from ) id 1kRKCl-0006Us-4G for pgsql-bugs@lists.postgresql.org; Sat, 10 Oct 2020 19:07:03 +0000 Received: from compute2.internal (compute2.nyi.internal [10.202.2.42]) by mailout.west.internal (Postfix) with ESMTP id 9CAF36DC; Sat, 10 Oct 2020 15:06:41 -0400 (EDT) Received: from mailfrontend2 ([10.202.2.163]) by compute2.internal (MEProxy); Sat, 10 Oct 2020 15:06:42 -0400 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d= messagingengine.com; h=cc:content-type:date:from:in-reply-to :message-id:mime-version:subject:to:x-me-proxy:x-me-proxy :x-me-sender:x-me-sender:x-sasl-enc; s=fm1; bh=3Xp7yhb1qFY8MMQ2G eYyfj5+XHEdpZg/a2a84xbMIrc=; b=LEW+UpBmUCFg3zC2L8RqfklgqT8su7YJW 3fHvdYks8lw2C8ri7gDMtUZIAmPyLVkF5MdYrq6omiGDWn2sIilubMwLXrWDc40v MABdsAs2wHR4BZUceNjpBTYBUMy9AEUn88HsCVec4mM22ZmCmb58E4m+r6Yz6Vpe OdxYLaETkkZelW9VkhlDfWH+2B0qxgfRaREIN/NfDAgEUXKiCDiG0gmmRg1ZPXam oaPMIkKVS7dC7IZKi6dVIk0TNpp1NokzZ2cVVC0NGEtcZZ4SsCA0lqb6crYlrvIB XDJQpx0FjYz1p1i/oJiFpaI5dXuhSZ5caqM4lJxUkZoDchhAU5NLQ== X-ME-Sender: X-ME-Proxy-Cause: gggruggvucftvghtrhhoucdtuddrgedujedrheefgddufeefucetufdoteggodetrfdotf fvucfrrhhofhhilhgvmecuhfgrshhtofgrihhlpdfqfgfvpdfurfetoffkrfgpnffqhgen uceurghilhhouhhtmecufedttdenucesvcftvggtihhpihgvnhhtshculddquddttddmne cujfgurhepfffhvffukfggtggujggfsehmtderredtredvnecuhfhrohhmpeetlhhvrghr ohcujfgvrhhrvghrrgcuoegrlhhvhhgvrhhrvgesrghlvhhhrdhnohdqihhprdhorhhgqe enucggtffrrghtthgvrhhnpeeflefhieelhfdtheehleevffekgfffueeitdehueektdek vdffhffhueetgeeuveenucffohhmrghinhepphhoshhtghhrvghsqhhlrdhorhhgnecukf hppeduledtrdelhedrudelrddugeeknecuvehluhhsthgvrhfuihiivgeptdenucfrrghr rghmpehmrghilhhfrhhomheprghlvhhhvghrrhgvsegrlhhvhhdrnhhoqdhiphdrohhrgh X-ME-Proxy: Received: from perhan.alvh.no-ip.org (unknown [190.95.19.148]) by mail.messagingengine.com (Postfix) with ESMTPA id 46C21306467D; Sat, 10 Oct 2020 15:06:40 -0400 (EDT) Received: by perhan.alvh.no-ip.org (Postfix, from userid 1000) id 94D9C2A0FAB; Sat, 10 Oct 2020 16:06:37 -0300 (-03) Date: Sat, 10 Oct 2020 16:06:37 -0300 From: Alvaro Herrera To: Tom Lane Cc: Henry Hinze , pgsql-bugs@lists.postgresql.org, Petr Jelinek , Peter Eisentraut Subject: Re: BUG #16643: PG13 - Logical replication - initial startup never finishes and gets stuck in startup loop Message-ID: <20201010190637.GA5774@alvherre.pgsql> MIME-Version: 1.0 Content-Type: multipart/mixed; boundary="mYCpIKhGyMATD0i+" Content-Disposition: inline In-Reply-To: <911656.1601501524@sss.pgh.pa.us> User-Agent: Mutt/1.10.1 (2018-07-13) X-Host-Lookup-Failed: Reverse DNS lookup failed for 64.147.123.20 (deferred) List-Id: List-Help: List-Subscribe: List-Post: List-Owner: List-Archive: Precedence: bulk --mYCpIKhGyMATD0i+ Content-Type: text/plain; charset=us-ascii Content-Disposition: inline On 2020-Sep-30, Tom Lane wrote: > After some digging around, I realize that that commit actually > resulted in a protocol break. libpqwalreceiver is expecting to > get an additional CommandComplete message after COPY OUT finishes, > per libpqrcv_endstreaming(), and it's no longer getting one. > > (I have not read the protocol document to see if this is per spec; > but spec or no, that's what libpqwalreceiver is expecting.) > > The question that this raises is how the heck did that get past > our test suites? It seems like the error should have been obvious > to even the most minimal testing. So I think I can answer this now. Petr, as intellectual author of this code I would appreciate your thoughts on this -- you probably understand this much better. Same goes for Peter as committer. I think the logical replication feature is missing an detailed architectural diagram. What is happening is that the tablesync code, which is in charge of initial syncing of tables, has two main code paths: the first one is an optimization (?) in LogicalRepSyncTableStart() that if the sync takes little time, the process will exit immediately (via finish_sync_worker()) without returning control to the main logical replication apply worker code. In the TAP tests, all tables are pretty small, so that optimization always fires. So in that case we never get control back to walrcv_endstreaming (the complainant in Henry's test case), which is why replication never fails in the tests. You can verify this because the final "return slotname" line in LogicalRepSyncTableStart has zero coverage. The other code path returns the slot name to ApplyWorkerMain, which does the normal walrcv_startstreaming / LogicalRepApplyLoop / endstreaming dance. And whenever that code path is taken, all the src/test/subscription tests fail with the missing command tag, and they work correctly when the command tag is restored, which is what we wanted. My proposal at this stage is to remove the optimization in LogicalRepSyncTableStart, per 0001, if only because the code becomes simpler (0002 reindents code). With this, we get a coverage increase for tablesync.c not only in the final "return slotname" but also for process_syncing_tables_for_sync() which is currently uncovered: https://coverage.postgresql.org/src/backend/replication/logical/tablesync.c.gcov.html#268 However, and this is one reason why I'd welcome Petr/Peter thoughts on this, I don't really understand what happens in LogicalRepApplyLoop afterwards with a tablesync worker; are we actually doing anything useful there, considering that the actual data copy seems to have occurred in the CopyFrom() call in copy_table? In other words, by the time we return control to ApplyWorkerMain with a slot name, isn't the work all done, and the only thing we need is to synchronize protocol and close the connection? Or is it? Another thing I noticed is that if I crank up the number of rows in the new t/100_bugs.pl test case(*), the two sync workers run in sequence -- not in parallel. What's going on with that? Shouldn't there be two simultaneous workers? (*) In the submitted patch, the test uses 10 million rows. I intend to reduce that to, say, 5000 rows in the committed version. Strictly speaking, there's no need for this test at all since with the LogicalRepSyncTableStart changes I propose, even t/001_rep_changes.pl fails. But since my new test case fails even without that change, I prefer to keep it anyway. --mYCpIKhGyMATD0i+ Content-Type: text/x-diff; charset=us-ascii Content-Disposition: attachment; filename="0001-Restore-double-command-completion-tags.patch"