Received: from malur.postgresql.org ([217.196.149.56]) by arkaria.postgresql.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_CBC_SHA1:256) (Exim 4.89) (envelope-from ) id 1iuHvU-0001uv-AZ for pgsql-bugs@arkaria.postgresql.org; Wed, 22 Jan 2020 15:28:16 +0000 Received: from localhost ([127.0.0.1] helo=malur.postgresql.org) by malur.postgresql.org with esmtp (Exim 4.89) (envelope-from ) id 1iuHvS-0000nr-4c for pgsql-bugs@arkaria.postgresql.org; Wed, 22 Jan 2020 15:28:14 +0000 Received: from magus.postgresql.org ([2a02:c0:301:0:ffff::29]) by malur.postgresql.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_CBC_SHA1:256) (Exim 4.89) (envelope-from ) id 1iuHvR-0000nk-ST for pgsql-bugs@lists.postgresql.org; Wed, 22 Jan 2020 15:28:13 +0000 Received: from sss.pgh.pa.us ([66.207.139.130]) by magus.postgresql.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.92) (envelope-from ) id 1iuHvO-00006T-Vc for pgsql-bugs@lists.postgresql.org; Wed, 22 Jan 2020 15:28:13 +0000 Received: from sss1.sss.pgh.pa.us (localhost [127.0.0.1]) by sss.pgh.pa.us (8.14.4/8.14.4) with ESMTP id 00MFS8KB007345; Wed, 22 Jan 2020 10:28:08 -0500 From: Tom Lane To: vadim@postgrespro.co.il cc: pgsql-bugs@lists.postgresql.org Subject: Re: BUG #16226: background worker "logical replication worker" (PID ) was terminated by signal 11: Segmentation In-reply-to: <16226-690712ac4467269a@postgresql.org> References: <16226-690712ac4467269a@postgresql.org> Comments: In-reply-to PG Bug reporting form message dated "Wed, 22 Jan 2020 07:43:23 +0000" MIME-Version: 1.0 Content-Type: text/plain; charset="UTF-8" Content-ID: <7343.1579706888.1@sss.pgh.pa.us> Content-Transfer-Encoding: 8bit Date: Wed, 22 Jan 2020 10:28:08 -0500 Message-ID: <7344.1579706888@sss.pgh.pa.us> List-Id: List-Help: List-Subscribe: List-Post: List-Owner: List-Archive: Precedence: bulk > We have 2 PostgreSQL servers with logical replication between Postgres 11.6 > (Primary) and 12.1 (Logical). Some times ago, we changed column type in a 2 > big tables from integer to text: > ... > , this of course led to a full rewrite both tables. We repated this > operation on both servers. And after that we started to get error like > "background worker "logical replication worker" (PID ) was terminated > by signal 11: Segmentation fault" and server goes to recovery mode. Not sure, but this seems like it might be explained by this recent bug fix: Author: Tom Lane Branch: master [4d9ceb001] 2019-11-22 11:31:19 -0500 Branch: REL_12_STABLE [a2aa224e0] 2019-11-22 11:31:19 -0500 Branch: REL_11_STABLE [b72a44c51] 2019-11-22 11:31:19 -0500 Branch: REL_10_STABLE [5d3fcb53a] 2019-11-22 11:31:19 -0500 Fix bogus tuple-slot management in logical replication UPDATE handling. slot_modify_cstrings seriously abused the TupleTableSlot API by relying on a slot's underlying data to stay valid across ExecClearTuple. Since this abuse was also quite undocumented, it's little surprise that the case got broken during the v12 slot rewrites. As reported in bug #16129 from Ondřej Jirman, this could lead to crashes or data corruption when a logical replication subscriber processes a row update. Problems would only arise if the subscriber's table contained columns of pass-by-ref types that were not being copied from the publisher. Fix by explicitly copying the datum/isnull arrays from the source slot that the old row was in already. This ends up being about the same thing that happened pre-v12, but hopefully in a less opaque and fragile way. We might've caught the problem sooner if there were any test cases dealing with updates involving non-replicated or dropped columns. Now there are. Back-patch to v10 where this code came in. Even though the failure does not manifest before v12, IMO this code is too fragile to leave as-is. In any case we certainly want the additional test coverage. Patch by me; thanks to Tomas Vondra for initial investigation. Discussion: https://postgr.es/m/16129-a0c0f48e71741e5f@postgresql.org regards, tom lane