Received: from malur.postgresql.org ([217.196.149.56]) by arkaria.postgresql.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_CBC_SHA1:256) (Exim 4.89) (envelope-from ) id 1iWpRa-00082a-1U for pgsql-bugs@arkaria.postgresql.org; Mon, 18 Nov 2019 22:24:26 +0000 Received: from localhost ([127.0.0.1] helo=malur.postgresql.org) by malur.postgresql.org with esmtp (Exim 4.89) (envelope-from ) id 1iWpRX-0000Vf-SZ for pgsql-bugs@arkaria.postgresql.org; Mon, 18 Nov 2019 22:24:23 +0000 Received: from makus.postgresql.org ([2001:4800:3e1:1::229]) by malur.postgresql.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_CBC_SHA1:256) (Exim 4.89) (envelope-from ) id 1iWpRX-0000VX-Gp for pgsql-bugs@lists.postgresql.org; Mon, 18 Nov 2019 22:24:23 +0000 Received: from out4-smtp.messagingengine.com ([66.111.4.28]) by makus.postgresql.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.92) (envelope-from ) id 1iWpRS-0006VM-PX for pgsql-bugs@lists.postgresql.org; Mon, 18 Nov 2019 22:24:22 +0000 Received: from compute5.internal (compute5.nyi.internal [10.202.2.45]) by mailout.nyi.internal (Postfix) with ESMTP id B0C772263C; Mon, 18 Nov 2019 17:24:17 -0500 (EST) Received: from mailfrontend2 ([10.202.2.163]) by compute5.internal (MEProxy); Mon, 18 Nov 2019 17:24:17 -0500 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=anarazel.de; h= date:from:to:cc:subject:message-id:references:mime-version :content-type:in-reply-to; s=fm3; bh=fz7tIDSMLXhn0q7PkakfjdNZYW+ pBXYr0a9QC9iFqEE=; b=SEem5n5aWyFIobRWUpsTB5/9OUcOBdA2r3L5aWXTqIc JoAUZOmXWQdYXXMQ/PTOrEZVyYXnKyr53MhxNXif3HOKKf1pa3SK/B+KcsPbqEz1 3m4sJZbki9AERLYNQBFfdonBQS8Vm9YxWQQIfP1+2tIRe66adFAJai5GHKeNEWcC P6zyzQ11iGRY/pxzc1ZUFxtfaRRT1lsFQU6PZWZlUNzhnnuW8o6tdgctItLOCvXo BPgXB5Xb8f/V2J+rwpVPqDNQV9oCal8P26d8zz9hVzilWQiqqHzBS8aXK02tZP45 CTrppTGi8YJ3UXfU4OsHG2wsgBzUJmFciCDe+0bGMsA== DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d= messagingengine.com; h=cc:content-type:date:from:in-reply-to :message-id:mime-version:references:subject:to:x-me-proxy :x-me-proxy:x-me-sender:x-me-sender:x-sasl-enc; s=fm1; bh=fz7tID SMLXhn0q7PkakfjdNZYW+pBXYr0a9QC9iFqEE=; b=ulM1VGn3euouyNr4csX3ng RQWrkWuEM7JFp1UGBYQ4kxGC6qwLb7JbQGU447rh4oc5h42vIVFR9L5UFa7Y504U aBOA6fEIrnnCiGid6fUxIhItShzF284MjgF6vcuwPYuH4Imu2OMqUpNqaIKVyptD 8nTRmKSr7Y+Hp8t/qZ1abFzgJ6HF0uKJ5fJxMPPHoTmT76mRZb4J1IBtQKQT+b4n d2x3mA2mzFqFVH727it4P5P9/04M5t/aJKsQFM/Z7l8PyTtVwRgwAtnHEmoCBWyd RMqM4OYAPCC4w0fKdBAJul6t1EYqqePW9l8gaQHBoSxYlxpcDI+6erZe9Tp3Xqww == X-ME-Sender: X-ME-Proxy-Cause: gggruggvucftvghtrhhoucdtuddrgedufedrudegiedgudeflecutefuodetggdotefrod ftvfcurfhrohhfihhlvgemucfhrghsthforghilhdpqfgfvfdpuffrtefokffrpgfnqfgh necuuegrihhlohhuthemuceftddtnecusecvtfgvtghiphhivghnthhsucdlqddutddtmd enucfjughrpeffhffvuffkfhggtggujgesthdtredttddtvdenucfhrhhomheptehnughr vghsucfhrhgvuhhnugcuoegrnhgurhgvshesrghnrghrrgiivghlrdguvgeqnecukfhppe eijedrudeitddrvddukedrvdefjeenucfrrghrrghmpehmrghilhhfrhhomheprghnughr vghssegrnhgrrhgriigvlhdruggvnecuvehluhhsthgvrhfuihiivgeptd X-ME-Proxy: Received: from intern.anarazel.de (c-67-160-218-237.hsd1.ca.comcast.net [67.160.218.237]) by mail.messagingengine.com (Postfix) with ESMTPA id 49DDB3060061; Mon, 18 Nov 2019 17:24:17 -0500 (EST) Date: Mon, 18 Nov 2019 14:24:16 -0800 From: Andres Freund To: Tomas Vondra Cc: andrey.salnikov@dataegret.com, pgsql-bugs@lists.postgresql.org Subject: Re: BUG #16125: Crash of PostgreSQL's wal sender during logical replication Message-ID: <20191118222416.dkn5cdmbxmtcemaf@alap3.anarazel.de> References: <16125-a78dafa032d53f7e@postgresql.org> <20191118205816.5wne6hxedrm7zd5p@development> MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: <20191118205816.5wne6hxedrm7zd5p@development> List-Id: List-Help: List-Subscribe: List-Post: List-Owner: List-Archive: Precedence: bulk Hi, On 2019-11-18 21:58:16 +0100, Tomas Vondra wrote: > and the ReorderBufferToastReplace does this: > > newtup = change->data.tp.newtuple; > > heap_deform_tuple(&newtup->tuple, desc, attrs, isnull); > > but that fails, because the tuple pointer happens to be 0x8, which is > clearly bogus. Not sure where that comes from, I don't recognize that as > a typical patter. It indicates that change->data.tp.newtuple is NULL, afaict. newtup->tuple boils down to ((char *) newtup->tuple) + offsetof(ReorderBufferTupleBuf, tuple) and offsetof(ReorderBufferTupleBuf, tuple) is 0x8. > Can you create a core dump (see [1]), and print 'change' and 'txn' in > frame #2? I wonder if some the other fields are bogus too (but it can't > be entirely true ...), and if the transaction got serialized. Please print change and *change, both, please. I suspect what's happening is that somehow a change that shouldn't have toast changes - e.g. a DELETE - somehow has toast changes. Which then triggers a failure in ReorderBufferToastReplace(), which expects newtuple to be valid. It's probably worthwhile to add an elog(ERROR) check for this, even if this does not turn out to be the case. > > This behaviour does not depends on defined data in tables, because we see it > > in different database with different sets of tables in publications. > > I'm not sure I really believe that. Surely there has to be something > special about your schema, or possibly access patter that triggers this > bug in your environment and not elsewhere. Yea. Are there any C triggers present? Any unusual extensions? Users of the transaction hook, for example? > > Looks like a real issue in logical replication. > > I will happy to provide an additional information about that issue, but i > > should know what else to need to collect for helping to solve this > > problem. > > > > Well, if you can create a reproducer, that'd be the best option, because > then we can investigate locally instead of the ping-ping here. > > But if that's not possible, let's start with the schema and the > additional information from the core file. > > I'd also like to see the contents of the WAL, particularly for the XID > triggering this issue. Please run pg_waldump and see how much data is > there for XID 1667601527. It does commit at 25EE/D6DE6EB8, not sure > where it starts. It may have subtransactions, so don't do just grep. Yea, that'd be helpful. Greetings, Andres Freund