Received: from malur.postgresql.org ([217.196.149.56]) by arkaria.postgresql.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_CBC_SHA1:256) (Exim 4.89) (envelope-from ) id 1iZyOT-0000Fx-Q1 for pgsql-bugs@arkaria.postgresql.org; Wed, 27 Nov 2019 14:34:14 +0000 Received: from localhost ([127.0.0.1] helo=malur.postgresql.org) by malur.postgresql.org with esmtp (Exim 4.89) (envelope-from ) id 1iZyCs-0002Ph-DU for pgsql-bugs@arkaria.postgresql.org; Wed, 27 Nov 2019 14:22:14 +0000 Received: from makus.postgresql.org ([2001:4800:3e1:1::229]) by malur.postgresql.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_CBC_SHA1:256) (Exim 4.89) (envelope-from ) id 1iZyCr-0002Pa-UB for pgsql-bugs@lists.postgresql.org; Wed, 27 Nov 2019 14:22:14 +0000 Received: from mail-qt1-x842.google.com ([2607:f8b0:4864:20::842]) by makus.postgresql.org with esmtps (TLS1.3:ECDHE_RSA_AES_128_GCM_SHA256:128) (Exim 4.92) (envelope-from ) id 1iZyCm-0005Gw-H7 for pgsql-bugs@lists.postgresql.org; Wed, 27 Nov 2019 14:22:12 +0000 Received: by mail-qt1-x842.google.com with SMTP id i17so25515575qtq.1 for ; Wed, 27 Nov 2019 06:22:08 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=2ndquadrant-com.20150623.gappssmtp.com; s=20150623; h=date:from:to:cc:subject:message-id:mime-version:content-disposition :content-transfer-encoding:in-reply-to:user-agent; bh=ESvP0JDThGCF9aH7U+i5HJVAMAEuSvY50BPZjb6rHM0=; b=JIGEUGIHyzEk/YZGFkazw1k/FUaUx4vRWPll+DIkqBA433OgfOXHPA0iqC+AdDf9jQ L7V51XXaxIikt2hnb2WMEuUwbOpSGL9RAloOyWBU0PT8Q69b6DbWO2p7BesKk07iT8cX 1T1vQXnzBUT8Xl+y1jz0pP81vcPn69hbPos4LUtRFi//82HFahTXBjf4x4BXs9ICgELc Gqvefi7VUQmxDj52bX0L8o7Z8P8HVA9v6o+rRlL4PDEJz3m3YZm9NFquikq0fHvOmIGZ R6LYTltuJnlNZr1Xf1ZruXoyjsfbR0x6hqisWo0fFQRil84FZLVkGeZ+3VppZx6SOlE7 P4Sg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:date:from:to:cc:subject:message-id:mime-version :content-disposition:content-transfer-encoding:in-reply-to :user-agent; bh=ESvP0JDThGCF9aH7U+i5HJVAMAEuSvY50BPZjb6rHM0=; b=GNA1jg110lTSOArTl3cwFlq+fFe/sAbClI3bbc0bY6MUUGmdA4iHXUgSqMQaY1hBsv OdPMCvnJCK+rXj+CPn5e40yCRgqR53XJJVAAoxBfjm6PVceDNbD4x60V3i0qsqIf+RdH vHaF0Axw9Vcr4DZ5XvDB4p4PaqpB29iaNRUlOy8aEmT4+llFYqp6stWrTGwtIMltXXuD nWUWgpaGFasL6lQvP49ttjMM4r58MBdpWa0WFiP5PVf0+VlOyzSOHGQpPdSYs9aLIOre GLwQI5VtgIvWU9Hm8EKy1AGuamTobsx5A2qV47d8GJj4+kWamXX16i+3mmNigAIoKVch vx0w== X-Gm-Message-State: APjAAAW2GpzM2soR9SBKyv4PL98IJipduKni8rk+60NlBKxdInkD1XSb VFSOb9IFLckBs++WDueQQQ5wwg== X-Google-Smtp-Source: APXvYqzfQHGdI0oLeD15CCK5yuCx6RkQOhE8E2G8YowayCJUs1XnlBrjBsXkUdIvG2KBho9wtmpuFg== X-Received: by 2002:aed:3b6c:: with SMTP id q41mr40349490qte.11.1574864527492; Wed, 27 Nov 2019 06:22:07 -0800 (PST) Received: from nimloth.alvh.no-ip.org ([179.56.30.174]) by smtp.gmail.com with ESMTPSA id c80sm6884132qkb.35.2019.11.27.06.22.06 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 27 Nov 2019 06:22:06 -0800 (PST) Received: by nimloth.alvh.no-ip.org (Postfix, from userid 1000) id 4BE1030043C; Wed, 27 Nov 2019 11:22:04 -0300 (-03) Date: Wed, 27 Nov 2019 11:22:04 -0300 From: Alvaro Herrera To: Andres Freund Cc: Tomas Vondra , Andrey Salnikov , pgsql-bugs@lists.postgresql.org Subject: Re: BUG #16125: Crash of PostgreSQL's wal sender during logical replication Message-ID: <20191127142204.GA13686@alvherre.pgsql> MIME-Version: 1.0 Content-Type: text/plain; charset=iso-8859-1 Content-Disposition: inline Content-Transfer-Encoding: 8bit In-Reply-To: <20191119232849.qhtuhq4udrqrz4he@alap3.anarazel.de> User-Agent: Mutt/1.10.1 (2018-07-13) List-Id: List-Help: List-Subscribe: List-Post: List-Owner: List-Archive: Precedence: bulk On 2019-Nov-19, Andres Freund wrote: > > > 2525:rmgr: Heap len (rec/tot): 1671/ 1671, tx: 1667601530, lsn: 25EE/D66F5508, prev 25EE/D66F54C8, desc: INSERT off 2, blkref #0: rel 1663/13018/88964792 blk 18475 > > > 2532:rmgr: Heap len (rec/tot): 54/ 54, tx: 1667601530, lsn: 25EE/D66F7EF0, prev 25EE/D66F7EA0, desc: DELETE off 2 , blkref #0: rel 1663/13018/88964792 blk 18475 > Hm. I don't think it necessarily means that. compute_new_xmax_infomask(), when > called from heap_delete(), will e.g. set HEAP_KEYS_UPDATED when > > if (old_infomask & HEAP_XMAX_INVALID) > > is true. Which is the most common case. I actually find it *more* > curious to *not* see that for a delete. I've not traced fully through > it, yet. > > > Alvaro, it's not clear to me whether the the multixact codepaths in > compute_new_xmax_infomask() actually work as expected for deletes. Hmm ... I think you're complaining about failing to set the HEAP_KEYS_UPDATED bit when the delete transaction is the same as the insert transaction. Grepping the new waldump file the OP sent for relfilenode 88964792 I saw this: rmgr: Heap len (rec/tot): 1671/ 1671, tx: 1667601529, lsn: 25EE/D66F1A78, prev 25EE/D66F1A38, desc: INSERT off 5, blkref #0: rel 1663/13018/88964792 blk 18460 rmgr: Heap len (rec/tot): 1671/ 1671, tx: 1667601530, lsn: 25EE/D66F5508, prev 25EE/D66F54C8, desc: INSERT off 2, blkref #0: rel 1663/13018/88964792 blk 18475 rmgr: Heap len (rec/tot): 48/ 48, tx: 1667601529, lsn: 25EE/D66F7E10, prev 25EE/D66F6778, desc: HEAP_CONFIRM off 5, blkref #0: rel 1663/13018/88964792 blk 18460 rmgr: Heap len (rec/tot): 54/ 54, tx: 1667601530, lsn: 25EE/D66F7EF0, prev 25EE/D66F7EA0, desc: DELETE off 2 , blkref #0: rel 1663/13018/88964792 blk 18475 rmgr: Heap len (rec/tot): 54/ 54, tx: 1667601536, lsn: 25EE/D6BA3BC0, prev 25EE/D6BA3B70, desc: LOCK off 5: xid 1667601536: flags 0 LOCK_ONLY EXCL_LOCK , blkref #0: rel 1663/13018/88964792 blk 20806 rmgr: Heap len (rec/tot): 3955/ 3955, tx: 1667601536, lsn: 25EE/D6BA3D00, prev 25EE/D6BA3CB0, desc: UPDATE off 5 xmax 1667601536 ; new off 2 xmax 0, blkref #0: rel 1663/13018/88964792 blk 18491, blkref #1: rel 1663/13018/88964792 blk 20806 rmgr: Heap len (rec/tot): 54/ 54, tx: 1667601538, lsn: 25EE/D6DE53D0, prev 25EE/D6DE33B0, desc: LOCK off 5: xid 1667601538: flags 0 LOCK_ONLY EXCL_LOCK , blkref #0: rel 1663/13018/88964792 blk 18460 rmgr: Heap len (rec/tot): 6563/ 6563, tx: 1667601538, lsn: 25EE/D6DE5408, prev 25EE/D6DE53D0, desc: UPDATE off 5 xmax 1667601538 ; new off 3 xmax 0, blkref #0: rel 1663/13018/88964792 blk 18499, blkref #1: rel 1663/13018/88964792 blk 18460 I wonder about the HEAP_CONFIRM there ... it's in a completely different transaction of course, so it's not related, but note that the corresponding insert (25EE/D66F1A78) looks like a regular INSERT, it doesn't indicate that it's a speculative insert, which it must have been or there wouldn't be a confirm. So I wonder if our insert just below is *also* a speculative insert ... which perhaps could cause the compute_new_xmax_infomask code to misbehave. I think pg_waldump should print out the speculativeness of an insertion. -- Álvaro Herrera https://www.2ndQuadrant.com/ PostgreSQL Development, 24x7 Support, Remote DBA, Training & Services