Received: from malur.postgresql.org ([217.196.149.56]) by arkaria.postgresql.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_CBC_SHA1:256) (Exim 4.89) (envelope-from ) id 1iaJY4-0006Vz-31 for pgsql-bugs@arkaria.postgresql.org; Thu, 28 Nov 2019 13:09:32 +0000 Received: from localhost ([127.0.0.1] helo=malur.postgresql.org) by malur.postgresql.org with esmtp (Exim 4.89) (envelope-from ) id 1iaJY2-0007mO-TQ for pgsql-bugs@arkaria.postgresql.org; Thu, 28 Nov 2019 13:09:30 +0000 Received: from magus.postgresql.org ([2a02:c0:301:0:ffff::29]) by malur.postgresql.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_CBC_SHA1:256) (Exim 4.89) (envelope-from ) id 1iaJY2-0007mH-Gx for pgsql-bugs@lists.postgresql.org; Thu, 28 Nov 2019 13:09:30 +0000 Received: from mail-qk1-x741.google.com ([2607:f8b0:4864:20::741]) by magus.postgresql.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_CBC_SHA1:256) (Exim 4.89) (envelope-from ) id 1iaJXz-0008Ud-9x for pgsql-bugs@lists.postgresql.org; Thu, 28 Nov 2019 13:09:30 +0000 Received: by mail-qk1-x741.google.com with SMTP id q28so3711462qkn.10 for ; Thu, 28 Nov 2019 05:09:26 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=2ndquadrant-com.20150623.gappssmtp.com; s=20150623; h=date:from:to:cc:subject:message-id:mime-version:content-disposition :content-transfer-encoding:in-reply-to:user-agent; bh=L/SwuM/APkRyBUKFzfb73KYSiXvaZEC32LM/bIMvdqs=; b=foSwaW9O69YMMxF1HCjGqZ9+4x/HfnYEICHaIiuSCQ5UtHFa3Ly6iKR2cjUQIdAUEa vkSiOYslTdELEc7q1jjlMhTyzGH78aVX95yVWm1ZzZOvxirWKFiBnCNCgGpn2WnI0yHy iX/peuGxkOThrVrVfZtNRixPKyLTIGflUpch3OO/WMB2TihVkgB4jEwakITCUAXucxpg ghjfiaUyWUL1twXSGvonFHe2Wj1qrH9+sK85wV254G6nZh0ud3Rx00d6wQlvaLHmu945 jmbRSXyLM5K4aEvzcGO2rYXN7iGXMmBFnLc0tZAzksuYgVEHRECI04pPKZ8mB6n7nCcs 6osw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:date:from:to:cc:subject:message-id:mime-version :content-disposition:content-transfer-encoding:in-reply-to :user-agent; bh=L/SwuM/APkRyBUKFzfb73KYSiXvaZEC32LM/bIMvdqs=; b=qmNPLrpnfwzYlBfojLkkWhKzuteakWzeVaRzJKlK6KDxlvDbrUXvcL7U8LqGiSsdAW SgPT5Xyfp2TxMoSKUCHHWD2WRAGs2tNd/Ogss/yud1ctaOk3wyh5KN+sg8jFiZo6xXl8 6k+QCOlYk2kV+Kx2aG2X/EYYjqwvCw2hUuQAkom4erryJhNOtXNg5iXiFuUxTyZVMbhs eJDDMvj4v8Zmqd84+W4X2zBU/+mT1/7NNNvsENUcerxVQYoTZ/8VyggmjfntbZr4U4C+ od0+I/RhySdU7chPZ74Gyob7oL4pz/ZKijPq80KUcTvO1qP2qXLbLfxq56YvK7UACZD4 2K/g== X-Gm-Message-State: APjAAAW4/bc1Q9hGV5HfHhLrwJ1f/AU9HmTLZ19qXpfsDX/iaA3tI3hL RSCbupoJ2OZKBmXaZh61eHKy1g== X-Google-Smtp-Source: APXvYqzZ3MYTlVFtmZz3mvoamElvuxrUCrT2v0q/Ucr4m8Tf4olBoYAvo6X7OIsH7NOVbbeTTWjxow== X-Received: by 2002:a37:4d16:: with SMTP id a22mr10215970qkb.237.1574946565098; Thu, 28 Nov 2019 05:09:25 -0800 (PST) Received: from nimloth.alvh.no-ip.org ([179.56.96.77]) by smtp.gmail.com with ESMTPSA id n198sm8578006qke.0.2019.11.28.05.09.24 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Thu, 28 Nov 2019 05:09:24 -0800 (PST) Received: by nimloth.alvh.no-ip.org (Postfix, from userid 1000) id 21CB7300663; Thu, 28 Nov 2019 10:09:22 -0300 (-03) Date: Thu, 28 Nov 2019 10:09:22 -0300 From: Alvaro Herrera To: Tomas Vondra Cc: Andres Freund , Andrey Salnikov , pgsql-bugs@lists.postgresql.org Subject: Re: BUG #16125: Crash of PostgreSQL's wal sender during logical replication Message-ID: <20191128130922.GA23504@alvherre.pgsql> MIME-Version: 1.0 Content-Type: text/plain; charset=iso-8859-1 Content-Disposition: inline Content-Transfer-Encoding: 8bit In-Reply-To: <20191127223933.ohe3wtdsssvgxdh5@development> User-Agent: Mutt/1.10.1 (2018-07-13) List-Id: List-Help: List-Subscribe: List-Post: List-Owner: List-Archive: Precedence: bulk > Hmm, so how would that transaction look like? My reading of the WAL > (based on the available relfilenode info) is this: > > Transaction 25EE/D66F0438 ASSIGNMENT xtop 1667601527: subxacts: 1667601528 > > -> seems like a savepoint, probably > > Heap 25EE/D66F11E8 INSERT off 4, blkref #0: rel 1663/13018/88964795 blk 3125 > Heap 25EE/D66F2118 INSERT off 7, blkref #0: rel 1663/13018/88964795 blk 3139 FPW > Btree 25EE/D66F54C8 INSERT_LEAF off 257, blkref #0: rel 1663/13018/88964797 blk 15840 > Heap 25EE/D66F5508 INSERT off 2, blkref #0: rel 1663/13018/88964792 blk 18475 > Btree 25EE/D66F6738 INSERT_LEAF off 140, blkref #0: rel 1663/13018/88964798 blk 2058 > Btree 25EE/D66F7E40 INSERT_LEAF off 28, blkref #0: rel 1663/13018/88964799 blk 5076 > Btree 25EE/D66F7EA0 INSERT_LEAF off 88, blkref #0: rel 1663/13018/88964800 blk 3412 > > -> insert into TOAST table (88964795) + TOAST idx (probably 88964797) > -> then an insert into table2 (88964792), and it's indexes I think it's an insert into table2 (which has at least four indexes), followed by insertion in 3 indexes, followed by failure to insert into the fourth index (which naturally does not appear in WAL). > Heap 25EE/D66F7EF0 DELETE off 2 , blkref #0: rel 1663/13018/88964792 blk 18475 Then the heap insertion is speculative-aborted because of failure to insert into the fourth index ... > Heap 25EE/D66F7F28 DELETE off 4 , blkref #0: rel 1663/13018/88964795 blk 3125 > Heap 25EE/D66F7F60 DELETE off 7 , blkref #0: rel 1663/13018/88964795 blk 3139 ... and this propagates to the toast table (toast_delete in heap_abort_speculative) I didn't look from this point down. I'll look at your reproducer script now; I think you need to make it so that the index specified in the ON CONFLICT clause is the fourth one in OID order. (Not really sure how relevant the other index insertions are, really.) -- Álvaro Herrera https://www.2ndQuadrant.com/ PostgreSQL Development, 24x7 Support, Remote DBA, Training & Services