Received: from malur.postgresql.org ([217.196.149.56]) by arkaria.postgresql.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_CBC_SHA1:256) (Exim 4.89) (envelope-from ) id 1hpxdA-0001Hu-8n for pgsql-hackers@arkaria.postgresql.org; Tue, 23 Jul 2019 16:27:12 +0000 Received: from localhost ([127.0.0.1] helo=malur.postgresql.org) by malur.postgresql.org with esmtp (Exim 4.89) (envelope-from ) id 1hpxd8-0006BM-Ra for pgsql-hackers@arkaria.postgresql.org; Tue, 23 Jul 2019 16:27:10 +0000 Received: from magus.postgresql.org ([2a02:c0:301:0:ffff::29]) by malur.postgresql.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_CBC_SHA1:256) (Exim 4.89) (envelope-from ) id 1hpxd8-0006BE-EC for pgsql-hackers@lists.postgresql.org; Tue, 23 Jul 2019 16:27:10 +0000 Received: from mail-io1-xd30.google.com ([2607:f8b0:4864:20::d30]) by magus.postgresql.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_CBC_SHA1:256) (Exim 4.89) (envelope-from ) id 1hpxd5-0006Qj-M6 for pgsql-hackers@postgresql.org; Tue, 23 Jul 2019 16:27:09 +0000 Received: by mail-io1-xd30.google.com with SMTP id o9so83215588iom.3 for ; Tue, 23 Jul 2019 09:27:07 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=telsasoft-com.20150623.gappssmtp.com; s=20150623; h=date:from:to:cc:subject:message-id:mime-version:content-disposition :user-agent; bh=xCIVvIszV+VDeGHt0Mc6wMTL3EI01cgqAxFjKbA+V0A=; b=RzfMlT6jNlhAAFQgBNzz810h5u2NYO4Qh3NTtUOjOHPHwpBCJhKieQUKud3gjGlMmZ +ZcpMxbzYxX3LpMgx4ugBBvec8x15DEp/Drm0fIBLGzuKs73XQLIpvCCNpiPFJN/JOZe p+iKha20N6YLEZwwru6lKtasA56bou1Hz9qo2kLzG00uCTwkQmKKCDJSVrkhBDRBE18t XJ9i5CpE0/eNVeF8XYAUdR3Z9WLY755ZPTU9r9NypFL+nP9WSQ+WKqmKcgk2HvPr5k9n NHl3zmanPhaMWkSvanSV6Id8WzOUu8b6pARHXt+R0PPGOUxWp5IudTdisFaj9FgKuDD/ 1cbg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:date:from:to:cc:subject:message-id:mime-version :content-disposition:user-agent; bh=xCIVvIszV+VDeGHt0Mc6wMTL3EI01cgqAxFjKbA+V0A=; b=kYgduD8oVuYvaYWEN66fxrWUwlQBmcjhkvrL15ALYVfA9Oj94/Jn8uJ8TMRFVXa6S4 c4nY9o0XMfRk4JdqJZV7WU/g5PRucJTNQhcIuJaIFV7pFBaWkO56gj97eyybNcju9wo3 2AnSp/tZk8VA8TDAbH0NOeLTy52eMbu22rWUty60ilUiv8uTLv5K+Ccv6bgOa7miQoBC LI0lG5RcOLKcDk03OvfXMRWE1oDqGyNolZ/JOmxv5EAmNg+/DY9t87PpGH4JJdaBV0TT 3Zq6NAzkQaH8BxyZKLnQoMajnjB6g4lpp2e70gkQRPxV+s3xxp3wDd/z8A9KwHUc07aT Lo1g== X-Gm-Message-State: APjAAAUFwc5m0arFiAfSWvxbc8KyKHQ2s/eX/F8lZuhK+iWOxgpEHnNL hAzi+UH928Bi7ibYiHCG+xI= X-Google-Smtp-Source: APXvYqyuddVZXILTZN6z8VE5WfQLZy7z0gl4eLyGUa78uid3pgjwKpyYLqPTtgHXTtCsBEwQtYkmZQ== X-Received: by 2002:a02:ca19:: with SMTP id i25mr82770011jak.6.1563899224683; Tue, 23 Jul 2019 09:27:04 -0700 (PDT) Received: from pryzbyj (charmander.telsasoft.com. [50.244.222.1]) by smtp.gmail.com with ESMTPSA id s4sm58238591iop.25.2019.07.23.09.27.03 (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Tue, 23 Jul 2019 09:27:04 -0700 (PDT) Received: by pryzbyj (Postfix, from userid 1000) id 378688008C2; Tue, 23 Jul 2019 11:27:03 -0500 (CDT) Date: Tue, 23 Jul 2019 11:27:03 -0500 From: Justin Pryzby To: pgsql-hackers@postgresql.org Cc: Thomas Munro Subject: stress test for parallel workers Message-ID: <20190723162703.GM22387@telsasoft.com> MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline User-Agent: Mutt/1.5.24 (2015-08-30) List-Id: List-Help: List-Subscribe: List-Post: List-Owner: List-Archive: Precedence: bulk Does anyone have a stress test for parallel workers ? On a customer's new VM, I got this several times while (trying to) migrate their DB: < 2019-07-23 10:33:51.552 CDT postgres >FATAL: postmaster exited during a parallel transaction < 2019-07-23 10:33:51.552 CDT postgres >STATEMENT: CREATE UNIQUE INDEX unused0_huawei_umts_nodeb_locell_201907_unique_idx ON child.unused0_huawei_umts_nodeb_locell_201907 USING btree ... There's nothing in dmesg nor in postgres logs. At first I thought it's maybe because of a network disconnection, then I thought it's because we ran out of space (wal), then they had a faulty blade. After that, I'd tried and failed to reproduce it a number of times, but it's just recurred during what was intended to be their final restore. I've set max_parallel_workers_per_gather=0, but I'm planning to try to diagnose an issue in another instance. Ideally a minimal test, since I'm apparently going to have to run under gdb to see how it's dying, or even what process is failing. DMI: VMware, Inc. VMware Virtual Platform/440BX Desktop Reference Platform, BIOS 6.00 09/21/2015 CentOS release 6.9 (Final) Linux alextelsasrv01 2.6.32-754.17.1.el6.x86_64 #1 SMP Tue Jul 2 12:42:48 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux version | PostgreSQL 11.4 on x86_64-pc-linux-gnu, compiled by gcc (GCC) 4.4.7 20120313 (Red Hat 4.4.7-23), 64-bit Justin