Received: from malur.postgresql.org ([217.196.149.56]) by arkaria.postgresql.org with esmtps (TLS1.3:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.92) (envelope-from ) id 1megE3-0004zB-CK for pgsql-bugs@arkaria.postgresql.org; Sun, 24 Oct 2021 16:19:59 +0000 Received: from localhost ([127.0.0.1] helo=malur.postgresql.org) by malur.postgresql.org with esmtp (Exim 4.92) (envelope-from ) id 1megE0-0000gy-B5 for pgsql-bugs@arkaria.postgresql.org; Sun, 24 Oct 2021 16:19:56 +0000 Received: from magus.postgresql.org ([2a02:c0:301:0:ffff::29]) by malur.postgresql.org with esmtps (TLS1.3:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.92) (envelope-from ) id 1megDz-0000gp-Sf for pgsql-bugs@lists.postgresql.org; Sun, 24 Oct 2021 16:19:56 +0000 Received: from mail-qk1-x72b.google.com ([2607:f8b0:4864:20::72b]) by magus.postgresql.org with esmtps (TLS1.3:ECDHE_RSA_AES_128_GCM_SHA256:128) (Exim 4.92) (envelope-from ) id 1megDs-0007jZ-By for pgsql-bugs@lists.postgresql.org; Sun, 24 Oct 2021 16:19:55 +0000 Received: by mail-qk1-x72b.google.com with SMTP id h20so9482346qko.13 for ; Sun, 24 Oct 2021 09:19:47 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=leadboat.com; s=google; h=date:from:to:cc:subject:message-id:references:mime-version :content-disposition:content-transfer-encoding:in-reply-to :user-agent; bh=Bth76wk2/VGxwZUsZ/HGgwSPwxpHyTcaZt+BTRNeSX8=; b=fepegWMG7YBeiGZnm92f3UxJIWoVVyPLAGYzqOpuEeoMpMBNeYnYizwxtzoIi3lcAy zKU3oIoPDWZgaP2h5mCPRJk+yG48HugHXx26pg5u6Z/3YdPHNiidasczWSzSUH6hFQAx VPN0lKPJXueJ23P8lPHUyZDR+U+FNvOTi2Vtg= X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:date:from:to:cc:subject:message-id:references :mime-version:content-disposition:content-transfer-encoding :in-reply-to:user-agent; bh=Bth76wk2/VGxwZUsZ/HGgwSPwxpHyTcaZt+BTRNeSX8=; b=cm4XJEd803b6qnLJH8xF0bNMUowvl2yfdSDX3UdV5N71sAHSIQLUnHzGCx1xc3zVv+ gl1lWlAT8FbM3OUwNYsO2bW9OGnh43+NLHmJt6ocXy9AXqrNj1+uOP0PXmh0GDCvdlSl 2m0B6O4YYz3vY3GQZO5VYwu7aGfzQCDLOG7XGEzSdw54wPPv6D2+0xtmnACfY5Vb0MlN BCtYytGKcfjPDFGS0zqGcWp9pzgvcOvRJ+0jyFA1nWEqMWnKv/joiHnEK4Kb+ixjcN3g lSTzlNqY+6KMUOlW4zBP2v7W29SG8FzIFz0O3ypt9V3VxYLRloMG/pCEunxQWlfYDNF8 TXBw== X-Gm-Message-State: AOAM533BjMgAaQd2buQxKHRjKZ4Zlc0amH9OU8Itzfgu/nh/LGYli6Ee +YgYmA1FIyepoTZYmYWPIUHBnnyIGUINtw== X-Google-Smtp-Source: ABdhPJyh8uh9m5vqTaKc5MHfbec1Hdjng85y4+mWlkeRhb7IKHXZgncwUsBH85FaHDaAadgzLzPBwA== X-Received: by 2002:a05:620a:1924:: with SMTP id bj36mr9661923qkb.422.1635092385977; Sun, 24 Oct 2021 09:19:45 -0700 (PDT) Received: from rfd.leadboat.com ([2600:1702:a20:5750::46]) by smtp.gmail.com with ESMTPSA id bp9sm7845018qkb.94.2021.10.24.09.19.44 (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Sun, 24 Oct 2021 09:19:45 -0700 (PDT) Date: Sun, 24 Oct 2021 09:19:42 -0700 From: Noah Misch To: Andrey Borodin Cc: PostgreSQL mailing lists , Michael Paquier , Tom Lane , Peter Geoghegan , Andres Freund Subject: conchuela timeouts since 2021-10-09 system upgrade Message-ID: <20211024161942.GB3945842@rfd.leadboat.com> References: <364BAA98-1A3F-4C39-B0D9-ED8C40EA902D@yandex-team.ru> <20210908064515.GA2496820@rfd.leadboat.com> <20210920044143.GA3414844@rfd.leadboat.com> <593C6C66-F19E-412A-9896-F1F5B272424A@yandex-team.ru> <20210925201012.GC4134968@rfd.leadboat.com> <20211017151205.GA2384251@rfd.leadboat.com> <9BA43784-E4A7-4638-900C-FC9868C388FA@yandex-team.ru> <20211019030212.GB3403374@rfd.leadboat.com> <20211024050028.GA3310519@rfd.leadboat.com> <64835CF9-D9AA-49BF-A685-01C23B1023C1@yandex-team.ru> MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Disposition: inline Content-Transfer-Encoding: 8bit In-Reply-To: <64835CF9-D9AA-49BF-A685-01C23B1023C1@yandex-team.ru> User-Agent: Mutt/1.5.24 (2015-08-30) List-Id: List-Help: List-Subscribe: List-Post: List-Owner: List-Archive: Archived-At: Precedence: bulk On Sun, Oct 24, 2021 at 02:45:38PM +0300, Andrey Borodin wrote: > > 24 окт. 2021 г., в 08:00, Noah Misch написал(а): > > Buildfarm member conchuela (DragonFly BSD 6.0) has gotten multiple > > "IPC::Run: timeout on timer" in the new tests. No other animal has. > > https://buildfarm.postgresql.org/cgi-bin/show_log.pl?nm=conchuela&dt=2021-10-24%2003%3A05%3A09 > > is an example run. The pgbench queries finished quickly, but the > > $pgbench_h->finish() apparently timed out after 180s. I guess this would be > > consistent with pgbench blocking in write(), waiting for something to empty a > > pipe buffer so it can write more. I thought finish() will drain any incoming > > I/O, though. This phenomenon has been appearing regularly via > > src/test/recovery/t/017_shm.pl[1], so this thread doesn't have a duty to > > resolve it. A stack trace of the stuck pgbench should be informative, though. > > Some thoughts: > 0. I doubt that psql\pgbench is stuck in these failures. Got it. If pgbench is a zombie, the fault does lie in IPC::Run or the kernel. > 1. All observed similar failures seem to be related to finish() sub of IPC::Run harness > 2. Finish must pump any pending data from process [0]. But it can hang if process is waiting for something. > 3. There is reported bug of finish [1]. But the description is slightly different. Since that report is about a Perl-process child on Linux, I think we can treat it as unrelated. These failures started on 2021-10-09, the day conchuela updated from DragonFly v4.4.3-RELEASE to DragonFly v6.0.0-RELEASE. It smells like a kernel bug. Since the theorized kernel bug seems not to affect src/test/subscription/t/015_stream.pl, I wonder if we can borrow a workaround from other tests. One thing in common with src/test/recovery/t/017_shm.pl and the newest failure sites is that they don't write anything to the child stdin. Does writing e.g. a single byte (that the child doesn't use) work around the problem? If not, does passing the script via stdin, like "pgbench -f- [0] https://metacpan.org/dist/IPC-Run/source/lib/IPC/Run.pm#L3481 > [1] https://github.com/toddr/IPC-Run/issues/57