Received: from malur.postgresql.org ([217.196.149.56]) by arkaria.postgresql.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_CBC_SHA1:256) (Exim 4.89) (envelope-from ) id 1hq4DW-0004iP-SI for pgsql-hackers@arkaria.postgresql.org; Tue, 23 Jul 2019 23:29:11 +0000 Received: from localhost ([127.0.0.1] helo=malur.postgresql.org) by malur.postgresql.org with esmtp (Exim 4.89) (envelope-from ) id 1hq4DV-0001Ro-Kx for pgsql-hackers@arkaria.postgresql.org; Tue, 23 Jul 2019 23:29:09 +0000 Received: from magus.postgresql.org ([2a02:c0:301:0:ffff::29]) by malur.postgresql.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_CBC_SHA1:256) (Exim 4.89) (envelope-from ) id 1hq4DV-0001Rh-CU for pgsql-hackers@lists.postgresql.org; Tue, 23 Jul 2019 23:29:09 +0000 Received: from sss.pgh.pa.us ([66.207.139.130]) by magus.postgresql.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_CBC_SHA1:256) (Exim 4.89) (envelope-from ) id 1hq4DT-0002Hm-Ad for pgsql-hackers@postgresql.org; Tue, 23 Jul 2019 23:29:09 +0000 Received: from sss1.sss.pgh.pa.us (localhost [127.0.0.1]) by sss.pgh.pa.us (8.14.4/8.14.4) with ESMTP id x6NNT5Il002951; Tue, 23 Jul 2019 19:29:05 -0400 From: Tom Lane To: Justin Pryzby cc: Thomas Munro , pgsql-hackers Subject: Re: stress test for parallel workers In-reply-to: <20190723230440.GU22387@telsasoft.com> References: <20190723162703.GM22387@telsasoft.com> <6762.1563902927@sss.pgh.pa.us> <20190723174222.GO22387@telsasoft.com> <20190723224255.GT22387@telsasoft.com> <20190723230440.GU22387@telsasoft.com> Comments: In-reply-to Justin Pryzby message dated "Tue, 23 Jul 2019 18:04:40 -0500" MIME-Version: 1.0 Content-Type: text/plain; charset="us-ascii" Content-ID: <2949.1563924544.1@sss.pgh.pa.us> Date: Tue, 23 Jul 2019 19:29:04 -0400 Message-ID: <2950.1563924544@sss.pgh.pa.us> List-Id: List-Help: List-Subscribe: List-Post: List-Owner: List-Archive: Precedence: bulk Justin Pryzby writes: > I want to say I'm almost certain it wasn't ENOSPC in other cases, since, > failing to find log output, I ran df right after the failure. The fact that you're not finding log output matching what was reported to the client seems to me to be a mighty strong indication that there *was* an ENOSPC problem. Can you reconfigure to put the postmaster log on a different volume? > But that gives me an idea: is it possible there's an issue with files being > held opened by worker processes ? Including by parallel workers? Probably > WALs, even after they're rotated ? If there were worker processes holding > opened lots of rotated WALs, that could cause ENOSPC, but that wouldn't be > obvious after they die, since the space would then be freed. Parallel workers aren't ever allowed to write, in the current implementation, so it's not real obvious why they'd have any WAL log files open at all. regards, tom lane