Received: from malur.postgresql.org ([217.196.149.56]) by arkaria.postgresql.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_CBC_SHA1:256) (Exim 4.89) (envelope-from ) id 1iZSM3-0003gA-Mk for pgsql-hackers@arkaria.postgresql.org; Tue, 26 Nov 2019 04:21:36 +0000 Received: from localhost ([127.0.0.1] helo=malur.postgresql.org) by malur.postgresql.org with esmtp (Exim 4.89) (envelope-from ) id 1iZSM1-000164-N9 for pgsql-hackers@arkaria.postgresql.org; Tue, 26 Nov 2019 04:21:33 +0000 Received: from magus.postgresql.org ([2a02:c0:301:0:ffff::29]) by malur.postgresql.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_CBC_SHA1:256) (Exim 4.89) (envelope-from ) id 1iZSM1-00015x-A6 for pgsql-hackers@lists.postgresql.org; Tue, 26 Nov 2019 04:21:33 +0000 Received: from mail-yw1-xc42.google.com ([2607:f8b0:4864:20::c42]) by magus.postgresql.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_CBC_SHA1:256) (Exim 4.89) (envelope-from ) id 1iZSLx-0001tB-E8 for pgsql-hackers@postgresql.org; Tue, 26 Nov 2019 04:21:32 +0000 Received: by mail-yw1-xc42.google.com with SMTP id n82so6399723ywc.7 for ; Mon, 25 Nov 2019 20:21:28 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=telsasoft-com.20150623.gappssmtp.com; s=20150623; h=date:from:to:cc:subject:message-id:references:mime-version :content-disposition:in-reply-to:user-agent; bh=IZhan1ukWtAKbTsJsHD8IwRceyF8m7n1L4QbU1+g5e8=; b=YJbg6ONDjLr6grZt+ohWcRZCNZOlIiNYXwHAYg+McC2JIwwXgdPR0MaaXcpeuK7ZIC 60BIx6tNjME+5V8dZfxOOGS/tncMBxerCHQi9isARL2dVNmnHxQ8jUFYaSi14noClU5T hG2ZeJwjjcEbgovQj2HXUSRMD6tiVpqr8nRW+iQ2I39Fe1pzUKOqLK3Wx8Dp9kUU52/v 7oUIGKxbzw9r93eYxiH+xqGszLelt+2nK5OO3axFxGVA1WD0yrMEh0JyWNlSiF35o3xT a3p7Iix9aOXv5lrgI8Av0xP/+UFOw4BZntNU30yNhX9Z8sAMQA5+cYRbTwOBkX2aNauI xuyw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:date:from:to:cc:subject:message-id:references :mime-version:content-disposition:in-reply-to:user-agent; bh=IZhan1ukWtAKbTsJsHD8IwRceyF8m7n1L4QbU1+g5e8=; b=oIdvpsyeg8OlwTuoBZlmgXr90pq6YP3/YpiXjnPlkCCO6VqzBQFmibS8VpSrxu11ou EGQPsKhFCjfkKtOMQzcXtguF9DTxSsfNFMKPurXK4GPfLs7WnO60CEY1h1qlYY/lpQD+ DGw7bJRSIfGZcvbY6ZfiHZHwgHYu2Zn3QqLTJDVzJEwEBPqOsRu5hIkGWk5wNwNkWhi9 EVTbDg7k1fpzJWqEKTJypQdEK6kTtiuh4mh7sC8FjStkn13h8j7v7kRlhtqTakce544P eml8lOz7egYsyKRYBG/07+ioeaLvK7aQOY4eEnUYUBzexEXVECE8ab1bFd0eTgrVqs6b wzgg== X-Gm-Message-State: APjAAAXSIHa7/0LS0ZVxifwyx8Bz1hLxE+OUEw6mSGff+GloPW2iAzna RmYqbNTXp7ADCRSsi8xoQwGBZg== X-Google-Smtp-Source: APXvYqyP5+kcvYIfpPiFWtUl9859IzD2e9xYZ8d0SvE/YvjQURcSfUXLIn+LXEYH1oqTCzkGmlQ40g== X-Received: by 2002:a0d:d681:: with SMTP id y123mr23176666ywd.240.1574742087002; Mon, 25 Nov 2019 20:21:27 -0800 (PST) Received: from pryzbyj (charmander.telsasoft.com. [50.244.222.1]) by smtp.gmail.com with ESMTPSA id l127sm4840437ywb.21.2019.11.25.20.21.25 (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Mon, 25 Nov 2019 20:21:26 -0800 (PST) Received: by pryzbyj (Postfix, from userid 1000) id A8CF4800A68; Mon, 25 Nov 2019 22:21:24 -0600 (CST) Date: Mon, 25 Nov 2019 22:21:24 -0600 From: Justin Pryzby To: Thomas Munro Cc: pgsql-hackers@postgresql.org Subject: Re: checkpointer: PANIC: could not fsync file: No such file or directory Message-ID: <20191126042124.GA3111@telsasoft.com> References: <20191119115759.GI30362@telsasoft.com> <20191119224910.GU30362@telsasoft.com> <20191120012226.GW30362@telsasoft.com> MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: <20191120012226.GW30362@telsasoft.com> User-Agent: Mutt/1.5.24 (2015-08-30) List-Id: List-Help: List-Subscribe: List-Post: List-Owner: List-Archive: Precedence: bulk I looked and found a new "hint". On Tue, Nov 19, 2019 at 05:57:59AM -0600, Justin Pryzby wrote: > < 2019-11-15 22:16:07.098 EST >PANIC: could not fsync file "base/16491/1731839470.2": No such file or directory > < 2019-11-15 22:16:08.751 EST >LOG: checkpointer process (PID 27388) was terminated by signal 6: Aborted An earlier segment of that relation had been opened successfully and was *still* opened: $ sudo grep 1731839470 /var/spool/abrt/ccpp-2019-11-15-22:16:08-27388/open_fds 63:/var/lib/pgsql/12/data/base/16491/1731839470 For context: $ sudo grep / /var/spool/abrt/ccpp-2019-11-15-22:16:08-27388/open_fds |tail -3 61:/var/lib/pgsql/12/data/base/16491/1757077748 62:/var/lib/pgsql/12/data/base/16491/1756223121.2 63:/var/lib/pgsql/12/data/base/16491/1731839470 So this may be an issue only with relations>segment (but, that interpretation could also be very naive).