Received: from malur.postgresql.org ([217.196.149.56]) by arkaria.postgresql.org with esmtps (TLS1.3:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.92) (envelope-from ) id 1lMmQL-0001TO-EK for pgsql-hackers@arkaria.postgresql.org; Thu, 18 Mar 2021 06:46:25 +0000 Received: from localhost ([127.0.0.1] helo=malur.postgresql.org) by malur.postgresql.org with esmtp (Exim 4.92) (envelope-from ) id 1lMmQK-0007qh-D1 for pgsql-hackers@arkaria.postgresql.org; Thu, 18 Mar 2021 06:46:24 +0000 Received: from magus.postgresql.org ([2a02:c0:301:0:ffff::29]) by malur.postgresql.org with esmtps (TLS1.3:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.92) (envelope-from ) id 1lMmQK-0007qZ-6O for pgsql-hackers@lists.postgresql.org; Thu, 18 Mar 2021 06:46:24 +0000 Received: from oss.nttdata.com ([49.212.34.109]) by magus.postgresql.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.92) (envelope-from ) id 1lMmQD-0004gV-0K for pgsql-hackers@postgresql.org; Thu, 18 Mar 2021 06:46:23 +0000 Received: from hnk.local (p2350207-ipbf2126funabasi.chiba.ocn.ne.jp [114.167.184.207]) by oss.nttdata.com (Postfix) with ESMTPSA id 496AB623C6; Thu, 18 Mar 2021 15:46:12 +0900 (JST) X-Virus-Status: Clean X-Virus-Scanned: clamav-milter 0.102.3 at oss.nttdata.com Subject: Re: fdatasync performance problem with large number of DB files To: Thomas Munro Cc: Paul Guo , Tom Lane , Michael Brown , pgsql-hackers References: <11bc2bb7-ecb5-3ad0-b39f-df632734cd81@discourse.org> <2255953.1615421770@sss.pgh.pa.us> <2258680.1615425905@sss.pgh.pa.us> <823DAB20-5A5E-4F7B-BF8C-D61FA574DEEA@vmware.com> <76de0e61-a553-6003-aeec-cb35ada791cf@oss.nttdata.com> From: Fujii Masao Message-ID: <52453bfc-40fb-7771-05ee-cbd04b6b9b6e@oss.nttdata.com> Date: Thu, 18 Mar 2021 15:46:11 +0900 User-Agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10.14; rv:78.0) Gecko/20100101 Thunderbird/78.8.1 MIME-Version: 1.0 In-Reply-To: Content-Type: text/plain; charset=utf-8; format=flowed Content-Language: en-US Content-Transfer-Encoding: 7bit List-Id: List-Help: List-Subscribe: List-Post: List-Owner: List-Archive: Archived-At: Precedence: bulk On 2021/03/17 12:45, Thomas Munro wrote: > On Tue, Mar 16, 2021 at 9:29 PM Fujii Masao wrote: >> On 2021/03/16 8:15, Thomas Munro wrote: >>> I don't want to add a hypothetical sync_after_crash=none, because it >>> seems like generally a bad idea. We already have a >>> running-with-scissors mode you could use for that: fsync=off. >> >> I heard that some backup tools sync the database directory when restoring it. >> I guess that those who use such tools might want the option to disable such >> startup sync (i.e., sync_after_crash=none) because it's not necessary. > > Hopefully syncfs() will return quickly in that case, without doing any work? Yes, in Linux. > >> They can skip that sync by fsync=off. But if they just want to skip only that >> startup sync and make subsequent recovery (or standby server) work with >> fsync=on, they would need to shutdown the server after that startup sync >> finishes, enable fsync, and restart the server. In this case, since the server >> is restarted with the state=DB_SHUTDOWNED_IN_RECOVERY, the startup sync >> would not be performed. This procedure is tricky. So IMO supporting >> sync_after_crash=none would be helpful for this case and simple. > > I still do not like this footgun :-) However, perhaps I am being > overly dogmatic. Consider the change in d8179b00, which decided that > I/O errors in this phase should be reported at LOG level rather than > ERROR. In contrast, my "sync_after_crash=wal" mode (which I need to > rebase over this) will PANIC in this case, because any syncing will be > handled through the usual checkpoint codepaths. > > Do you think it would be OK to commit this feature with just "fsync" > and "syncfs", and then to continue to consider adding "none" as a > possible separate commit? +1. "syncfs" feature is useful whether we also support "none" mode or not. It's good idea to commit "syncfs" in advance. Regards, -- Fujii Masao Advanced Computing Technology Center Research and Development Headquarters NTT DATA CORPORATION