Received: from malur.postgresql.org ([217.196.149.56]) by arkaria.postgresql.org with esmtps (TLS1.3:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.92) (envelope-from ) id 1lK9gz-0006Sb-75 for pgsql-hackers@arkaria.postgresql.org; Thu, 11 Mar 2021 01:00:45 +0000 Received: from localhost ([127.0.0.1] helo=malur.postgresql.org) by malur.postgresql.org with esmtp (Exim 4.92) (envelope-from ) id 1lK9gy-0001e0-3g for pgsql-hackers@arkaria.postgresql.org; Thu, 11 Mar 2021 01:00:44 +0000 Received: from magus.postgresql.org ([2a02:c0:301:0:ffff::29]) by malur.postgresql.org with esmtps (TLS1.3:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.92) (envelope-from ) id 1lK9gx-0001do-TC for pgsql-hackers@lists.postgresql.org; Thu, 11 Mar 2021 01:00:43 +0000 Received: from oss.nttdata.com ([49.212.34.109]) by magus.postgresql.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.92) (envelope-from ) id 1lK9gv-0002bR-MG for pgsql-hackers@postgresql.org; Thu, 11 Mar 2021 01:00:43 +0000 Received: from hnk.local (p1439104-ipngn8101funabasi.chiba.ocn.ne.jp [180.0.143.104]) by oss.nttdata.com (Postfix) with ESMTPSA id 84CC962600; Thu, 11 Mar 2021 10:00:37 +0900 (JST) X-Virus-Status: Clean X-Virus-Scanned: clamav-milter 0.102.3 at oss.nttdata.com Subject: Re: fdatasync performance problem with large number of DB files To: Thomas Munro , Michael Brown Cc: pgsql-hackers References: <11bc2bb7-ecb5-3ad0-b39f-df632734cd81@discourse.org> From: Fujii Masao Message-ID: Date: Thu, 11 Mar 2021 10:00:37 +0900 User-Agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10.14; rv:78.0) Gecko/20100101 Thunderbird/78.8.0 MIME-Version: 1.0 In-Reply-To: Content-Type: text/plain; charset=utf-8; format=flowed Content-Language: en-US Content-Transfer-Encoding: 7bit List-Id: List-Help: List-Subscribe: List-Post: List-Owner: List-Archive: Archived-At: Precedence: bulk On 2021/03/11 8:30, Thomas Munro wrote: > On Thu, Mar 11, 2021 at 11:38 AM Thomas Munro wrote: >> On Thu, Mar 11, 2021 at 11:01 AM Michael Brown >> wrote: >>> * is there a knob missing we can configure? >>> * can we get an opt-in knob to use a single sync() call instead of a >>> recursive fsync()? >>> * would you be open to merging a patch providing said knob? >>> * is there something else we missed? >> >> As discussed on that other thread, I don't think sync() is an option >> (it doesn't wait on all OSes or in the standard and it doesn't report >> errors). syncfs() on Linux 5.8+ looks like a good candidate though, >> and I think we'd consider a patch like that. I mean, I even posted >> one[1] in that other thread. There will of course be cases where >> that's slower (small database sharing filesystem with other software >> that has a lot of dirty data to write back). > > Thinking about this some more, if you were to propose a patch like > that syncfs() one but make it a configurable option, I'd personally be > in favour of trying to squeeze it into v14. Others might object on > commitfest procedural grounds, I dunno, but I think this is a real > operational issue and that's a fairly simple and localised change. +1 to push this kind of change into v14!! > I've run into a couple of users who have just commented that recursive > fsync() code out! BTW, we can skip that recursive fsync() by disabling fsync GUC even without commenting out the code? Regards, -- Fujii Masao Advanced Computing Technology Center Research and Development Headquarters NTT DATA CORPORATION