From: Fujii Masao <masao.fujii@oss.nttdata.com>
To: Thomas Munro <thomas.munro@gmail.com>
Cc: Paul Guo <guopa@vmware.com>
Cc: Tom Lane <tgl@sss.pgh.pa.us>
Cc: Michael Brown <michael.brown@discourse.org>
Cc: pgsql-hackers <pgsql-hackers@postgresql.org>
Subject: Re: fdatasync performance problem with large number of DB files
Date: Thu, 18 Mar 2021 15:46:11 +0900
Message-ID: <52453bfc-40fb-7771-05ee-cbd04b6b9b6e@oss.nttdata.com> (raw)
In-Reply-To: <CA+hUKGJTERHhpAja3KvwF=0qPTgnFB=QPrR=sHmqWKJ2o2mr2Q@mail.gmail.com>
References: <11bc2bb7-ecb5-3ad0-b39f-df632734cd81@discourse.org>
<CA+hUKGLt7zoFcJemmXzL=ZB3Rdse6Ds_Mm05G5Ogai_yZRp2=A@mail.gmail.com>
<CA+hUKGLgb4ZMV6xQrZb8N5ax0g+EZR=b0dtg9pPJmsBtaun1-A@mail.gmail.com>
<2255953.1615421770@sss.pgh.pa.us>
<CA+hUKGJ6bpG1ghgZccJ57fpyZRHNu2xJkWRmaEP4TEBuXutS-Q@mail.gmail.com>
<2258680.1615425905@sss.pgh.pa.us>
<CA+hUKGLkcJ=9qD69NU0W6SEz-JGzdDtjjF2TedoYKgD2ky=atg@mail.gmail.com>
<CA+hUKGLABi5asn5yFXBrAC0d3dGBN8QHvLq=27sa7thCQV4Z1w@mail.gmail.com>
<CA+hUKGK0hZ1jBCF6r+J9=CVwpAUTU7EB3_SUp5Owp0zym4eksg@mail.gmail.com>
<823DAB20-5A5E-4F7B-BF8C-D61FA574DEEA@vmware.com>
<CA+hUKGJi11x6cHJQNeDnLRvk1GMX18DJQaKFqwCOjgeFnbiVKA@mail.gmail.com>
<76de0e61-a553-6003-aeec-cb35ada791cf@oss.nttdata.com>
<CA+hUKGJTERHhpAja3KvwF=0qPTgnFB=QPrR=sHmqWKJ2o2mr2Q@mail.gmail.com>
On 2021/03/17 12:45, Thomas Munro wrote:
> On Tue, Mar 16, 2021 at 9:29 PM Fujii Masao <masao.fujii@oss.nttdata.com> wrote:
>> On 2021/03/16 8:15, Thomas Munro wrote:
>>> I don't want to add a hypothetical sync_after_crash=none, because it
>>> seems like generally a bad idea. We already have a
>>> running-with-scissors mode you could use for that: fsync=off.
>>
>> I heard that some backup tools sync the database directory when restoring it.
>> I guess that those who use such tools might want the option to disable such
>> startup sync (i.e., sync_after_crash=none) because it's not necessary.
>
> Hopefully syncfs() will return quickly in that case, without doing any work?
Yes, in Linux.
>
>> They can skip that sync by fsync=off. But if they just want to skip only that
>> startup sync and make subsequent recovery (or standby server) work with
>> fsync=on, they would need to shutdown the server after that startup sync
>> finishes, enable fsync, and restart the server. In this case, since the server
>> is restarted with the state=DB_SHUTDOWNED_IN_RECOVERY, the startup sync
>> would not be performed. This procedure is tricky. So IMO supporting
>> sync_after_crash=none would be helpful for this case and simple.
>
> I still do not like this footgun :-) However, perhaps I am being
> overly dogmatic. Consider the change in d8179b00, which decided that
> I/O errors in this phase should be reported at LOG level rather than
> ERROR. In contrast, my "sync_after_crash=wal" mode (which I need to
> rebase over this) will PANIC in this case, because any syncing will be
> handled through the usual checkpoint codepaths.
>
> Do you think it would be OK to commit this feature with just "fsync"
> and "syncfs", and then to continue to consider adding "none" as a
> possible separate commit?
+1. "syncfs" feature is useful whether we also support "none" mode or not.
It's good idea to commit "syncfs" in advance.
Regards,
--
Fujii Masao
Advanced Computing Technology Center
Research and Development Headquarters
NTT DATA CORPORATION
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Reply to all the recipients using the --to and --cc options:
reply via email
To: pgsql-hackers@postgresql.org
Cc: masao.fujii@oss.nttdata.com, thomas.munro@gmail.com, guopa@vmware.com, tgl@sss.pgh.pa.us, michael.brown@discourse.org
Subject: Re: fdatasync performance problem with large number of DB files
In-Reply-To: <52453bfc-40fb-7771-05ee-cbd04b6b9b6e@oss.nttdata.com>
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
This inbox is served by DDX for PostgreSQL; see mirroring instructions
for how to clone and mirror all data and code used for this inbox