From: Nathan Bossart <nathandbossart@gmail.com>
To: Andy Fan <zhihuifan1213@163.com>
Cc: Andres Freund <andres@anarazel.de>
Cc: Justin Pryzby <pryzby@telsasoft.com>
Cc: Maxim Orlov <orlovmg@gmail.com>
Cc: Pavel Borisov <pashkin.elfe@gmail.com>
Cc: Bossart, Nathan <bossartn@amazon.com>
Cc: Maxim Orlov <m.orlov@postgrespro.ru>
Cc: pgsql-hackers@lists.postgresql.org
Subject: Re: Pre-allocating WAL files
Date: Wed, 22 Jan 2025 09:56:33 -0600
Message-ID: <Z5EVMXSWGN7_ViZ7@nathan> (raw)
In-Reply-To: <87o6zz4pld.fsf@163.com>
References: <CACG=ezb5Os_ohi1caZJLKsGJc-nSSqGXFrMjmKw4qN8fG_=NEg@mail.gmail.com>
<20220301144044.GG25269@telsasoft.com>
<20220301150704.GA1030078@nathanxps13>
<20220317231212.GB1415284@nathanxps13>
<20220408203003.GA1630183@nathanxps13>
<87sepc4zcg.fsf@163.com>
<Z4_C04ZJ_SymvYes@nathan>
<Z4_Hmj2R8NXnAI-K@nathan>
<vfhyldq3rtnwd4uxbb7ro2opmvbmfswvkycavmzcqb55mmo4jk@g2d2n6xnwicb>
<87o6zz4pld.fsf@163.com>
On Wed, Jan 22, 2025 at 01:14:22AM +0000, Andy Fan wrote:
> Andres Freund <andres@anarazel.de> writes:
>> FWIW, I've seen the fsyncs around recycling being a rather substantial
>> bottleneck. To the point of the main benefit of larger segments being the
>> reduction in number of fsyncs at the end of a checkpoint. I think we should
>> be able to make the fsyncs a lot more efficient by batching them, first rename
>> a bunch of files, then fsync them and the directory. The current pattern
>> bascially requires a separate filesystem jouranl flush for each WAL segment.
>
> For education purpose, how to fsync files in batch? 'man fsync' tells me
> user can only fsync one file each time.
>
> int fsync(int fd);
>
> The fsync manual seems not saying fsync on a directory would fsync all
> the files under that directory.
I think Andres means that we should wait until the end of recycling to
fsync() the directory so that we aren't flushing it for every single
recycled segment. This sort of batching approach could also work well with
pre_sync_fname(), so that by the time we actually call fsync() on the
files, it has very little to do.
--
nathan
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Reply to all the recipients using the --to and --cc options:
reply via email
To: pgsql-hackers@postgresql.org
Cc: nathandbossart@gmail.com, zhihuifan1213@163.com, andres@anarazel.de, pryzby@telsasoft.com, orlovmg@gmail.com, pashkin.elfe@gmail.com, bossartn@amazon.com, m.orlov@postgrespro.ru, pgsql-hackers@lists.postgresql.org
Subject: Re: Pre-allocating WAL files
In-Reply-To: <Z5EVMXSWGN7_ViZ7@nathan>
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
This inbox is served by DDX for PostgreSQL; see mirroring instructions
for how to clone and mirror all data and code used for this inbox