From: Andy Fan <zhihuifan1213@163.com>
To: Nathan Bossart <nathandbossart@gmail.com>
Cc: Justin Pryzby <pryzby@telsasoft.com>,  Maxim Orlov <orlovmg@gmail.com>,
  Pavel Borisov <pashkin.elfe@gmail.com>,  "Bossart, Nathan"
 <bossartn@amazon.com>,  Maxim Orlov <m.orlov@postgrespro.ru>,  Andres
 Freund <andres@anarazel.de>,  pgsql-hackers@lists.postgresql.org
Subject: Re: Pre-allocating WAL files
In-Reply-To: <20220408203003.GA1630183@nathanxps13> (Nathan Bossart's message
	of "Fri, 8 Apr 2022 13:30:03 -0700")
References: <20201225200953.jjkrytlrzojbndh5@alap3.anarazel.de>
	<51BAAB3B-9D3C-4155-B125-42E4F06DCDF4@amazon.com>
	<163352276350.15435.3988688227048304232.pgcf@coridan.postgresql.org>
	<CB15BEBD-98FC-4E72-86AE-513D34014176@amazon.com>
	<CALT9ZEG+R9ne6xTyEVSiLhZ7wJAQ1Jq-9XVZs=zbd_jQJyPTdA@mail.gmail.com>
	<CACG=ezb5Os_ohi1caZJLKsGJc-nSSqGXFrMjmKw4qN8fG_=NEg@mail.gmail.com>
	<20220301144044.GG25269@telsasoft.com>
	<20220301150704.GA1030078@nathanxps13>
	<20220317231212.GB1415284@nathanxps13>
	<20220408203003.GA1630183@nathanxps13>
Date: Tue, 21 Jan 2025 03:31:27 +0000
Message-ID: <87sepc4zcg.fsf@163.com>
MIME-Version: 1.0
Content-Type: text/plain
Archived-At: <https://www.postgresql.org/message-id/87sepc4zcg.fsf%40163.com>
Precedence: bulk


Hi Nathan,

Come from [0] and thanks for working on this. Here are some design
review/question after my first going through the patch.

1. walwriter vs checkpointer?  I prefer to walwriter for now because.. 

a. checkpointer is hard to do it in a timely manner either because
checkpoint itself may take a long time or the checkpoint_timeout
is much bigger than commit_delay. but walwriter could do this timely.
I think this is an important consideration for this feature. 

b. We want walwriter to run with low latency to flush out async
commits. This is true, but preallocating a wal doesn't increase the
latency too much. After all, even user uses the aysnc commit, the walfile
allocating is done by walwriter already in our current implementation.

2. How many xlogfile should be preallocated by checkpointer/walwriter
once. In your patch it is controled by wal-preallocate-max-size. How
about just preallocate *the next one* xlogfile for the simplification
purpose?

3. Why is the purpose of preallocated_segments directory? what in my
mind is we just prellocate the normal filename so that XLogWrite could
open it directly. This is same as what wal_recycle does and we can reuse
the same strategy to clean up them if they are not needed anymore.

So the poc in my mind for this feature is:
- keep track the latested reallocated (by wal_recycle or preallocated)
logfile in XLogCtl.
- logwriter check current wal insert_pos and prellocate the *next one*
walfile if it doesn't preallocated yet.
- we need to handle race condition carefully between wal_recycle, user
backend and preallocation. 

[0] https://www.postgresql.org/message-id/Z46BwCNAEjLyW85Z%40nathan

-- 
Best Regards
Andy Fan