public inbox for [email protected]  

help / color / mirror / Atom feed

From: Andy Fan <[email protected]>
To: Nathan Bossart <[email protected]>
Cc: Justin Pryzby <[email protected]>
Cc: Maxim Orlov <[email protected]>
Cc: Pavel Borisov <[email protected]>
Cc: Bossart, Nathan <[email protected]>
Cc: Maxim Orlov <[email protected]>
Cc: Andres Freund <[email protected]>
Cc: [email protected]
Subject: Re: Pre-allocating WAL files
Date: Tue, 21 Jan 2025 03:31:27 +0000
Message-ID: <[email protected]> (raw)
In-Reply-To: <20220408203003.GA1630183@nathanxps13>
References: <[email protected]>
	<[email protected]>
	<163352276350.15435.3988688227048304232.pgcf@coridan.postgresql.org>
	<[email protected]>
	<CALT9ZEG+R9ne6xTyEVSiLhZ7wJAQ1Jq-9XVZs=zbd_jQJyPTdA@mail.gmail.com>
	<CACG=ezb5Os_ohi1caZJLKsGJc-nSSqGXFrMjmKw4qN8fG_=NEg@mail.gmail.com>
	<[email protected]>
	<20220301150704.GA1030078@nathanxps13>
	<20220317231212.GB1415284@nathanxps13>
	<20220408203003.GA1630183@nathanxps13>


Hi Nathan,

Come from [0] and thanks for working on this. Here are some design
review/question after my first going through the patch.

1. walwriter vs checkpointer?  I prefer to walwriter for now because.. 

a. checkpointer is hard to do it in a timely manner either because
checkpoint itself may take a long time or the checkpoint_timeout
is much bigger than commit_delay. but walwriter could do this timely.
I think this is an important consideration for this feature. 

b. We want walwriter to run with low latency to flush out async
commits. This is true, but preallocating a wal doesn't increase the
latency too much. After all, even user uses the aysnc commit, the walfile
allocating is done by walwriter already in our current implementation.

2. How many xlogfile should be preallocated by checkpointer/walwriter
once. In your patch it is controled by wal-preallocate-max-size. How
about just preallocate *the next one* xlogfile for the simplification
purpose?

3. Why is the purpose of preallocated_segments directory? what in my
mind is we just prellocate the normal filename so that XLogWrite could
open it directly. This is same as what wal_recycle does and we can reuse
the same strategy to clean up them if they are not needed anymore.

So the poc in my mind for this feature is:
- keep track the latested reallocated (by wal_recycle or preallocated)
logfile in XLogCtl.
- logwriter check current wal insert_pos and prellocate the *next one*
walfile if it doesn't preallocated yet.
- we need to handle race condition carefully between wal_recycle, user
backend and preallocation. 

[0] https://www.postgresql.org/message-id/Z46BwCNAEjLyW85Z%40nathan

-- 
Best Regards
Andy Fan

view thread (17+ messages)  latest in thread

reply

Reply instructions:

You may reply publicly to this message via plain-text email
using any one of the following methods:

* Reply to all the recipients using the --to and --cc options:
  reply via email

  To: [email protected]
  Cc: [email protected], [email protected], [email protected], [email protected], [email protected], [email protected], [email protected], [email protected], [email protected]
  Subject: Re: Pre-allocating WAL files
  In-Reply-To: <[email protected]>

* Save the following mbox file, import it into your mail client,
  and reply-to-all from there: mbox

This inbox is served by agora; see mirroring instructions
for how to clone and mirror all data and code used for this inbox