From: Nathan Bossart <nathandbossart@gmail.com>
To: Andy Fan <zhihuifan1213@163.com>
Cc: Justin Pryzby <pryzby@telsasoft.com>
Cc: Maxim Orlov <orlovmg@gmail.com>
Cc: Pavel Borisov <pashkin.elfe@gmail.com>
Cc: Bossart, Nathan <bossartn@amazon.com>
Cc: Maxim Orlov <m.orlov@postgrespro.ru>
Cc: Andres Freund <andres@anarazel.de>
Cc: pgsql-hackers@lists.postgresql.org
Subject: Re: Pre-allocating WAL files
Date: Tue, 21 Jan 2025 09:52:51 -0600
Message-ID: <Z4_C04ZJ_SymvYes@nathan> (raw)
In-Reply-To: <87sepc4zcg.fsf@163.com>
References: <51BAAB3B-9D3C-4155-B125-42E4F06DCDF4@amazon.com>
<163352276350.15435.3988688227048304232.pgcf@coridan.postgresql.org>
<CB15BEBD-98FC-4E72-86AE-513D34014176@amazon.com>
<CALT9ZEG+R9ne6xTyEVSiLhZ7wJAQ1Jq-9XVZs=zbd_jQJyPTdA@mail.gmail.com>
<CACG=ezb5Os_ohi1caZJLKsGJc-nSSqGXFrMjmKw4qN8fG_=NEg@mail.gmail.com>
<20220301144044.GG25269@telsasoft.com>
<20220301150704.GA1030078@nathanxps13>
<20220317231212.GB1415284@nathanxps13>
<20220408203003.GA1630183@nathanxps13>
<87sepc4zcg.fsf@163.com>
On Tue, Jan 21, 2025 at 03:31:27AM +0000, Andy Fan wrote:
> Come from [0] and thanks for working on this. Here are some design
> review/question after my first going through the patch.
Thanks for taking a look.
> 1. walwriter vs checkpointer? I prefer to walwriter for now because..
>
> a. checkpointer is hard to do it in a timely manner either because
> checkpoint itself may take a long time or the checkpoint_timeout
> is much bigger than commit_delay. but walwriter could do this timely.
> I think this is an important consideration for this feature.
>
> b. We want walwriter to run with low latency to flush out async
> commits. This is true, but preallocating a wal doesn't increase the
> latency too much. After all, even user uses the aysnc commit, the walfile
> allocating is done by walwriter already in our current implementation.
I attempted to deal with this by having pre-allocation requests set the
checkpointer's latch and performing the pre-allocation within the
checkpointer's main loop and during write delays. However, checkpointing
does a number of other things that could just as easily delay
pre-allocation, so it's probably worth considering the WAL writer.
> 2. How many xlogfile should be preallocated by checkpointer/walwriter
> once. In your patch it is controled by wal-preallocate-max-size. How
> about just preallocate *the next one* xlogfile for the simplification
> purpose?
We could probably start with something like that. IIRC it was difficult to
create workloads where you'd need more than 1-2 at a time, provided
whatever is pre-allocating refills the pool quickly.
> 3. Why is the purpose of preallocated_segments directory? what in my
> mind is we just prellocate the normal filename so that XLogWrite could
> open it directly. This is same as what wal_recycle does and we can reuse
> the same strategy to clean up them if they are not needed anymore.
The purpose is to limit the use of pre-allocated segments to only
situations where WAL recycling is not sufficient. Basically, if writing a
record would require a new segment to be created, we can quickly pull a
pre-allocated one instead of creating it ourselves. Besides simplifying
matters, this prevents a lot of unnecessary pre-allocation, since many
workloads will almost never need anything beyond the recycled segments.
--
nathan
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Reply to all the recipients using the --to and --cc options:
reply via email
To: pgsql-hackers@postgresql.org
Cc: nathandbossart@gmail.com, zhihuifan1213@163.com, pryzby@telsasoft.com, orlovmg@gmail.com, pashkin.elfe@gmail.com, bossartn@amazon.com, m.orlov@postgrespro.ru, andres@anarazel.de, pgsql-hackers@lists.postgresql.org
Subject: Re: Pre-allocating WAL files
In-Reply-To: <Z4_C04ZJ_SymvYes@nathan>
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
This inbox is served by DDX for PostgreSQL; see mirroring instructions
for how to clone and mirror all data and code used for this inbox