pg.ddx.io  pgsql-hackers@postgresql.org mailing list archive  

help / color / mirror / Atom feed
From: Nathan Bossart <nathandbossart@gmail.com>
To: pgsql-hackers@postgresql.org
Subject: Re: remove more archiving overhead
Date: Tue, 22 Feb 2022 09:37:11 -0800
Message-ID: <20220222173711.GA3852671@nathanxps13> (raw)
In-Reply-To: <20220222011948.GA3850532@nathanxps13>
References: <20220222011948.GA3850532@nathanxps13>

On Mon, Feb 21, 2022 at 05:19:48PM -0800, Nathan Bossart wrote:
> I also spent some time investigating whether durably renaming the archive
> status files was even necessary.  In theory, it shouldn't be.  If a crash
> happens before the rename is persisted, we might try to re-archive files,
> but your archive_command/archive_library should handle that.  If the file
> was already recycled or removed, the archiver will skip it (thanks to
> 6d8727f).  But when digging further, I found that WAL file recyling uses
> durable_rename_excl(), which has the following note:
> 
> 	 * Note that a crash in an unfortunate moment can leave you with two links to
> 	 * the target file.
> 
> IIUC this means that in theory, a crash at an unfortunate moment could
> leave you with a .ready file, the file to archive, and another link to that
> file with a "future" WAL filename.  If you re-archive the file after it has
> been reused, you could end up with corrupt WAL archives.  I think this
> might already be possible today.  Syncing the directory after every rename
> probably reduces the likelihood quite substantially, but if
> RemoveOldXlogFiles() quickly picks up the .done file and attempts
> recycling before durable_rename() calls fsync() on the directory,
> presumably the same problem could occur.

In my testing, I found that when I killed the server just before unlink()
during WAL recyling, I ended up with links to the same file in pg_wal after
restarting.  My latest test produced links to the same file for the current
WAL file and the next one.  Maybe WAL recyling should use durable_rename()
instead of durable_rename_excl().

-- 
Nathan Bossart
Amazon Web Services: https://aws.amazon.com





view thread (40+ messages)  latest in thread

Message-ID: <20220222173711.GA3852671@nathanxps13>
Permalink:  ../20220222173711.GA3852671@nathanxps13/
Also on:    postgresql.org/message-id/20220222173711.GA3852671@nathanxps13

 · 

reply
Reply instructions:

You may reply publicly to this message via plain-text email
using any one of the following methods:

* Reply to all the recipients using the --to and --cc options:
  reply via email

  To: pgsql-hackers@postgresql.org
  Cc: nathandbossart@gmail.com
  Subject: Re: remove more archiving overhead
  In-Reply-To: <20220222173711.GA3852671@nathanxps13>

* Save the following mbox file, import it into your mail client,
  and reply-to-all from there: mbox

This inbox is served by DDX for PostgreSQL; see mirroring instructions
for how to clone and mirror all data and code used for this inbox