pg.ddx.io  pgsql-committers@postgresql.org mailing list archive  
help / color / mirror / Atom feed
From: Alvaro Herrera <alvherre@alvh.no-ip.org>
To: pgsql-committers@lists.postgresql.org
Subject: pgsql: Avoid creating archive status ".ready" files too early
Date: Mon, 23 Aug 2021 19:54:12 +0000
Message-ID: <E1mIG1M-0001cs-Ir@gemulon.postgresql.org> (raw)

Avoid creating archive status ".ready" files too early

WAL records may span multiple segments, but XLogWrite() does not
wait for the entire record to be written out to disk before
creating archive status files.  Instead, as soon as the last WAL page of
the segment is written, the archive status file is created, and the
archiver may process it.  If PostgreSQL crashes before it is able to
write and flush the rest of the record (in the next WAL segment), the
wrong version of the first segment file lingers in the archive, which
causes operations such as point-in-time restores to fail.

To fix this, keep track of records that span across segments and ensure
that segments are only marked ready-for-archival once such records have
been completely written to disk.

This has always been wrong, so backpatch all the way back.

Author: Nathan Bossart <bossartn@amazon.com>
Reviewed-by: Kyotaro Horiguchi <horikyota.ntt@gmail.com>
Reviewed-by: Ryo Matsumura <matsumura.ryo@fujitsu.com>
Reviewed-by: Andrey Borodin <x4mmm@yandex-team.ru>
Discussion: https://postgr.es/m/CBDDFA01-6E40-46BB-9F98-9340F4379505@amazon.com

Branch
------
REL_12_STABLE

Details
-------
https://git.postgresql.org/pg/commitdiff/5065aeafb0b7593c04d3bc5bc2a86037f32143fc

Modified Files
--------------
src/backend/access/transam/xlog.c  | 216 +++++++++++++++++++++++++++++++++++--
src/backend/postmaster/walwriter.c |   7 ++
src/include/access/xlog.h          |   1 +
src/include/access/xlogdefs.h      |   1 +
4 files changed, 215 insertions(+), 10 deletions(-)



view thread (7+ messages)  latest in thread

Message-ID: <E1mIG1M-0001cs-Ir@gemulon.postgresql.org>
Permalink:  ../E1mIG1M-0001cs-Ir@gemulon.postgresql.org/
Also on:    postgresql.org/message-id/E1mIG1M-0001cs-Ir@gemulon.postgresql.org

 · 

reply

Reply instructions:

You may reply publicly to this message via plain-text email
using any one of the following methods:

* Reply to all the recipients from your mail client; the link fills in
  the headers below: reply via email
  Some clients, webmail especially, drop In-Reply-To and References from
  a link, and the reply then starts a new thread: use the mbox below.

  To: pgsql-committers@postgresql.org
  Cc: alvherre@alvh.no-ip.org
  Subject: Re: pgsql: Avoid creating archive status ".ready" files too early
  In-Reply-To: <E1mIG1M-0001cs-Ir@gemulon.postgresql.org>
  References: <E1mIG1M-0001cs-Ir@gemulon.postgresql.org>

* Save the following mbox file, import it into your mail client,
  and reply-to-all from there: mbox

This inbox is served by DDX for PostgreSQL; see mirroring instructions
for how to clone and mirror all data and code used for this inbox