pg.ddx.io  pgsql-committers@postgresql.org mailing list archive  
help / color / mirror / Atom feed
From: Thomas Munro <tmunro@postgresql.org>
To: pgsql-committers@lists.postgresql.org
Subject: pgsql: Add circular WAL decoding buffer, take II.
Date: Fri, 18 Mar 2022 05:49:10 +0000
Message-ID: <E1nV5U6-000EF7-6r@gemulon.postgresql.org> (raw)

Add circular WAL decoding buffer, take II.

Teach xlogreader.c to decode the WAL into a circular buffer.  This will
support optimizations based on looking ahead, to follow in a later
commit.

 * XLogReadRecord() works as before, decoding records one by one, and
   allowing them to be examined via the traditional XLogRecGetXXX()
   macros and certain traditional members like xlogreader->ReadRecPtr.

 * An alternative new interface XLogReadAhead()/XLogNextRecord() is
   added that returns pointers to DecodedXLogRecord objects so that it's
   now possible to look ahead in the WAL stream while replaying.

 * In order to be able to use the new interface effectively while
   streaming data, support is added for the page_read() callback to
   respond to a new nonblocking mode with XLREAD_WOULDBLOCK instead of
   waiting for more data to arrive.

No direct user of the new interface is included in this commit, though
XLogReadRecord() uses it internally.  Existing code doesn't need to
change, except in a few places where it was accessing reader internals
directly and now needs to go through accessor macros.

Reviewed-by: Julien Rouhaud <rjuju123@gmail.com>
Reviewed-by: Tomas Vondra <tomas.vondra@enterprisedb.com>
Reviewed-by: Andres Freund <andres@anarazel.de> (earlier versions)
Discussion: https://postgr.es/m/CA+hUKGJ4VJN8ttxScUFM8dOKX0BrBiboo5uz1cq=AovOddfHpA@mail.gmail.com

Branch
------
master

Details
-------
https://git.postgresql.org/pg/commitdiff/3f1ce973467a0d285961bf2f99b11d06e264e2c1

Modified Files
--------------
src/backend/access/transam/generic_xlog.c |   6 +-
src/backend/access/transam/xlog.c         |  18 +-
src/backend/access/transam/xlogreader.c   | 645 ++++++++++++++++++++++++------
src/backend/access/transam/xlogrecovery.c |   4 +-
src/backend/access/transam/xlogutils.c    |   2 +-
src/backend/replication/logical/decode.c  |   2 +-
src/bin/pg_rewind/parsexlog.c             |   2 +-
src/bin/pg_waldump/pg_waldump.c           |  25 +-
src/include/access/xlogreader.h           | 154 +++++--
src/tools/pgindent/typedefs.list          |   2 +
10 files changed, 686 insertions(+), 174 deletions(-)



Message-ID: <E1nV5U6-000EF7-6r@gemulon.postgresql.org>
Permalink:  ../E1nV5U6-000EF7-6r@gemulon.postgresql.org/
Also on:    postgresql.org/message-id/E1nV5U6-000EF7-6r@gemulon.postgresql.org

 · 

reply

Reply instructions:

You may reply publicly to this message via plain-text email
using any one of the following methods:

* Reply to all the recipients from your mail client; the link fills in
  the headers below: reply via email
  Some clients, webmail especially, drop In-Reply-To and References from
  a link, and the reply then starts a new thread: use the mbox below.

  To: pgsql-committers@postgresql.org
  Cc: tmunro@postgresql.org
  Subject: Re: pgsql: Add circular WAL decoding buffer, take II.
  In-Reply-To: <E1nV5U6-000EF7-6r@gemulon.postgresql.org>
  References: <E1nV5U6-000EF7-6r@gemulon.postgresql.org>

* Save the following mbox file, import it into your mail client,
  and reply-to-all from there: mbox

This inbox is served by DDX for PostgreSQL; see mirroring instructions
for how to clone and mirror all data and code used for this inbox