From: Thomas Munro <tmunro@postgresql.org>
To: pgsql-committers@lists.postgresql.org
Subject: pgsql: Prefetch data referenced by the WAL, take II.
Date: Thu, 07 Apr 2022 07:44:20 +0000
Message-ID: <E1ncMoW-000hhv-4s@gemulon.postgresql.org> (raw)
Prefetch data referenced by the WAL, take II.
Introduce a new GUC recovery_prefetch. When enabled, look ahead in the
WAL and try to initiate asynchronous reading of referenced data blocks
that are not yet cached in our buffer pool. For now, this is done with
posix_fadvise(), which has several caveats. Since not all OSes have
that system call, "try" is provided so that it can be enabled where
available. Better mechanisms for asynchronous I/O are possible in later
work.
Set to "try" for now for test coverage. Default setting to be finalized
before release.
The GUC wal_decode_buffer_size limits the distance we can look ahead in
bytes of decoded data.
The existing GUC maintenance_io_concurrency is used to limit the number
of concurrent I/Os allowed, based on pessimistic heuristics used to
infer that I/Os have begun and completed. We'll also not look more than
maintenance_io_concurrency * 4 block references ahead.
Reviewed-by: Julien Rouhaud <rjuju123@gmail.com>
Reviewed-by: Tomas Vondra <tomas.vondra@2ndquadrant.com>
Reviewed-by: Alvaro Herrera <alvherre@2ndquadrant.com> (earlier version)
Reviewed-by: Andres Freund <andres@anarazel.de> (earlier version)
Reviewed-by: Justin Pryzby <pryzby@telsasoft.com> (earlier version)
Tested-by: Tomas Vondra <tomas.vondra@2ndquadrant.com> (earlier version)
Tested-by: Jakub Wartak <Jakub.Wartak@tomtom.com> (earlier version)
Tested-by: Dmitry Dolgov <9erthalion6@gmail.com> (earlier version)
Tested-by: Sait Talha Nisanci <Sait.Nisanci@microsoft.com> (earlier version)
Discussion: https://postgr.es/m/CA%2BhUKGJ4VJN8ttxScUFM8dOKX0BrBiboo5uz1cq%3DAovOddfHpA%40mail.gmail.com
Branch
------
master
Details
-------
https://git.postgresql.org/pg/commitdiff/5dc0418fab281d017a61a5756240467af982bdfd
Modified Files
--------------
doc/src/sgml/config.sgml | 64 ++
doc/src/sgml/monitoring.sgml | 86 +-
doc/src/sgml/wal.sgml | 12 +
src/backend/access/transam/Makefile | 1 +
src/backend/access/transam/xlog.c | 2 +
src/backend/access/transam/xlogprefetcher.c | 1082 +++++++++++++++++++++++++
src/backend/access/transam/xlogreader.c | 27 +-
src/backend/access/transam/xlogrecovery.c | 179 ++--
src/backend/access/transam/xlogutils.c | 27 +-
src/backend/catalog/system_views.sql | 14 +
src/backend/storage/buffer/bufmgr.c | 4 +
src/backend/storage/freespace/freespace.c | 3 +-
src/backend/storage/ipc/ipci.c | 3 +
src/backend/storage/smgr/md.c | 6 +-
src/backend/utils/adt/pgstatfuncs.c | 5 +-
src/backend/utils/misc/guc.c | 55 +-
src/backend/utils/misc/postgresql.conf.sample | 6 +
src/include/access/xlog.h | 1 +
src/include/access/xlogprefetcher.h | 53 ++
src/include/access/xlogreader.h | 8 +
src/include/access/xlogutils.h | 3 +-
src/include/catalog/catversion.h | 2 +-
src/include/catalog/pg_proc.dat | 7 +
src/include/utils/guc.h | 4 +
src/include/utils/guc_tables.h | 1 +
src/test/regress/expected/rules.out | 11 +
src/tools/pgindent/typedefs.list | 6 +
27 files changed, 1595 insertions(+), 77 deletions(-)
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Reply to all the recipients from your mail client; the link fills in
the headers below: reply via email
Some clients, webmail especially, drop In-Reply-To and References from
a link, and the reply then starts a new thread: use the mbox below.
To: pgsql-committers@postgresql.org
Cc: tmunro@postgresql.org
Subject: Re: pgsql: Prefetch data referenced by the WAL, take II.
In-Reply-To: <E1ncMoW-000hhv-4s@gemulon.postgresql.org>
References: <E1ncMoW-000hhv-4s@gemulon.postgresql.org>
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
This inbox is served by DDX for PostgreSQL; see mirroring instructions
for how to clone and mirror all data and code used for this inbox