Date: Wed, 7 Oct 2020 02:58:38 +0200
From: Tomas Vondra <tomas.vondra@2ndquadrant.com>
To: Thomas Munro <thomas.munro@gmail.com>
Cc: Stephen Frost <sfrost@snowman.net>,
	Dmitry Dolgov <9erthalion6@gmail.com>,
	David Steele <david@pgmasters.net>,
	Andres Freund <andres@anarazel.de>,
	Alvaro Herrera <alvherre@2ndquadrant.com>,
	pgsql-hackers <pgsql-hackers@postgresql.org>
Subject: Re: WIP: WAL prefetch (another approach)
Message-ID: <20201007005838.lrpsbpckc5b7giwk@development>
References: <20200803154659.nzf5qzuozts5dujp@development>
 <CA+hUKGKfsWj+4qPfOkC4YtYtVO33yrCty1uWxBp3R1OSqWEcyA@mail.gmail.com>
 <20200806104701.j3xnkczd6qelfnhj@development>
 <CA+hUKGKciMMJbE-1+9oPKte977b++UU_X16kkOXTF_7QjP4+jA@mail.gmail.com>
 <20200901131426.5lse2jlepm3uligv@development>
 <CA+hUKG+9NUUbdQqFNCz2SKBdyCCD7SQwJy-Y6_PZmCnc+h79ig@mail.gmail.com>
 <20200901141826.gbhqz664brtt4y2t@development>
 <CA+hUKGLr6nU7yb8CO7_DEppBmPCOkCUvE6HBza8himYV_s5nuQ@mail.gmail.com>
 <20200908231235.lsgat5ni35h6odqt@development>
 <CA+hUKG+2Vw3UAVNJSfz5_zhRcHUWEBDrpB7pyQ85Yroep0AKbw@mail.gmail.com>
MIME-Version: 1.0
Content-Type: text/plain; charset=us-ascii; format=flowed
Content-Disposition: inline
In-Reply-To: 
 <CA+hUKG+2Vw3UAVNJSfz5_zhRcHUWEBDrpB7pyQ85Yroep0AKbw@mail.gmail.com>
Precedence: bulk

On Thu, Sep 24, 2020 at 11:38:45AM +1200, Thomas Munro wrote:
>On Wed, Sep 9, 2020 at 11:16 AM Tomas Vondra
><tomas.vondra@2ndquadrant.com> wrote:
>> OK, thanks for looking into this. I guess I'll wait for an updated patch
>> before testing this further. The storage has limited capacity so I'd
>> have to either reduce the amount of data/WAL or juggle with the WAL
>> segments somehow. Doesn't seem worth it.
>
>Here's a new WIP version that works for archive-based recovery in my tests.
>
>The main change I have been working on is that there is now just a
>single XLogReaderState, so no more double-reading and double-decoding
>of the WAL.  It provides XLogReadRecord(), as before, but now you can
>also read further ahead with XLogReadAhead().  The user interface is
>much like before, except that the GUCs changed a bit.  They are now:
>
>  recovery_prefetch=on
>  recovery_prefetch_fpw=off
>  wal_decode_buffer_size=256kB
>  maintenance_io_concurrency=10
>
>I recommend setting maintenance_io_concurrency and
>wal_decode_buffer_size much higher than those defaults.
>

I think you've left the original GUC (replaced by the buffer size) in
the postgresql.conf.sample file. Confused me for a bit ;-)

I've done a bit of testing and so far it seems to work with WAL archive,
so I'll do more testing and benchmarking over the next couple days.


regards

-- 
Tomas Vondra                  http://www.2ndQuadrant.com
PostgreSQL Development, 24x7 Support, Remote DBA, Training & Services