pg.ddx.io  pgsql-hackers@postgresql.org mailing list archive  

help / color / mirror / Atom feed
From: Tomas Vondra <tomas.vondra@enterprisedb.com>
To: Thomas Munro <thomas.munro@gmail.com>
Cc: Stephen Frost <sfrost@snowman.net>
Cc: Andres Freund <andres@anarazel.de>
Cc: Jakub Wartak <Jakub.Wartak@tomtom.com>
Cc: Alvaro Herrera <alvherre@2ndquadrant.com>
Cc: Tomas Vondra <tomas.vondra@2ndquadrant.com>
Cc: Dmitry Dolgov <9erthalion6@gmail.com>
Cc: David Steele <david@pgmasters.net>
Cc: pgsql-hackers <pgsql-hackers@postgresql.org>
Subject: Re: WIP: WAL prefetch (another approach)
Date: Thu, 8 Apr 2021 14:39:18 +0200
Message-ID: <d44b9ae2-2308-128f-9b65-ad14d2bed6ef@enterprisedb.com> (raw)
In-Reply-To: <CA+hUKG+a4wn7B-qBu9SLc1HAbZRSKbp1Ep=rDeveM-UQz84+jg@mail.gmail.com>
References: <CA+hUKGJ1=pOiNjSgXYJnjE3OyRtp8tjMRcON256e5EFpzPpgtA@mail.gmail.com>
	<CA+hUKGKFeYPL9K+SRixcsx1+6HsHhqK+POZyrnnZjw1jERpGcQ@mail.gmail.com>
	<20210210215033.GA27507@tamriel.snowman.net>
	<103ec76d-8bbc-09e1-460c-8785751431b5@enterprisedb.com>
	<20210212044614.5wqjkjbcmbwzever@alap3.anarazel.de>
	<20210213213930.GM27507@tamriel.snowman.net>
	<eed83eaa-91e1-91fe-32d0-ece36c0fa810@enterprisedb.com>
	<20210214231815.GO27507@tamriel.snowman.net>
	<17b3f450-048c-f3a6-7086-259523f33cbc@enterprisedb.com>
	<20210317214331.GF20766@tamriel.snowman.net>
	<28c3352e-5388-6410-aae7-05907079e6c9@enterprisedb.com>
	<CA+hUKGLHTjbBPFX6E1ZCGVyQD30-vWtH=N6EcHEzW13q+jh7FQ@mail.gmail.com>
	<8dd2dcc2-0b6b-a4ab-c1b3-032c0a5ab1c3@enterprisedb.com>
	<CA+hUKGKmkKfH1h9XG5ctYCgqeOHGdN5bHMoQMZLR=+kLY0PQzA@mail.gmail.com>
	<CA+hUKGLsF8sEE+n3qoaV6+XjmYEwYoFVDbLPbYTGz7oGkL=iZA@mail.gmail.com>
	<bab127df-c5c0-6e33-69b1-31471d61ff62@enterprisedb.com>
	<CA+hUKG+a4wn7B-qBu9SLc1HAbZRSKbp1Ep=rDeveM-UQz84+jg@mail.gmail.com>



On 4/8/21 1:46 PM, Thomas Munro wrote:
> On Thu, Apr 8, 2021 at 3:27 AM Tomas Vondra
> <tomas.vondra@enterprisedb.com> wrote:
>> On 4/7/21 1:24 PM, Thomas Munro wrote:
>>> I made one other simplifying change: previously, the prefetch module
>>> would read the WAL up to the "written" LSN (so, allowing itself to
>>> read data that had been written but not yet flushed to disk by the
>>> walreceiver), though it still waited until a record's LSN was
>>> "flushed" before replaying.  That allowed prefetching to happen
>>> concurrently with the WAL flush, which was nice, but it felt a little
>>> too "special".  I decided to remove that part for now, and I plan to
>>> look into making standbys work more like primary servers, using WAL
>>> buffers, the WAL writer and optionally the standard log-before-data
>>> rule.
>>
>> Not sure, but the removal seems unnecessary. I'm worried that this will
>> significantly reduce the amount of data that we'll be able to prefetch.
>> How likely it is that we have data that is written but not flushed?
>> Let's assume the replica is lagging and network bandwidth is not the
>> bottleneck - how likely is this "has to be flushed" a limit for the
>> prefetching?
> 
> Yeah, it would have been nice to include that but it'll have to be for
> v15 due to lack of time to convince myself that it was correct.  I do
> intend to look into more concurrency of that kind for v15.  I have
> pushed these patches, updated to be disabled by default.  I will look
> into how I can run a BF animal that has it enabled during the recovery
> tests for coverage.  Thanks very much to everyone on this thread for
> all the discussion and testing so far.
> 

OK, understood. I'll rerun the benchmarks on this version, and if
there's a significant negative impact we can look into that during the
stabilization phase.

regards

-- 
Tomas Vondra
EnterpriseDB: http://www.enterprisedb.com
The Enterprise PostgreSQL Company





view thread (174+ messages)  latest in thread

Message-ID: <d44b9ae2-2308-128f-9b65-ad14d2bed6ef@enterprisedb.com>
Permalink:  ../d44b9ae2-2308-128f-9b65-ad14d2bed6ef@enterprisedb.com/
Also on:    postgresql.org/message-id/d44b9ae2-2308-128f-9b65-ad14d2bed6ef@enterprisedb.com

 · 

reply
Reply instructions:

You may reply publicly to this message via plain-text email
using any one of the following methods:

* Reply to all the recipients using the --to and --cc options:
  reply via email

  To: pgsql-hackers@postgresql.org
  Cc: tomas.vondra@enterprisedb.com, thomas.munro@gmail.com, sfrost@snowman.net, andres@anarazel.de, Jakub.Wartak@tomtom.com, alvherre@2ndquadrant.com, tomas.vondra@2ndquadrant.com, 9erthalion6@gmail.com, david@pgmasters.net
  Subject: Re: WIP: WAL prefetch (another approach)
  In-Reply-To: <d44b9ae2-2308-128f-9b65-ad14d2bed6ef@enterprisedb.com>

* Save the following mbox file, import it into your mail client,
  and reply-to-all from there: mbox

This inbox is served by DDX for PostgreSQL; see mirroring instructions
for how to clone and mirror all data and code used for this inbox