Subject: Re: WIP: WAL prefetch (another approach)
To: Thomas Munro <thomas.munro@gmail.com>, Tom Lane <tgl@sss.pgh.pa.us>
Cc: Andres Freund <andres@anarazel.de>, Stephen Frost <sfrost@snowman.net>,
 Alvaro Herrera <alvherre@2ndquadrant.com>, David Steele
 <david@pgmasters.net>, Dmitry Dolgov <9erthalion6@gmail.com>,
 Jakub Wartak <Jakub.Wartak@tomtom.com>,
 pgsql-hackers <pgsql-hackers@postgresql.org>
References: <8dd2dcc2-0b6b-a4ab-c1b3-032c0a5ab1c3@enterprisedb.com>
 <CA+hUKGKmkKfH1h9XG5ctYCgqeOHGdN5bHMoQMZLR=+kLY0PQzA@mail.gmail.com>
 <CA+hUKGLsF8sEE+n3qoaV6+XjmYEwYoFVDbLPbYTGz7oGkL=iZA@mail.gmail.com>
 <CA+hUKG+a4wn7B-qBu9SLc1HAbZRSKbp1Ep=rDeveM-UQz84+jg@mail.gmail.com>
 <773932.1619022622@sss.pgh.pa.us>
 <b971373e-5f05-e691-d83c-c72a4f67167b@enterprisedb.com>
 <CA+hUKGJKDibdw5HzOdgh+tFsA7=wwJ3O6p+qYwbgAD_xkLWA3Q@mail.gmail.com>
 <CAOuzzgrn7iKnFRsB4MHp3UisEQAGgZMbk_ViTN4HV4-Ksq8zCg@mail.gmail.com>
 <892344.1619054465@sss.pgh.pa.us>
 <CA+hUKGJKKh8TbA+6977zK5TKUSsTspBHsr0s4S42q7ynkxcPGQ@mail.gmail.com>
 <20210428163245.ff67pb2ii5irr5zr@alap3.anarazel.de>
 <2808045.1619628335@sss.pgh.pa.us>
 <CA+hUKGJ03Q-TmE8-JzAm04_1maYus6_kX7XB--JBh1K63agSbQ@mail.gmail.com>
 <3502526.1619925367@sss.pgh.pa.us>
 <CA+hUKGLGKnzTDhW9rnzyYb7yvUiCoGK5pB74qQf3YYLZYSX4OA@mail.gmail.com>
From: Tomas Vondra <tomas.vondra@enterprisedb.com>
Message-ID: <f2be6caa-5a7a-990b-c56e-a29454ae1cee@enterprisedb.com>
Date: Tue, 4 May 2021 14:37:22 +0200
User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:78.0) Gecko/20100101
 Thunderbird/78.8.1
MIME-Version: 1.0
In-Reply-To: 
 <CA+hUKGLGKnzTDhW9rnzyYb7yvUiCoGK5pB74qQf3YYLZYSX4OA@mail.gmail.com>
Content-Type: text/plain; charset=utf-8; format=flowed
Content-Language: en-US
Content-Transfer-Encoding: 7bit
Archived-At: 
 <https://www.postgresql.org/message-id/f2be6caa-5a7a-990b-c56e-a29454ae1cee%40enterprisedb.com>
Precedence: bulk


On 5/3/21 7:42 AM, Thomas Munro wrote:
> On Sun, May 2, 2021 at 3:16 PM Tom Lane <tgl@sss.pgh.pa.us> wrote:
>> That last point means that there was some hard-to-hit problem even
>> before any of the recent WAL-related changes.  However, 323cbe7c7
>> (Remove read_page callback from XLogReader) increased the failure
>> rate by at least a factor of 5, and 1d257577e (Optionally prefetch
>> referenced data) seems to have increased it by another factor of 4.
>> But it looks like f003d9f87 (Add circular WAL decoding buffer)
>> didn't materially change the failure rate.
> 
> Oh, wow.  There are several surprising results there.  Thanks for
> running those tests for so long so that we could see the rarest
> failures.
> 
> Even if there are somehow *two* causes of corruption, one preexisting
> and one added by the refactoring or decoding patches, I'm struggling
> to understand how the chance increases with 1d2575, since that only
> adds code that isn't reached when not enabled (though I'm going to
> re-review that).
> 
>> Considering that 323cbe7c7 was supposed to be just refactoring,
>> and 1d257577e is allegedly disabled-by-default, these are surely
>> not the results I was expecting to get.
> 
> +1
> 
>> It seems like it's still an open question whether all this is
>> a real bug, or flaky hardware.  I have seen occasional kernel
>> freezeups (or so I think -- machine stops responding to keyboard
>> or network input) over the past year or two, so I cannot in good
>> conscience rule out the flaky-hardware theory.  But it doesn't
>> smell like that kind of problem to me.  I think what we're looking
>> at is a timing-sensitive bug that was there before (maybe long
>> before?) and these commits happened to make it occur more often
>> on this particular hardware.  This hardware is enough unlike
>> anything made in the past decade that it's not hard to credit
>> that it'd show a timing problem that nobody else can reproduce.
> 
> Hmm, yeah that does seem plausible.  It would be nice to see a report
> from any other system though.  I'm still trying, and reviewing...
> 

FWIW I've ran the test (make installcheck-parallel in a loop) on four 
different machines - two x86_64 ones, and two rpi4. The x86 boxes did 
~1000 rounds each (and one of them had 5 local replicas) without any 
issue. The rpi4 machines did ~50 rounds each, also without failures.

Obviously, it's possible there's something that neither of those (very 
different systems) triggers, but I'd say it might also be a hint that 
this really is a hw issue on the old ppc macs. Or maybe something very 
specific to that arch.


regards

-- 
Tomas Vondra
EnterpriseDB: http://www.enterprisedb.com
The Enterprise PostgreSQL Company