From: Tomas Vondra <tomas.vondra@enterprisedb.com>
To: Thomas Munro <thomas.munro@gmail.com>
To: Tom Lane <tgl@sss.pgh.pa.us>
Cc: Andres Freund <andres@anarazel.de>
Cc: Stephen Frost <sfrost@snowman.net>
Cc: Alvaro Herrera <alvherre@2ndquadrant.com>
Cc: David Steele <david@pgmasters.net>
Cc: Dmitry Dolgov <9erthalion6@gmail.com>
Cc: Jakub Wartak <Jakub.Wartak@tomtom.com>
Cc: pgsql-hackers <pgsql-hackers@postgresql.org>
Subject: Re: WIP: WAL prefetch (another approach)
Date: Tue, 4 May 2021 14:37:22 +0200
Message-ID: <f2be6caa-5a7a-990b-c56e-a29454ae1cee@enterprisedb.com> (raw)
In-Reply-To: <CA+hUKGLGKnzTDhW9rnzyYb7yvUiCoGK5pB74qQf3YYLZYSX4OA@mail.gmail.com>
References: <8dd2dcc2-0b6b-a4ab-c1b3-032c0a5ab1c3@enterprisedb.com>
<CA+hUKGKmkKfH1h9XG5ctYCgqeOHGdN5bHMoQMZLR=+kLY0PQzA@mail.gmail.com>
<CA+hUKGLsF8sEE+n3qoaV6+XjmYEwYoFVDbLPbYTGz7oGkL=iZA@mail.gmail.com>
<CA+hUKG+a4wn7B-qBu9SLc1HAbZRSKbp1Ep=rDeveM-UQz84+jg@mail.gmail.com>
<773932.1619022622@sss.pgh.pa.us>
<b971373e-5f05-e691-d83c-c72a4f67167b@enterprisedb.com>
<CA+hUKGJKDibdw5HzOdgh+tFsA7=wwJ3O6p+qYwbgAD_xkLWA3Q@mail.gmail.com>
<CAOuzzgrn7iKnFRsB4MHp3UisEQAGgZMbk_ViTN4HV4-Ksq8zCg@mail.gmail.com>
<892344.1619054465@sss.pgh.pa.us>
<CA+hUKGJKKh8TbA+6977zK5TKUSsTspBHsr0s4S42q7ynkxcPGQ@mail.gmail.com>
<20210428163245.ff67pb2ii5irr5zr@alap3.anarazel.de>
<2808045.1619628335@sss.pgh.pa.us>
<CA+hUKGJ03Q-TmE8-JzAm04_1maYus6_kX7XB--JBh1K63agSbQ@mail.gmail.com>
<3502526.1619925367@sss.pgh.pa.us>
<CA+hUKGLGKnzTDhW9rnzyYb7yvUiCoGK5pB74qQf3YYLZYSX4OA@mail.gmail.com>
On 5/3/21 7:42 AM, Thomas Munro wrote:
> On Sun, May 2, 2021 at 3:16 PM Tom Lane <tgl@sss.pgh.pa.us> wrote:
>> That last point means that there was some hard-to-hit problem even
>> before any of the recent WAL-related changes. However, 323cbe7c7
>> (Remove read_page callback from XLogReader) increased the failure
>> rate by at least a factor of 5, and 1d257577e (Optionally prefetch
>> referenced data) seems to have increased it by another factor of 4.
>> But it looks like f003d9f87 (Add circular WAL decoding buffer)
>> didn't materially change the failure rate.
>
> Oh, wow. There are several surprising results there. Thanks for
> running those tests for so long so that we could see the rarest
> failures.
>
> Even if there are somehow *two* causes of corruption, one preexisting
> and one added by the refactoring or decoding patches, I'm struggling
> to understand how the chance increases with 1d2575, since that only
> adds code that isn't reached when not enabled (though I'm going to
> re-review that).
>
>> Considering that 323cbe7c7 was supposed to be just refactoring,
>> and 1d257577e is allegedly disabled-by-default, these are surely
>> not the results I was expecting to get.
>
> +1
>
>> It seems like it's still an open question whether all this is
>> a real bug, or flaky hardware. I have seen occasional kernel
>> freezeups (or so I think -- machine stops responding to keyboard
>> or network input) over the past year or two, so I cannot in good
>> conscience rule out the flaky-hardware theory. But it doesn't
>> smell like that kind of problem to me. I think what we're looking
>> at is a timing-sensitive bug that was there before (maybe long
>> before?) and these commits happened to make it occur more often
>> on this particular hardware. This hardware is enough unlike
>> anything made in the past decade that it's not hard to credit
>> that it'd show a timing problem that nobody else can reproduce.
>
> Hmm, yeah that does seem plausible. It would be nice to see a report
> from any other system though. I'm still trying, and reviewing...
>
FWIW I've ran the test (make installcheck-parallel in a loop) on four
different machines - two x86_64 ones, and two rpi4. The x86 boxes did
~1000 rounds each (and one of them had 5 local replicas) without any
issue. The rpi4 machines did ~50 rounds each, also without failures.
Obviously, it's possible there's something that neither of those (very
different systems) triggers, but I'd say it might also be a hint that
this really is a hw issue on the old ppc macs. Or maybe something very
specific to that arch.
regards
--
Tomas Vondra
EnterpriseDB: http://www.enterprisedb.com
The Enterprise PostgreSQL Company
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Reply to all the recipients using the --to and --cc options:
reply via email
To: pgsql-hackers@postgresql.org
Cc: tomas.vondra@enterprisedb.com, thomas.munro@gmail.com, tgl@sss.pgh.pa.us, andres@anarazel.de, sfrost@snowman.net, alvherre@2ndquadrant.com, david@pgmasters.net, 9erthalion6@gmail.com, Jakub.Wartak@tomtom.com
Subject: Re: WIP: WAL prefetch (another approach)
In-Reply-To: <f2be6caa-5a7a-990b-c56e-a29454ae1cee@enterprisedb.com>
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
This inbox is served by DDX for PostgreSQL; see mirroring instructions
for how to clone and mirror all data and code used for this inbox