From: Noah Misch <noah@leadboat.com>
To: David Steele <david@pgmasters.net>
Cc: Kyotaro Horiguchi <horikyota.ntt@gmail.com>
Cc: pgsql-hackers@postgresql.org
Cc: don@seiler.us
Subject: Re: Race between KeepFileRestoredFromArchive() and restartpoint
Date: Thu, 4 Aug 2022 07:21:10 -0700
Message-ID: <20220804142110.GA3878007@rfd.leadboat.com> (raw)
In-Reply-To: <bd296352-77a6-2c46-5e3e-a548f74c2494@pgmasters.net>
References: <e1f1fb79-f668-0e7c-9841-763a023a7187@pgmasters.net>
<20220802143727.GA3796043@rfd.leadboat.com>
<20220804062456.GA3844351@rfd.leadboat.com>
<20220804.170640.1425291966829818860.horikyota.ntt@gmail.com>
<bd296352-77a6-2c46-5e3e-a548f74c2494@pgmasters.net>
On Thu, Aug 04, 2022 at 06:32:38AM -0400, David Steele wrote:
> On 8/4/22 04:06, Kyotaro Horiguchi wrote:
> >At Wed, 3 Aug 2022 23:24:56 -0700, Noah Misch <noah@leadboat.com> wrote in
> >>>>I think in this case a HINT might be sufficient to at least keep people from
> >>>>wasting time tracking down a problem that has already been fixed.
> >>
> >>Here's a patch to add that HINT. I figure it's better to do this before next
> >>week's minor releases. In the absence of objections, I'll push this around
> >>2022-08-05 14:00 UTC.
> >
> >+1
>
> Looks good to me as well.
Thanks for reviewing.
> >>>>However, there is another issue [1] that might argue for a back patch if
> >>>>this patch (as I believe) would fix the issue.
> >>>
> >>>>[1] https://www.postgresql.org/message-id/CAHJZqBDxWfcd53jm0bFttuqpK3jV2YKWx%3D4W7KxNB4zzt%2B%2BqFg%40ma...
> >>>
> >>>That makes sense. Each iteration of the restartpoint recycle loop has a 1/N
> >>>chance of failing. Recovery adds >N files between restartpoints. Hence, the
> >>>WAL directory grows without bound. Is that roughly the theory in mind?
> >>
> >>On further reflection, I don't expect it to happen that way. Each failure
> >>message is LOG-level, so the remaining recycles still happen. (The race
> >>condition can yield an ERROR under PreallocXlogFiles(), but recycling is
> >>already done at that point.)
> >
> >I agree to this.
>
> Hmmm, OK. We certainly have a fairly serious issue here, i.e. pg_wal growing
> without bound. Even if we are not sure what is causing it, how confident are
> we that the patches applied to v15 would fix it?
I'll say 7%; certainly too low to stop investigating. The next things I'd want
to see are:
- select name, setting, source from pg_settings where setting <> boot_val;
- log_checkpoints log entries, and other log entries having the same PID
If the theory about checkpoint skipping holds, there should be a period where
the volume of WAL replayed greatly exceeds max_wal_size, yet 0-1 restartpoints
begin and 0 restartpoints complete.
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Reply to all the recipients using the --to and --cc options:
reply via email
To: pgsql-hackers@postgresql.org
Cc: noah@leadboat.com, david@pgmasters.net, horikyota.ntt@gmail.com, don@seiler.us
Subject: Re: Race between KeepFileRestoredFromArchive() and restartpoint
In-Reply-To: <20220804142110.GA3878007@rfd.leadboat.com>
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
This inbox is served by DDX for PostgreSQL; see mirroring instructions
for how to clone and mirror all data and code used for this inbox