From: Noah Misch <noah@leadboat.com>
To: Kyotaro Horiguchi <horikyota.ntt@gmail.com>
Cc: robertmhaas@gmail.com
Cc: pgsql-hackers@postgresql.org
Cc: 9erthalion6@gmail.com
Cc: andrew.dunstan@2ndquadrant.com
Cc: hlinnaka@iki.fi
Cc: michael@paquier.xyz
Subject: Re: [HACKERS] WAL logging problem in 9.4.3?
Date: Sun, 29 Mar 2020 21:41:01 -0700
Message-ID: <20200330044101.GA2324620@rfd.leadboat.com> (raw)
In-Reply-To: <20200323.172027.2270553329883636814.horikyota.ntt@gmail.com>
References: <20200316034647.GA1121601@rfd.leadboat.com>
<20200321190127.GA1763544@rfd.leadboat.com>
<20200321224920.GB1763544@rfd.leadboat.com>
<20200323.172027.2270553329883636814.horikyota.ntt@gmail.com>
I think attached v41nm is ready for commit. Would anyone like to vote against
back-patching this? It's hard to justify lack of back-patch for a data-loss
bug, but this is atypically invasive. (I'm repeating the question, since some
folks missed my 2020-02-18 question.) Otherwise, I'll push this on Saturday.
On Mon, Mar 23, 2020 at 05:20:27PM +0900, Kyotaro Horiguchi wrote:
> At Sat, 21 Mar 2020 15:49:20 -0700, Noah Misch <noah@leadboat.com> wrote in
> > The proximate cause is the RelFileNodeSkippingWAL() call that we added to
> > MarkBufferDirtyHint(). MarkBufferDirtyHint() runs in parallel workers, but
> > parallel workers have zeroes for pendingSyncHash and rd_*Subid.
> > Kyotaro, can you look through the affected code and propose a strategy for
> > good coexistence of parallel query with the WAL skipping mechanism?
>
> Bi-directional communication between leader and workers is too-much.
> It wouldn't be acceptable to inhibit the problematic operations on
> workers such like heap-prune or btree pin removal. If we do pending
> syncs just before worker start, it won't fix the issue.
>
> The attached patch passes a list of pending-sync relfilenodes at
> worker start.
If you were to issue pending syncs and also cease skipping WAL for affected
relations, that would fix the issue. Your design is better, though. I made
two notable changes:
- The patch was issuing syncs or FPIs every time a parallel worker exited. I
changed it to skip most of smgrDoPendingSyncs() in parallel workers, like
AtEOXact_RelationMap() does.
- PARALLEL_KEY_PENDING_SYNCS is most similar to PARALLEL_KEY_REINDEX_STATE and
PARALLEL_KEY_COMBO_CID. parallel.c, not execParallel.c, owns those. I
moved PARALLEL_KEY_PENDING_SYNCS to parallel.c, which also called for style
changes in the associated storage.c functions.
Since pendingSyncHash is always NULL under XLogIsNeeded(), I also removed some
XLogIsNeeded() tests that immediately preceded !pendingSyncHash tests.
Attachments:
[application/x-tar-gz] skip-wal-v41nm.tar.gz (192.3K, ../20200330044101.GA2324620@rfd.leadboat.com/2-skip-wal-v41nm.tar.gz)
download
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Reply to all the recipients using the --to and --cc options:
reply via email
To: pgsql-hackers@postgresql.org
Cc: noah@leadboat.com, horikyota.ntt@gmail.com, robertmhaas@gmail.com, 9erthalion6@gmail.com, andrew.dunstan@2ndquadrant.com, hlinnaka@iki.fi, michael@paquier.xyz
Subject: Re: [HACKERS] WAL logging problem in 9.4.3?
In-Reply-To: <20200330044101.GA2324620@rfd.leadboat.com>
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
This inbox is served by DDX for PostgreSQL; see mirroring instructions
for how to clone and mirror all data and code used for this inbox