Date: Fri, 20 Sep 2019 15:03:32 -0700
From: Andres Freund <andres@anarazel.de>
To: Tom Lane <tgl@sss.pgh.pa.us>
Cc: Andrew Dunstan <andrew@dunslane.net>,
	Kuntal Ghosh <kuntalghosh.2007@gmail.com>,
	Michael Paquier <michael@paquier.xyz>,
	Tomas Vondra <tomas.vondra@2ndquadrant.com>,
	Robert Haas <robertmhaas@gmail.com>,
	Thomas Munro <thomas.munro@enterprisedb.com>,
	PostgreSQL Hackers <pgsql-hackers@lists.postgresql.org>
Subject: Re: subscriptionCheck failures on nightjar
Message-ID: <20190920220332.qhd4ym26wa76ajqt@alap3.anarazel.de>
References: <20190917194510.iqwyl3be62pz7l27@development>
 <20190918005815.GB8909@paquier.xyz>
 <CAGz5QCJv5JbRDsATDTkJqq7h9F7u0QLnNnLHfxR1nEOa4DnkJQ@mail.gmail.com>
 <20190918215808.yonxqgycme6pbctp@development>
 <20190919042305.GA21144@paquier.xyz>
 <CAGz5QC+5_mPFoDj7ZSMV0gwvMY+kdOp4t1w=TTDpzuV9F2-X6g@mail.gmail.com>
 <20190920170831.aaljabal6lyivre5@alap3.anarazel.de>
 <29511.1569011121@sss.pgh.pa.us>
 <20190920212603.7zlgrlwtdirbmuw7@alap3.anarazel.de>
 <2636.1569016167@sss.pgh.pa.us>
MIME-Version: 1.0
Content-Type: text/plain; charset=us-ascii
Content-Disposition: inline
In-Reply-To: <2636.1569016167@sss.pgh.pa.us>
Precedence: bulk

Hi,

On 2019-09-20 17:49:27 -0400, Tom Lane wrote:
> Andres Freund <andres@anarazel.de> writes:
> > On 2019-09-20 16:25:21 -0400, Tom Lane wrote:
> >> I recreated my freebsd-9-under-qemu setup and I can still reproduce
> >> the problem, though not with high reliability (order of 1 time in 10).
> >> Anything particular you want logged?
> 
> > A DEBUG2 log would help a fair bit, because it'd log some information
> > about what changes the "horizons" determining when data may be removed.
> 
> Actually, what I did was as attached [1], and I am getting traces like
> [2].  The problem seems to occur only when there are two or three
> processes concurrently creating the same snapshot file.  It's not
> obvious from the debug trace, but the snapshot file *does* exist
> after the music stops.
> 
> It is very hard to look at this trace and conclude anything other
> than "rename(2) is broken, it's not atomic". Nothing in our code
> has deleted the file: no checkpoint has started, nor do we see
> the DEBUG1 output that CheckPointSnapBuild ought to produce.
> But fsync_fname momentarily can't see it (and then later another
> process does see it).

Yikes. No wondering most of us weren't able to reproduce the
problem. And that staring at our code didn't point to a bug.

Nice catch.


> In short, what we got here is OS bugs that have probably been
> resolved years ago.
> 
> The question is what to do next.  Should we just retire these
> specific buildfarm critters, or do we want to push ahead with
> getting rid of the PANIC here?

Hm. Given that the fsync failing is actually an issue, I'm somewhat
disinclined to remove the PANIC. It's not like only raising an ERROR
actually solves anything, except making the problem even harder to
diagnose? Or that we otherwise are ok, with renames not being atomic?

So I'd be tentatively in favor of either upgrading, replacing the
filesystem (perhaps ZFS isn't buggy in the same way?), or retiring
those animals.

Greetings,

Andres Freund