Date: Sat, 7 Mar 2020 21:53:28 +0100
From: Julien Rouhaud <rjuju123@gmail.com>
To: Tom Lane <tgl@sss.pgh.pa.us>
Cc: Michael Paquier <michael@paquier.xyz>,
	Michael Paquier <michael.paquier@gmail.com>,
	pgsql-hackers <pgsql-hackers@postgresql.org>
Subject: Re: Add an optional timeout clause to isolationtester step.
Message-ID: <20200307205328.ffidfyo3me46qscn@nol>
References: <20200227080735.l32fqcauy73lon7o@nol>
 <20200303080642.GA1890@paquier.xyz>
 <20200303092551.GB1890@paquier.xyz>
 <20200304051510.GE2593@paquier.xyz>
 <CAOBaU_Z=qkE42VQ25mps+zyAz+WUn3wZoev4uhGW+gAXBBS2YA@mail.gmail.com>
 <20200305035354.GQ2593@paquier.xyz>
 <20200306131547.GA2904@nol>
 <20200307014142.GC1531@paquier.xyz>
 <20200307061615.c47sju2bytuiaazv@nol>
 <8441.1583595994@sss.pgh.pa.us>
MIME-Version: 1.0
Content-Type: text/plain; charset=us-ascii
Content-Disposition: inline
In-Reply-To: <8441.1583595994@sss.pgh.pa.us>
Precedence: bulk

On Sat, Mar 07, 2020 at 10:46:34AM -0500, Tom Lane wrote:
> Julien Rouhaud <rjuju123@gmail.com> writes:
> > On Sat, Mar 07, 2020 at 10:41:42AM +0900, Michael Paquier wrote:
> >> On Fri, Mar 06, 2020 at 02:15:47PM +0100, Julien Rouhaud wrote:
> >>> Here's a patch to add an optional "timeout val" clause to isolationtester's
> >>> step definition.  When used, isolationtester will actively wait on the query
> >>> rather than continuing with the permutation next step, and will issue a cancel
> >>> once the defined timeout is reached.
>
> >> One problem with this approach is that it does address the stability
> >> of the test on very slow machines, and there are some of them in the
> >> buildfarm.
>
> > Looking at timeouts.spec and e.g. a7921f71a3c, it seems that we already chose
> > to fix this problem by having a timeout long enough to statisfy the slower
> > buildfarm members, even when running on fast machines, so I assumed that the
> > same approach could be used here.
>
> The arbitrarily-set timeouts that exist in some of the isolation tests
> are horrid kluges that have caused us lots of headaches in the past
> and no doubt will again in the future.  Aside from occasionally failing
> when a machine is particularly overloaded, they cause the tests to
> take far longer than necessary on decently-fast machines.

Yeah, I have no doubt that it has been a pain, and this patch is clearly not a
bullet-proof solution.

> So ideally
> we'd get rid of those entirely in favor of some more-dynamic approach.
> Admittedly, I have no proposal for what that would be.

The fault injection framework that was previously discussed would cover most of
the usecase I can think of, but that's a way bigger project.

> But adding yet
> more ways to set a (guaranteed-to-be-wrong) timeout seems like the
> wrong direction to be going in.

Fair enough, I'll mark the patch as rejected then.

> What's the actual need that you're trying to deal with?

Testing the correct behavior of non trivial commands, such as CIC/reindex
concurrently, that fails during the execution.