Date: Wed, 23 Jun 2021 09:08:35 -0400
From: Alvaro Herrera <alvherre@alvh.no-ip.org>
To: Fabien COELHO <coelho@cri.ensmp.fr>
Cc: Michael Paquier <michael@paquier.xyz>,
	Andrew Dunstan <andrew@dunslane.net>,
	Thomas Munro <thomas.munro@gmail.com>,
	Gregory Smith <gregsmithpgsql@gmail.com>,
	PostgreSQL-development <pgsql-hackers@postgresql.org>,
	david.christensen@crunchydata.com
Subject: Re: pgbench logging broken by time logic changes
Message-ID: <202106231308.lxr26cahwxuc@alvherre.pgsql>
MIME-Version: 1.0
Content-Type: text/plain; charset=iso-8859-1
Content-Disposition: inline
Content-Transfer-Encoding: 8bit
In-Reply-To: <alpine.DEB.2.22.394.2106231125540.1117163@pseudo>
Archived-At: 
 <https://www.postgresql.org/message-id/202106231308.lxr26cahwxuc%40alvherre.pgsql>
Precedence: bulk

On 2021-Jun-23, Fabien COELHO wrote:

> +# cool check that we are around 2 seconds
> +# The rate may results in an unlucky schedule which triggers
> +# an early exit, hence the loose bound.
> +#
> +# THIS TEST IS COMMENTED OUT BUT PLEASE LET IT THERE SO THAT
> +# IT CAN BE ENABLED EASILY.
> +#
> +## ok(1.5 < $delay && $delay < 2.5, "-T 2 run around 2 seconds");

I think you should use Test::More's "skip" for this, perhaps something
like this:

SKIP: {
   skip "This test is unreliable";

   # explain why
   ok(1.5 < $delay && $delay < 2.5, "-T 2 run around 2 seconds");
}

... or, actually, even better would be to use a TODO block, so that the
test is run and reports its status, but if it happens not to succeed it
will not cause the whole test to fail.  That way you'll accumulate some
evidence that may serve to improve the test in the future until it
works fully:

TODO: {
   local $TODO = "Ths test is unreliable";

   ok(1.5 < $delay && $delay < 2.5, "-T 2 run around 2 seconds");
}

-- 
Álvaro Herrera       Valdivia, Chile
"El Maquinismo fue proscrito so pena de cosquilleo hasta la muerte"
(Ijon Tichy en Viajes, Stanislaw Lem)