Date: Thu, 17 Jun 2021 21:05:28 -0700
From: Noah Misch <noah@leadboat.com>
To: Andrew Dunstan <andrew@dunslane.net>
Cc: Fabien COELHO <coelho@cri.ensmp.fr>,
	Gregory Smith <gregsmithpgsql@gmail.com>,
	PostgreSQL-development <pgsql-hackers@postgresql.org>,
	david.christensen@crunchydata.com
Subject: Re: pgbench logging broken by time logic changes
Message-ID: <20210618040528.GA1066073@rfd.leadboat.com>
References: 
 <CAHLJuCW_8Vpcr0=t6O_gozrg3wXXWXZXDioYJd3NhvKriqgpfQ@mail.gmail.com>
 <alpine.DEB.2.22.394.2106162032110.2407635@pseudo>
 <e05abaab-7bd2-93ed-eeca-3f3ceb542506@dunslane.net>
MIME-Version: 1.0
Content-Type: text/plain; charset=us-ascii
Content-Disposition: inline
In-Reply-To: <e05abaab-7bd2-93ed-eeca-3f3ceb542506@dunslane.net>
User-Agent: Mutt/1.5.24 (2015-08-30)
Archived-At: 
 <https://www.postgresql.org/message-id/20210618040528.GA1066073%40rfd.leadboat.com>
Precedence: bulk

On Wed, Jun 16, 2021 at 03:13:30PM -0400, Andrew Dunstan wrote:
> On 6/16/21 2:59 PM, Fabien COELHO wrote:
> > The key feedback for me is the usual one: what is not tested does not
> > work. Wow:-)
> 
> Agreed.
> 
> > I'm unhappy because I already added tap tests for time-sensitive
> > features (-T and others, maybe logging aggregates, cannot remember),
> > which have been removed because they could fail under some
> > circonstances (eg very very very very slow hosts), or required some
> > special handling (a few lines of code) in pgbench, and the net result
> > of this is there is not a single test in place for some features:-(
> 
> I'm not familiar with exactly what happened in this case, but tests need
> to be resilient over a wide range of performance characteristics. One
> way around this issue might be to have a way of detecting that it's on a
> slow platform and if so either skipping tests (Test::More provides
> plenty of support for this) or expecting different results.

Detection would need the host to be consistently slow, like running under
Valgrind or a 20-year-old CPU.  We also test on systems having highly-variable
performance due to other processes competing for the same hardware.  I'd
perhaps add a "./configure --enable-realtime-tests" option that enables
affected tests.  Testers should use the option whenever the execution
environment has sufficient reserved CPU.