From: Tom Lane <tgl@sss.pgh.pa.us>
To: Noah Misch <noah@leadboat.com>
cc: Andres Freund <andres@anarazel.de>,
        Petr Jelinek <petr.jelinek@2ndquadrant.com>,
        Peter Eisentraut <peter.eisentraut@2ndquadrant.com>,
        Erik Rijkers <er@xs4all.nl>, David Steele <david@pgmasters.net>,
        PostgreSQL-development <pgsql-hackers@postgresql.org>,
        Robert Haas <robertmhaas@gmail.com>,
        Craig Ringer <craig@2ndquadrant.com>,
        pgsql-hackers-owner@postgresql.org
Subject: Re: snapbuild woes
In-reply-to: <20170429214201.GB799290@rfd.leadboat.com>
References: <20170408143059.nrzuwyvv5apnhhta@alap3.anarazel.de>
 <20170412063142.GC2870454@tornado.leadboat.com>
 <d6f4a15c-b95d-54b1-0b06-f24bcd0b6fcb@2ndquadrant.com>
 <20170412172151.nauhiagitlpfxief@alap3.anarazel.de>
 <20170413045812.GA2917375@tornado.leadboat.com>
 <20170417020404.GB2988768@tornado.leadboat.com>
 <20170418041657.hvwss6hz73ghozp5@alap3.anarazel.de>
 <20170420000906.75t6uo7bof5pazrq@alap3.anarazel.de>
 <ad637c84-534d-b291-9b6c-458f1ff7f54b@2ndquadrant.com>
 <20170422053458.z7kmzopqdyi2tai4@alap3.anarazel.de>
 <20170429214201.GB799290@rfd.leadboat.com>
Comments: In-reply-to Noah Misch <noah@leadboat.com>
	message dated "Sat, 29 Apr 2017 14:42:01 -0700"
MIME-Version: 1.0
Content-Type: text/plain; charset="us-ascii"
Content-ID: <4386.1493642807.1@sss.pgh.pa.us>
Date: Mon, 01 May 2017 08:46:47 -0400
Message-ID: <4387.1493642807@sss.pgh.pa.us>
Precedence: bulk
Sender: pgsql-hackers-owner@postgresql.org

Noah Misch <noah@leadboat.com> writes:
> On Fri, Apr 21, 2017 at 10:34:58PM -0700, Andres Freund wrote:
>> ... I was wondering about adding
>> a loop that simply runs for like 30s and then quits or such, but who
>> knows.

> If the probabilistic test catches the bug even 5% of the time in typical
> configurations, the buildfarm will rapidly identify any regression.  I'd
> choose a 7s test that detects the bug 5% of the time over a 30s test that
> detects it 99% of the time.  (When I wrote src/bin/pgbench/t/001_pgbench.pl
> for a probabilistic bug, I sized that test to finish in 1s and catch its bug
> half the time.  In its case, only two buildfarm members were able to
> demonstrate the original bug, so 5% detection would have been too low.)

30sec is kind of a big lump from a buildfarm standpoint, especially if
you mean "it runs for 30s on my honkin' fast workstation".  I'm fine
with individual tests that run for ~ 1sec.

(This is top-of-mind for me right now because I've been looking around
for ways to speed up the regression tests.)

			regards, tom lane


-- 
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers