Date: Thu, 24 Feb 2022 07:00:33 -0800
From: Andres Freund <andres@anarazel.de>
To: Peter Eisentraut <peter.eisentraut@enterprisedb.com>
Cc: pgsql-hackers <pgsql-hackers@postgresql.org>,
	Andrew Dunstan <andrew@dunslane.net>,
	Daniel Gustafsson <daniel@yesql.se>, Tom Lane <tgl@sss.pgh.pa.us>,
	Dagfinn Ilmari =?utf-8?Q?Manns=C3=A5ker?= <ilmari@ilmari.org>
Subject: Re: Readd use of TAP subtests
Message-ID: <20220224150033.5lql2lkiv7y5kkme@alap3.anarazel.de>
References: <9878e1e2-2ce3-6df0-3498-07d967534f5e@enterprisedb.com>
 <87zgpbnr59.fsf@wibble.ilmari.org>
 <6131CFD1-EC47-41ED-999E-AB28B40DDEAA@yesql.se>
 <87wnkfnq9m.fsf@wibble.ilmari.org>
 <7e67b831-9eae-cd19-c96c-c1e59d916498@dunslane.net>
 <87tufjnodg.fsf@wibble.ilmari.org>
 <3662434.1638984660@sss.pgh.pa.us>
 <b7e1cbbf-d4dd-8c33-42be-c1a81b162363@enterprisedb.com>
 <e8a8768c-81a6-2eca-db96-f6e741b4ffad@enterprisedb.com>
MIME-Version: 1.0
Content-Type: text/plain; charset=us-ascii
Content-Disposition: inline
In-Reply-To: <e8a8768c-81a6-2eca-db96-f6e741b4ffad@enterprisedb.com>
Archived-At: <https://www.postgresql.org/message-id/20220224150033.5lql2lkiv7y5kkme%40alap3.anarazel.de>
Precedence: bulk

Hi,

On 2022-02-24 11:16:03 +0100, Peter Eisentraut wrote:
> Now that we have switched everything to done_testing(), the subtests feature
> isn't that relevant anymore, but it might still be useful to get better
> output when running with PROVE_FLAGS=--verbose.

I've incidentally played with subtests yesterdays, when porting
src/interfaces/libpq/test/regress.pl to a tap test. Unfortunately it seems
that subtests aren't actually specified in the tap format, and that different
libraries generate different output formats. The reason this matters somewhat
is that meson's testrunner can parse tap and give nicer progress / error
reports. But since subtests aren't in the spec it can't currently parse
them...

Open issue since 2015:
https://github.com/TestAnything/Specification/issues/2

The perl ecosystem is so moribund :(.


> t/001_basic.pl ..
> # Subtest: vacuumlo --help
>     ok 1 - exit code 0
>     ok 2 - goes to stdout
>     ok 3 - nothing to stderr
>     1..3

It's clearly better.

We can approximate some of it by using is_deeply() and comparing exit, stdout,
stderr at once. Particularly for helpers like program_help() that are used in
a lot of places.

Greetings,

Andres Freund