Content-Type: text/plain;
	charset=us-ascii
Mime-Version: 1.0 (Mac OS X Mail 16.0 \(3696.100.31\))
Subject: Re: TAP output format in pg_regress
From: Daniel Gustafsson <daniel@yesql.se>
In-Reply-To: <5e6bc6a6-0a4e-109d-2b7d-c286dc58d609@enterprisedb.com>
Date: Tue, 5 Jul 2022 00:06:04 +0200
Cc: Andres Freund <andres@anarazel.de>,
 PostgreSQL Developers <pgsql-hackers@lists.postgresql.org>
Content-Transfer-Encoding: quoted-printable
Message-Id: <461F8D8E-77D8-4F15-84C6-F7FD239018C0@yesql.se>
References: <BD4B107D-7E53-4794-ACBA-275BEB4327C9@yesql.se>
 <49A54862-C682-4B60-9155-05F178FA79DC@yesql.se>
 <20220321234924.kcc3q26o7sx6rnfa@alap3.anarazel.de>
 <E35C4A74-7265-4106-908E-5D3E5B9D8C2E@yesql.se>
 <489D2B5A-B26F-4918-BC07-B426329169E3@yesql.se>
 <5e6bc6a6-0a4e-109d-2b7d-c286dc58d609@enterprisedb.com>
To: Peter Eisentraut <peter.eisentraut@enterprisedb.com>
Archived-At: 
 <https://www.postgresql.org/message-id/461F8D8E-77D8-4F15-84C6-F7FD239018C0%40yesql.se>
Precedence: bulk

> On 4 Jul 2022, at 16:27, Peter Eisentraut =
<peter.eisentraut@enterprisedb.com> wrote:
>=20
> On 29.06.22 21:50, Daniel Gustafsson wrote:
>> Attached is a new version of this patch, which completes the TAP =
output format
>> option such that all codepaths emitting output are TAP compliant.  =
The verbose
>> option is fixed to to not output extraneous newlines which the =
previous PoC
>> did.  The output it made to conform to the original TAP spec since =
v13/14 TAP
>> parsers seem less common than those that can handle the original =
spec.  Support
>> for the new format additions should be quite simple to add should we =
want that.
>> Running pg_regress --verbose should give the current format output.
>> I did end up combining TAP and --verbose into a single patch, as the =
TAP format
>> sort of depends on the verbose flag as TAP has no verbose mode.  I =
can split it
>> into two separate should a reviewer prefer that.
>=20
> I'm not sure what to make of all these options.  I think providing a =
TAP output for pg_regress is a good idea.  But then do we still need the =
old output?  Is it worth maintaining two output formats that display =
exactly the same thing in slightly different ways?

If we believe that TAP is good enough for human consumption and not just =
as
input to test runners then we don't.  Personally I think the traditional =
format
is more pleasant to read than raw TAP output when running tests.

> What is the purpose of the --verbose option?  When and how is one =
supposed to activate that?  The proposed default format now hides the =
fact that some tests are started in parallel.

The discussion on this was in =
20220221164736.rq3ornzjdkmwk2wo@alap3.anarazel.de
where it was proposed that we could cut the boilerplate.  Thinking on it =
I
agree that the parallel run info should be included even without =
--verbose so
I'll add that back.  In general, running with --verbose should ideally =
only be
required for troubleshooting setup/teardown issues with testing.

>  I remember the last time I wanted to tweak the output of the parallel =
tests, people were very attached to the particular timing and spacing of =
the current output.  So I'm not sure people will like this.
>=20
> The timing output is very popular.  Where is that in the TAP output?

As I mentioned in the mail upthread, TAP runners generally hide that =
info and
only show a running total.  That being said, I do agree with adding back =
so
I'll do that in a new version of the patch.

> More generally, what do you envision we do with this feature?  Who is =
it for, what are the tradeoffs, etc.?

In general, my thinking with this was that normal testruns started with =
make
check (or similar) by developers would use the traditional format =
(albeit less
verbose) and that the TAP output was for automated test runners in =
general and
the meson test runner in particular.  The TAP format is an opt-in with =
the
traditional format being the default.

The tradeoff is of course that maintaining two output formats is more =
work than
maintaining one, but it's not really something we change all that often =
so that
might not be too heavy a burden.

I personally didn't see us replacing the traditional format for "human
readable" runs, if that's where the discussion is heading then the patch =
can
look quite different.  Having test output format parity with supported =
back
branches seemed like a good idea to me at the time of writing at least.

--
Daniel Gustafsson		https://vmware.com/