Feedback-ID: id4a34324:Fastmail
Date: Mon, 4 Jul 2022 13:06:17 -0700
From: Andres Freund <andres@anarazel.de>
To: Tom Lane <tgl@sss.pgh.pa.us>
Cc: Peter Eisentraut <peter.eisentraut@enterprisedb.com>,
	Daniel Gustafsson <daniel@yesql.se>,
	PostgreSQL Developers <pgsql-hackers@lists.postgresql.org>
Subject: Re: TAP output format in pg_regress
Message-ID: <20220704200617.vmuu3cuhhfmnzzpm@awork3.anarazel.de>
References: <BD4B107D-7E53-4794-ACBA-275BEB4327C9@yesql.se>
 <49A54862-C682-4B60-9155-05F178FA79DC@yesql.se>
 <20220321234924.kcc3q26o7sx6rnfa@alap3.anarazel.de>
 <E35C4A74-7265-4106-908E-5D3E5B9D8C2E@yesql.se>
 <489D2B5A-B26F-4918-BC07-B426329169E3@yesql.se>
 <5e6bc6a6-0a4e-109d-2b7d-c286dc58d609@enterprisedb.com>
 <1558373.1656945577@sss.pgh.pa.us>
MIME-Version: 1.0
Content-Type: text/plain; charset=us-ascii
Content-Disposition: inline
In-Reply-To: <1558373.1656945577@sss.pgh.pa.us>
Archived-At: 
 <https://www.postgresql.org/message-id/20220704200617.vmuu3cuhhfmnzzpm%40awork3.anarazel.de>
Precedence: bulk

Hi,

> Peter Eisentraut <peter.eisentraut@enterprisedb.com> writes:
> > I'm not sure what to make of all these options.  I think providing a TAP
> > output for pg_regress is a good idea.  But then do we still need the old
> > output?  Is it worth maintaining two output formats that display exactly
> > the same thing in slightly different ways?
>
> Probably is, because this is bad:

> > ... The proposed default format now hides the
> > fact that some tests are started in parallel.  I remember the last time
> > I wanted to tweak the output of the parallel tests, people were very
> > attached to the particular timing and spacing of the current output.  So
> > I'm not sure people will like this.
>
> and so is this:
>
> > The timing output is very popular.  Where is that in the TAP output?
>
> Both of those things are fairly critical for test development.  You
> need to know what else might be running in parallel with a test case,
> and you need to know whether you just bloated the runtime unreasonably.

That should be doable with tap as well - afaics the output of that could
nearly be the same as now, preceded by a #.

The test timing output could (and I think should) also be output - but if I
read the tap specification correctly, we'd either need to make it part of the
test "description" or on a separate line.


On 2022-07-04 10:39:37 -0400, Tom Lane wrote:
> More generally, I'm unhappy about the proposal that TAP should become
> the default output.  There is nothing particularly human-friendly
> about it, whereas the existing format is something we have tuned to
> our liking over literally decades.  I don't mind if there's a way to
> get TAP when you're actually intending to feed it into a TAP-parsing
> tool, but I am not a TAP-parsing tool and I don't see why I should
> have to put up with it.

I'm mostly interested in the tap format because meson's testrunner can parse
it - unsurprisingly it doesn't understand the current regress output. It's a
lot nicer to immediately be pointed to the failed test(s) than having to scan
through the output "manually".

Greetings,

Andres Freund