public inbox for [email protected]
help / color / mirror / Atom feedFrom: Andrew Dunstan <[email protected]>
To: Jelte Fennema-Nio <[email protected]>
To: Tom Lane <[email protected]>
Cc: Robert Haas <[email protected]>
Cc: Jacob Champion <[email protected]>
Cc: Daniel Gustafsson <[email protected]>
Cc: Andres Freund <[email protected]>
Cc: PostgreSQL Hackers <[email protected]>
Subject: Re: RFC: adding pytest as a supported test framework
Date: Sat, 15 Jun 2024 10:45:16 -0400
Message-ID: <[email protected]> (raw)
In-Reply-To: <CAGECzQTOE1KdTBwq3vfXFH4dtXSGt+bcP5zU8aXVA9XJEfQUog@mail.gmail.com>
References: <CAOYmi+kThkM9Z87u=R_Wi7fCor2i+UZKAyq0UCyprzCwTQvqgA@mail.gmail.com>
<[email protected]>
<[email protected]>
<CA+Tgmobhsqh8c+ySXaPf5MdVwFYW6ceQ05N7pav6dzecxJGJoQ@mail.gmail.com>
<CAGECzQTtGoPGQMZactCAytbPvNcA59Zk7RA7SEmzObyAm7K=5w@mail.gmail.com>
<CA+TgmoZCQtUu4u7XJ1umD8k5pTuM3h2zLDyQ1uoVOP-==gJz_Q@mail.gmail.com>
<CAGECzQTzjVaMrKi5SCqL+H63pOx8r1DP0JpJ0v2SfaE0fMaNmg@mail.gmail.com>
<CA+TgmoYvLLAJthL9ev-qc3gVLYX5WC3y3T5hsObaEnVOAx+e6g@mail.gmail.com>
<CAGECzQQ6nHqhwBOr3fW_+_nxn82TJxMPZGAeXD-WKKJPYHYQqw@mail.gmail.com>
<CA+TgmoboC3zPe5uzY58_QxGaGexGGpSBTNSDMZ4_Tzo-nrne=w@mail.gmail.com>
<CAOYmi+moGFuBaaAHq=kixGk0YuOZan09Wn5O4WAzA-xLTEumaA@mail.gmail.com>
<CA+TgmoY4DRMVEV7CVo1D8p==ExsN69W89rF7-Ax4V=M9HGWL7g@mail.gmail.com>
<CA+TgmoZV8B35Mvx8DqHtJjHDQxscJhwOWzNWiPj+dsUhwaUfSg@mail.gmail.com>
<[email protected]>
<CAGECzQTOE1KdTBwq3vfXFH4dtXSGt+bcP5zU8aXVA9XJEfQUog@mail.gmail.com>
On 2024-06-14 Fr 18:11, Jelte Fennema-Nio wrote:
> On Fri, 14 Jun 2024 at 17:49, Tom Lane<[email protected]> wrote:
>> But what I'd really like to see is some comparison of the
>> language-provided testing facilities that we're proposing
>> to depend on. Why is pytest (or whatever) better than Test::More?
> Some advantages of pytest over Test::More:
>
> 1. It's much easier to debug failing tests using the output that
> pytest gives. A good example of this is on pytest its homepage[1]
> (i.e. it shows the value given to the call to inc in the error)
> 2. No need to remember a specific comparison DSL
> (is/isnt/is_deeply/like/ok/cmp_ok/isa_ok), just put assert in front of
> a boolean expression and your output is great (if you want to add a
> message too for clarity you can use: assert a == b, "the world is
> ending")
> 3. Very easy to postgres log files on stderr/stdout when a test fails.
> This might be possible/easy with Perl too, but we currently don't do
> that. So right now for many failures you're forced to traverse the
> build/testrun/... directory tree to find the logs.
I see the fact that we stash the output in a file as a feature. Without
it, capturing failure information in the buildfarm client would be more
difficult, especially if there are multiple failures. So this is
actually something I think we would need for any alternative framework.
Maybe we need an environment setting that would output the
regress_log_00whatever file to stderr on failure. That should be pretty
easy to arrange in the END handler for PostgreSQL::Test::Utils.
> 4. Pytest has autodiscovery of test files and functions, so we
> probably wouldn't have to specify all of the exact test files anymore
> in the meson.build files.
I find this comment a bit ironic. We don't need to do that with the
Makefiles, and the requirement to do so was promoted as a meson feature
rather than a limitation, ISTR.
> Regarding 2, there are ~150 checks that are using a suboptimal way of
> testing for a comparison. Mostly a lot that could use "like(..., ...)"
> instead of "ok(... ~= ...)"
> ❯ grep '\bok(.*=' **.pl | wc -l
> 151
Well, let's fix those. I would be tempted to use cmp_ok() for just about
all of them.
But the fact that Test::More has a handful of test primitives rather
than just one strikes me as a relatively minor complaint.
cheers
andrew
--
Andrew Dunstan
EDB: https://www.enterprisedb.com
view thread (77+ messages) latest in thread
reply
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Reply to all the recipients using the --to and --cc options:
reply via email
To: [email protected]
Cc: [email protected], [email protected], [email protected], [email protected], [email protected], [email protected], [email protected]
Subject: Re: RFC: adding pytest as a supported test framework
In-Reply-To: <[email protected]>
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
This inbox is served by agora; see mirroring instructions
for how to clone and mirror all data and code used for this inbox