MIME-Version: 1.0
References: <CAOYmi+kThkM9Z87u=R_Wi7fCor2i+UZKAyq0UCyprzCwTQvqgA@mail.gmail.com>
 <20240610200411.byj6sv2vpgol6wcf@awork3.anarazel.de> <CAOYmi+niQdwFdX7srOiD8zdme_rxEp2m4JGdqzK=+dS6dpV2Og@mail.gmail.com>
 <20240612155040.u6cvatdb5tiwcxci@awork3.anarazel.de> <FAC2FCFE-8C70-4996-9598-34DD7596BB41@yesql.se>
 <CA+Tgmobhsqh8c+ySXaPf5MdVwFYW6ceQ05N7pav6dzecxJGJoQ@mail.gmail.com>
 <CAGECzQTtGoPGQMZactCAytbPvNcA59Zk7RA7SEmzObyAm7K=5w@mail.gmail.com>
 <CA+TgmoZCQtUu4u7XJ1umD8k5pTuM3h2zLDyQ1uoVOP-==gJz_Q@mail.gmail.com>
 <CAGECzQTzjVaMrKi5SCqL+H63pOx8r1DP0JpJ0v2SfaE0fMaNmg@mail.gmail.com>
 <CA+TgmoYvLLAJthL9ev-qc3gVLYX5WC3y3T5hsObaEnVOAx+e6g@mail.gmail.com>
 <CAGECzQQ6nHqhwBOr3fW_+_nxn82TJxMPZGAeXD-WKKJPYHYQqw@mail.gmail.com>
 <CA+TgmoboC3zPe5uzY58_QxGaGexGGpSBTNSDMZ4_Tzo-nrne=w@mail.gmail.com>
 <CAOYmi+moGFuBaaAHq=kixGk0YuOZan09Wn5O4WAzA-xLTEumaA@mail.gmail.com>
 <CA+TgmoY4DRMVEV7CVo1D8p==ExsN69W89rF7-Ax4V=M9HGWL7g@mail.gmail.com>
 <CA+TgmoZV8B35Mvx8DqHtJjHDQxscJhwOWzNWiPj+dsUhwaUfSg@mail.gmail.com> <2302192.1718380169@sss.pgh.pa.us>
In-Reply-To: <2302192.1718380169@sss.pgh.pa.us>
From: Jelte Fennema-Nio <postgres@jeltef.nl>
Date: Sat, 15 Jun 2024 00:11:04 +0200
Message-ID: <CAGECzQTOE1KdTBwq3vfXFH4dtXSGt+bcP5zU8aXVA9XJEfQUog@mail.gmail.com>
Subject: Re: RFC: adding pytest as a supported test framework
To: Tom Lane <tgl@sss.pgh.pa.us>
Cc: Robert Haas <robertmhaas@gmail.com>, Jacob Champion <jacob.champion@enterprisedb.com>, 
	Daniel Gustafsson <daniel@yesql.se>, Andres Freund <andres@anarazel.de>, 
	PostgreSQL Hackers <pgsql-hackers@postgresql.org>
Content-Type: text/plain; charset="UTF-8"
Content-Transfer-Encoding: quoted-printable
Archived-At: <https://www.postgresql.org/message-id/CAGECzQTOE1KdTBwq3vfXFH4dtXSGt%2BbcP5zU8aXVA9XJEfQUog%40mail.gmail.com>
Precedence: bulk

On Fri, 14 Jun 2024 at 17:49, Tom Lane <tgl@sss.pgh.pa.us> wrote:
> But what I'd really like to see is some comparison of the
> language-provided testing facilities that we're proposing
> to depend on.  Why is pytest (or whatever) better than Test::More?

Some advantages of pytest over Test::More:

1. It's much easier to debug failing tests using the output that
pytest gives. A good example of this is on pytest its homepage[1]
(i.e. it shows the value given to the call to inc in the error)
2. No need to remember a specific comparison DSL
(is/isnt/is_deeply/like/ok/cmp_ok/isa_ok), just put assert in front of
a boolean expression and your output is great (if you want to add a
message too for clarity you can use: assert a =3D=3D b, "the world is
ending")
3. Very easy to postgres log files on stderr/stdout when a test fails.
This might be possible/easy with Perl too, but we currently don't do
that. So right now for many failures you're forced to traverse the
build/testrun/... directory tree to find the logs.
4. Pytest has autodiscovery of test files and functions, so we
probably wouldn't have to specify all of the exact test files anymore
in the meson.build files.

Regarding 2, there are ~150 checks that are using a suboptimal way of
testing for a comparison. Mostly a lot that could use "like(..., ...)"
instead of "ok(... ~=3D ...)"
=E2=9D=AF grep '\bok(.*=3D' **.pl | wc -l
151

[1]: https://docs.pytest.org/en/8.2.x/#a-quick-example