MIME-Version: 1.0
References: <CAOYmi+kThkM9Z87u=R_Wi7fCor2i+UZKAyq0UCyprzCwTQvqgA@mail.gmail.com>
 <20240610200411.byj6sv2vpgol6wcf@awork3.anarazel.de> <CAOYmi+niQdwFdX7srOiD8zdme_rxEp2m4JGdqzK=+dS6dpV2Og@mail.gmail.com>
 <20240612155040.u6cvatdb5tiwcxci@awork3.anarazel.de> <FAC2FCFE-8C70-4996-9598-34DD7596BB41@yesql.se>
 <CA+Tgmobhsqh8c+ySXaPf5MdVwFYW6ceQ05N7pav6dzecxJGJoQ@mail.gmail.com>
 <CAGECzQTtGoPGQMZactCAytbPvNcA59Zk7RA7SEmzObyAm7K=5w@mail.gmail.com>
 <CA+TgmoZCQtUu4u7XJ1umD8k5pTuM3h2zLDyQ1uoVOP-==gJz_Q@mail.gmail.com>
 <CAGECzQTzjVaMrKi5SCqL+H63pOx8r1DP0JpJ0v2SfaE0fMaNmg@mail.gmail.com>
 <CA+TgmoYvLLAJthL9ev-qc3gVLYX5WC3y3T5hsObaEnVOAx+e6g@mail.gmail.com>
 <CAGECzQQ6nHqhwBOr3fW_+_nxn82TJxMPZGAeXD-WKKJPYHYQqw@mail.gmail.com>
 <CA+TgmoboC3zPe5uzY58_QxGaGexGGpSBTNSDMZ4_Tzo-nrne=w@mail.gmail.com>
 <CAOYmi+moGFuBaaAHq=kixGk0YuOZan09Wn5O4WAzA-xLTEumaA@mail.gmail.com>
 <CA+TgmoY4DRMVEV7CVo1D8p==ExsN69W89rF7-Ax4V=M9HGWL7g@mail.gmail.com>
 <CA+TgmoZV8B35Mvx8DqHtJjHDQxscJhwOWzNWiPj+dsUhwaUfSg@mail.gmail.com> <2302192.1718380169@sss.pgh.pa.us>
In-Reply-To: <2302192.1718380169@sss.pgh.pa.us>
From: Jacob Champion <jacob.champion@enterprisedb.com>
Date: Tue, 18 Jun 2024 07:35:35 -0700
Message-ID: <CAOYmi+k214g1tgREvQkUcr=OCiVKyaTDu1ja+OGAygG1y=jhPQ@mail.gmail.com>
Subject: Re: RFC: adding pytest as a supported test framework
To: Tom Lane <tgl@sss.pgh.pa.us>
Cc: Robert Haas <robertmhaas@gmail.com>, Jelte Fennema-Nio <postgres@jeltef.nl>, 
	Daniel Gustafsson <daniel@yesql.se>, Andres Freund <andres@anarazel.de>, 
	PostgreSQL Hackers <pgsql-hackers@postgresql.org>
Content-Type: text/plain; charset="UTF-8"
Content-Transfer-Encoding: quoted-printable
Archived-At: <https://www.postgresql.org/message-id/CAOYmi%2Bk214g1tgREvQkUcr%3DOCiVKyaTDu1ja%2BOGAygG1y%3DjhPQ%40mail.gmail.com>
Precedence: bulk

On Fri, Jun 14, 2024 at 8:49=E2=80=AFAM Tom Lane <tgl@sss.pgh.pa.us> wrote:
> I think that's an oversimplified analysis.  Sure, the languages are
> both Turing-complete, but for our purposes here they are both simply
> glue languages around some set of testing facilities.  Some of those
> facilities will be provided by the base languages (plus whatever
> extension modules we choose to require) and some by code we write.
> The overall experience of writing tests will be determined by the
> testing facilities far more than by the language used for glue.

+1. As an example, the more extensive (and high-quality) a language's
standard library, the more tests you'll be able to write. Convincing
committers to adopt a new third-party dependency is hard (for good
reason); the strength of the standard library should be considered as
a point of technical comparison.

> That being the case, I do agree with your point that Python
> equivalents to most of PostgreSQL::Test will need to be built up PDQ.
> Maybe they can be better than the originals, in features or ease of
> use, but "not there at all" is not better.

There's a wide gulf between "not there at all" and "reimplement it all
as a prerequisite for v1" as Robert proposed. I'm arguing for a middle
ground.

> But what I'd really like to see is some comparison of the
> language-provided testing facilities that we're proposing
> to depend on.  Why is pytest (or whatever) better than Test::More?

People are focusing a lot on failure reporting, and I agree with them,
but I did try to include more than just that in my OP.

I'll requote what I personally think is the #1 killer feature of
pytest, which prove and Test::More appear to be unable to accomplish
on their own: configurable isolation of tests from each other via
fixtures [1].

    Problem 1 (rerun failing tests): One architectural roadblock to this
    in our Test::More suite is that tests depend on setup that's done by
    previous tests. pytest allows you to declare each test's setup
    requirements via pytest fixtures, letting the test runner build up the
    world exactly as it needs to be for a single isolated test. These
    fixtures may be given a "scope" so that multiple tests may share the
    same setup for performance or other reasons.

When I'm doing red-to-green feature development (e.g. OAuth) or
green-to-green refactoring (e.g. replacement of libiddawc with libcurl
in OAuth), quick cycle time and reduction of noise is extremely
important. I want to be able to rerun just the single red test I care
about before moving on.

(Tests may additionally be organized with custom attributes. My OAuth
suite contains tests that must run slowly due to mandatory timeouts;
I've marked them as slow, and they can be easily skipped from the test
runner.)

2. The ability to break into a test case with the built-in debugger
[2] is also fantastic for quick red-green work. Much better than
print() statements.

(Along similar lines, even the ability to use Python's built-in REPL
increases velocity. Python users understand that they can execute
`python3` and be dropped into a sandbox to try out syntax or some
unfamiliar library. Searching for how to do this in Perl results in a
handful of custom-built scripts; people here may know which to use as
a Perl monk, sure, but the point is to make it easy for newcomers to
write tests.)

> I also wonder about integration of python-based testing with what
> we already have.  A significant part of what you called the meson
> work had to do with persuading pg_regress, isolationtester, etc
> to output test results in the common format established by TAP.
> Am I right in guessing that pytest will have nothing to do with that?

Andres covered this pretty well. I will note that I had problems with
pytest-tap itself [3], and I'm unclear whether that represents a bug
in pytest-tap or a bug in pytest.

Thanks,
--Jacob

[1] https://docs.pytest.org/en/stable/how-to/fixtures.html
[2] https://docs.pytest.org/en/stable/how-to/failures.html
[3] https://github.com/python-tap/pytest-tap/issues/30