Message-ID: <a38765d0-da19-4121-aaf9-cbf111da5461@dunslane.net>
Date: Sat, 15 Jun 2024 10:45:16 -0400
MIME-Version: 1.0
User-Agent: Mozilla Thunderbird
From: Andrew Dunstan <andrew@dunslane.net>
Subject: Re: RFC: adding pytest as a supported test framework
To: Jelte Fennema-Nio <postgres@jeltef.nl>, Tom Lane <tgl@sss.pgh.pa.us>
Cc: Robert Haas <robertmhaas@gmail.com>,
 Jacob Champion <jacob.champion@enterprisedb.com>,
 Daniel Gustafsson <daniel@yesql.se>, Andres Freund <andres@anarazel.de>,
 PostgreSQL Hackers <pgsql-hackers@postgresql.org>
References: <CAOYmi+kThkM9Z87u=R_Wi7fCor2i+UZKAyq0UCyprzCwTQvqgA@mail.gmail.com>
 <20240612155040.u6cvatdb5tiwcxci@awork3.anarazel.de>
 <FAC2FCFE-8C70-4996-9598-34DD7596BB41@yesql.se>
 <CA+Tgmobhsqh8c+ySXaPf5MdVwFYW6ceQ05N7pav6dzecxJGJoQ@mail.gmail.com>
 <CAGECzQTtGoPGQMZactCAytbPvNcA59Zk7RA7SEmzObyAm7K=5w@mail.gmail.com>
 <CA+TgmoZCQtUu4u7XJ1umD8k5pTuM3h2zLDyQ1uoVOP-==gJz_Q@mail.gmail.com>
 <CAGECzQTzjVaMrKi5SCqL+H63pOx8r1DP0JpJ0v2SfaE0fMaNmg@mail.gmail.com>
 <CA+TgmoYvLLAJthL9ev-qc3gVLYX5WC3y3T5hsObaEnVOAx+e6g@mail.gmail.com>
 <CAGECzQQ6nHqhwBOr3fW_+_nxn82TJxMPZGAeXD-WKKJPYHYQqw@mail.gmail.com>
 <CA+TgmoboC3zPe5uzY58_QxGaGexGGpSBTNSDMZ4_Tzo-nrne=w@mail.gmail.com>
 <CAOYmi+moGFuBaaAHq=kixGk0YuOZan09Wn5O4WAzA-xLTEumaA@mail.gmail.com>
 <CA+TgmoY4DRMVEV7CVo1D8p==ExsN69W89rF7-Ax4V=M9HGWL7g@mail.gmail.com>
 <CA+TgmoZV8B35Mvx8DqHtJjHDQxscJhwOWzNWiPj+dsUhwaUfSg@mail.gmail.com>
 <2302192.1718380169@sss.pgh.pa.us>
 <CAGECzQTOE1KdTBwq3vfXFH4dtXSGt+bcP5zU8aXVA9XJEfQUog@mail.gmail.com>
Content-Language: en-US
Autocrypt: addr=andrew@dunslane.net; keydata=
 xsBNBE7KWFkBCAClridxur2AIc7eW2AR7izbfp3EnNefie2HbLF0izW5Ik5UjX2HBXBx4syI
 gY6b0ugohXrr274+baoAlvSbq6cAoQuEVrk5IZFzt20b1Xkx65FwGSEj526yiKLocqkJceSq
 Xr9xcA5SGY+FZv441chh5SU92v4q6z+6LPpoHOh97ptAVXZYNTtU0LevyvD5lja0TzbvJm6C
 eFXitJfnm1pLEr0DGJCR/iUOl/N62Kh4855zZC7NHIjQHPOvV5Stz/l5ilDhvGVk+xkXFPys
 SjZoUr1rXhYLpiyi5sR0X9FHXT0KnGuz1F5ERO7ZTLSSQ6fJwPj6gOk9K+vvoKvoeql5ABEB
 AAHNJEFuZHJldyBEdW5zdGFuIDxhbmRyZXdAZHVuc2xhbmUubmV0PsLAmwQTAQgARQIbAwIX
 gAIZAQULCQgHAgMiAgEGFQoJCAsCBBYCAwECHgcWIQTkPlhGHfx8v0RpFaWZ+n/LWfw7gQUC
 ZFlxxwUJGVGAbgAKCRCZ+n/LWfw7gXikB/9ZdcUy6CTBFIIuL/bVsc1eLEW/gJBjJBF6HxNY
 xgEkAgXAp4Lg4A5U+QB9GouFr7+GYxF0BU4hzoGhNPUWltxnHdMWP8nC/38LAqgMi8L/bbsm
 HW5YPBdWYaAZAPJQVfOAgjTbRUb26KSprpyrrJKW0ZmrZfjhNPcQ72jpWzoPLQqx2X6B0fru
 1jq+cBh8lb6r1mJTim1T3JIn+F/v5VpdQS+EL8xqsHkfzKjIPsW3CIXpkypSk6saA55Rkkbl
 26AW8ftPVB0Q6Lnn6FLt9CP0MGNixBQ55yq8r1K+nCBvCCjvQjM8RDm0UUum0WNl+ifQgTLO
 E8TWEnwVtkBf+3QWzsBNBE7KWFkBCADRnOM0FCzsYW6jtncg+dWIagjUZpvaClmqn/sJluLa
 Q3v1VXMQJzYs3eC1gh386W+XBwLRpDj3jzH81lX+p73Re3d3oJW7X+ffsxuzu5ZVdMUkqBYo
 nkAbKxr6gyJ12F/+JkUVzLcoTN+d/7YsQvUVi7NaKH8mJgjz112O4fUe3p9wfAaFa0RXHc5S
 GPzRTYRRlv/XZBIho4J2tkZOnteZJZ+GbxQVlINt6fd8P6al3MWOvpP/ExJPguEfjOsO6Njy
 xjo3WfpD4lHMOR/Oc3/8mScEF84rF2jXbsFgelWnbPWAvXY+pD0dXOFRkagGmC/viwBDqq5b
 5tk76kKmUbZxABEBAAHCwHwEGAEIACYCGwwWIQTkPlhGHfx8v0RpFaWZ+n/LWfw7gQUCZFlx
 5wUJGVGAjgAKCRCZ+n/LWfw7gf+iB/4g8CPY5jihf5r/8EsoIGe2H+dpVmpPF8YGBzTIvCz/
 fQoOq8AX/pE76QEuFnFZWfjw+wgBXgCVmkox2Eflkk6z4ND3pcwGZ6CfCxTQCDk/dij+2DQ4
 6bmDCy/sBgcbz9mTpoLC11HLoPae6YN9nBNQRZDcEFEu54OaVOqlIdbA6m+POIBCXZdHOFc0
 WoDTgxHRzC1jgQNidyd6tKqcsVJs0dzF0oKTmFFmUAqTdJO12LBuNA1rlqrR3EtpYk8B/wtS
 5dIMD7Q8hwQpL+4C6GNpb6ZKnPkLi47pDOLhz2qBrqN+rqUEsT3YnExYpzj5yOBi+FlmV1Hw
 49QYe1sn2ZPs
In-Reply-To: <CAGECzQTOE1KdTBwq3vfXFH4dtXSGt+bcP5zU8aXVA9XJEfQUog@mail.gmail.com>
Content-Type: text/plain; charset=UTF-8; format=flowed
Content-Transfer-Encoding: 8bit
Archived-At: <https://www.postgresql.org/message-id/a38765d0-da19-4121-aaf9-cbf111da5461%40dunslane.net>
Precedence: bulk


On 2024-06-14 Fr 18:11, Jelte Fennema-Nio wrote:
> On Fri, 14 Jun 2024 at 17:49, Tom Lane<tgl@sss.pgh.pa.us>  wrote:
>> But what I'd really like to see is some comparison of the
>> language-provided testing facilities that we're proposing
>> to depend on.  Why is pytest (or whatever) better than Test::More?
> Some advantages of pytest over Test::More:
>
> 1. It's much easier to debug failing tests using the output that
> pytest gives. A good example of this is on pytest its homepage[1]
> (i.e. it shows the value given to the call to inc in the error)
> 2. No need to remember a specific comparison DSL
> (is/isnt/is_deeply/like/ok/cmp_ok/isa_ok), just put assert in front of
> a boolean expression and your output is great (if you want to add a
> message too for clarity you can use: assert a == b, "the world is
> ending")
> 3. Very easy to postgres log files on stderr/stdout when a test fails.
> This might be possible/easy with Perl too, but we currently don't do
> that. So right now for many failures you're forced to traverse the
> build/testrun/... directory tree to find the logs.


I see the fact that we stash the output in a file as a feature. Without 
it, capturing failure information in the buildfarm client would be more 
difficult, especially if there are multiple failures. So this is 
actually something I think we would need for any alternative framework.

Maybe we need an environment setting that would output the 
regress_log_00whatever file to stderr on failure.  That should be pretty 
easy to arrange in the END handler for PostgreSQL::Test::Utils.


> 4. Pytest has autodiscovery of test files and functions, so we
> probably wouldn't have to specify all of the exact test files anymore
> in the meson.build files.


I find this comment a bit ironic. We don't need to do that with the 
Makefiles, and the requirement to do so was promoted as a meson feature 
rather than a limitation, ISTR.


> Regarding 2, there are ~150 checks that are using a suboptimal way of
> testing for a comparison. Mostly a lot that could use "like(..., ...)"
> instead of "ok(... ~= ...)"
> ❯ grep '\bok(.*=' **.pl | wc -l
> 151


Well, let's fix those. I would be tempted to use cmp_ok() for just about 
all of them.

But the fact that Test::More has a handful of test primitives rather 
than just one strikes me as a relatively minor complaint.


cheers


andrew


-- 

Andrew Dunstan
EDB: https://www.enterprisedb.com