Subject: Re: Contributing test cases to improve coverage

From: Tom Lane <[email-masked]>
Date: Wed, 12 Jun 2024 14:22:47 -0400

J F <[email-masked]> writes:
> For postgres, I am looking at adding test cases to test suite in
> test/src/regress/. I have gone through (a)-(e), and managed to produced
> some test cases. As an example, I claim the test case
> ```
> CREATE RECURSIVE VIEW a(b) AS SELECT'' ;
> SELECT FROM a WHERE NULL;
> ```
> could kill the following mutation at optimizer/plan/setrefs.c, 502:5--502:33
> Original binary operator expression:
> ```
> rte->rtekind == RTE_SUBQUERY
> ````
> Replacement expression:
> ```
> (rte->rtekind) >= RTE_SUBQUERY
> ```

I am quite confused about what is the point of this.  You have not
found any actual bug, nor have you demonstrated that this test case
could discover a likely future bug that wouldn't be detected another
way.  Moreover, it seems like the process would lead to some very
large number of equally marginal test cases.  We aren't likely to
accept such a patch, because we are concerned about keeping down the
runtime of the test suite.

			regards, tom lane

---

From: J F <[email-masked]>
Date: Wed, 12 Jun 2024 20:51:48 +0100

> runtime of the test suite.

The point of this project is to improve the coverage of PostgreSQL’s
preexisting test suite. Writing a test suite to achieve close to 100%
coverage is challenging, but I have proposed a workflow to automate this
process.

I assert that no test case in the regression test suite currently covers
the comparator in the expression rte->rtekind == RTE_SUBQUERY. I propose
adding a new test case that addresses exactly this. In the future, if
someone accidentally modifies the operator to become >=, it will trigger
incorrect behavior when certain queries are executed. This test case will
catch that issue.

I get that the test cases in /regress are likely reserved for actual bugs
found and are designed to run quickly. Would it be a good idea to have a
separate, more rigorous test suite that runs longer but provides better
code coverage?

Regards,
Jon

---

From: Ashutosh Bapat <[email-masked]>
Date: Thu, 13 Jun 2024 19:29:44 +0530

> coverage is challenging, but I have proposed a workflow to automate this

We monitor code coverage. But this is doing more than that. It will find
out the places in code, which if changed, will cause bugs. That seems
useful to avoid refactoring mistakes, esp. when people rely on regression
tests to tell whether their code changes are sane. But in PostgreSQL, we
rely heavily on reviewers and committers to do that instead of tests.
Still, the tests produced by this tool will help catch bugs that human eyes
can not. As Tom said, managing that battery of tests may not be worth it.
Basically, I am flip-flopping on the usefulness of this effort.

> incorrect behavior when certain queries are executed. This test case will

Usually PostgreSQL developers know that rtekind is an enum so they are very
very unlikely to use anything other than == and !=. Such a change will be
caught by a reviewer. I think many of the tests that this tool will produce
will fall in this category.

> There are practical difficulties like maintaining the expected outputs for
such a large battery of tests. But maybe some external project could.

BTW, have you considered perl tests, isolation tests etc. Tests in regress/
do not cover many subsystems e.g. replication.
-- 
Best Wishes,
Ashutosh Bapat