Subject: Re: Starting build-system work (Windows/Mac users please speak up)
To: Kartik Ohri <kartikohri13@gmail.com>
References: <5EC17E17.7070002@anastigmatix.net>
 <5ED0FCB0.405@anastigmatix.net> <5ED10C6D.2020000@anastigmatix.net>
 <5ED10F36.7060304@anastigmatix.net>
 <CAASLQ4PAQ0PNAxJ5ad9U6o+RT2ip19FXvgoK2y1kGWaJbTZ1WQ@mail.gmail.com>
 <5ED15825.9060504@anastigmatix.net>
 <CAASLQ4OXAb+_J=oK54PH5eYuXjkV9KMsoXEcp-TYaQPqxk1Xbg@mail.gmail.com>
 <5ED270BA.9010001@anastigmatix.net> <5ED3D488.1010401@anastigmatix.net>
 <5ED474CA.3000106@anastigmatix.net>
 <CAASLQ4PfvmgKPp+aFf_r5RsBwigc+EC5Hm2CuzwJSTDCT64i-A@mail.gmail.com>
 <5ED4EA10.4030801@anastigmatix.net>
 <CAASLQ4Px21jyb2vJYSrjF5P0RqhA-G_DDWR9QHG68r+W+AqbxQ@mail.gmail.com>
 <CAASLQ4NYE-TW6hjbkSOuAY2Z4NM2cYzqSQ9ahO2LAUhdP9axeQ@mail.gmail.com>
 <5ED7E74B.9020800@anastigmatix.net>
 <CAASLQ4Od6-0=3+4ENKh1Z-Wcu5Wir-bwUn_ccO2UCmDhoR4vtQ@mail.gmail.com>
 <5EDA576E.2000204@anastigmatix.net>
 <CAASLQ4OzPwZB6GG_cXNW4WU4zpHa6wm8Q3WRWNv5pZpis+gf3w@mail.gmail.com>
 <5EDBA5D9.8090506@anastigmatix.net>
 <CAASLQ4P8e9XfUh17HFhYE4eMtq5-C4dpHQAX+hCsia-UFNAjtA@mail.gmail.com>
Cc: pljava-dev@lists.postgresql.org
From: Chapman Flack <chap@anastigmatix.net>
Message-ID: <5EE7AE8C.2040201@anastigmatix.net>
Date: Mon, 15 Jun 2020 13:23:24 -0400
User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:38.0) Gecko/20100101
 Thunderbird/38.7.0
MIME-Version: 1.0
In-Reply-To: 
 <CAASLQ4P8e9XfUh17HFhYE4eMtq5-C4dpHQAX+hCsia-UFNAjtA@mail.gmail.com>
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
Precedence: bulk

On 06/15/20 12:47, Kartik Ohri wrote:
> been able to redirect all the output to stdout instead. However, this
> redirects errors as well without failing the build. I am thinking of
> manually processing the output to catch errors and manually terminate the
> build in such a case. However, I think there might be a simpler solution to
> use psql in quiet mode. What are your views on this ?


I was thinking about that last night again. The current "tests" in the
examples are written to report most problems they test for as warnings
and not errors, because the first error would stop everything, and I would
rather see as many issues as possible. (Of course you still get an error
if the test itself actually breaks, which also counts as finding a problem!
You just don't find the ones after it then.)

I was looking through psql's docs again last night because I really wish
it had some kind of

\if any-warnings-happened
  \quit status=1
\endif

which seem like surprising things for it to be missing, but as far as
I can tell, it really is missing both: no way to test for warnings (it's
clearly possible, because server responses do include them), and no way
to specify an exit status to \quit. Weird.

(You might check the psql docs yourself to be sure I haven't missed
something. It seems very weird for those abilities to be lacking, but
I'm honestly not seeing them.)

In part, this is because the tests in the examples DDR aren't written
with a more conventional testing framework. What I would really like to
do is move to a combination of pgTAP in the database and tap4j in Java,
and integrate them by providing a method maybe executeQueryAsTAP(qry),
where you give it an SQL qry that produces pgTAP output, and the method
knows that and parses the results and gives you a tap4j results object.
Also the inverse, a PL/Java function that returns a tap4j object as a
TAP-formatted query result.

That's a longer-term goal; there were a couple prerequisites needing
to happen in tap4j, and I sent one and they merged it [1], and I suggested
another and didn't have time to do it myself and no one else has yet [2],
though I see they recently added a "good first issue" tag for it.

Of course assuming the tap4j prerequisites get implemented, there'd
still be the matter of rewriting all the PL/Java tests as tap. Which is
definitely what I want, but it would still be work.

So for now we're stuck with test failures as warnings, and psql having
(unless I missed something) no good way to tell you there were warnings.
That leads down the ugly road of you trying to parse log text and look
for warnings, and not make the nar-maven-plugin-like mistake of thinking
later lines of warnings are errors, or be fooled by keywords inside other
messages, and all of those well-known horrors.

Unless ... we use something besides psql to connect to the database and
run the tests. For example, make travis_setup_pljava and travis_test_pljava
into a Python script using one of Python's pgsql drivers (there are a few
choices) to connect to the server and run the SET, CREATE EXTENSION, and
SELECT commands, and notice any warnings. Or the same in another language
if you prefer it to Python.

(I'm not very familiar with Powershell but I was thinking it sounded
powerful enough it might directly support ODBC connections or something,
and sure enough, [3].)

Then it's a simple matter of using a real API to do the queries and
retrieve whatever warnings came back.

It is worth checking that the choice of script language and PostgreSQL
driver will actually support retrieving warnings! They all should,
but apparently there's a driver for Go at least [4] that doesn't. That
would be frustrating, especially to find out after doing some work.

A final issue is whether to try to do this in one common script language
across platforms. It might be natural to do something in Python on Linux
and in Powershell on Windows (though I guess you can get Python on Windows
or even Powershell on Linux, but relying on an ODBC driver binary could
add complexity).

One language guaranteed to be present on any platform for testing PL/Java
will be ... Java, and that comes with jshell starting with Java 9, so
there could be that argument for writing a jshell script and using pgjdbc.
Fewer other dependencies then, and no native binary ones.

Regards,
-Chap


[1] https://github.com/tupilabs/tap4j/pull/38
[2] https://github.com/tupilabs/tap4j/issues/39
[3]
https://stackoverflow.com/questions/9217650/connect-to-remote-postgresql-database-using-powershell
[4]
https://stackoverflow.com/questions/42070023/how-to-get-postgresql-procedure-warning-messages