Subject: Re: Starting build-system work (Windows/Mac users please speak up)

From: Chapman Flack <[email-masked]>
Date: Sun, 31 May 2020 23:23:54 -0400

On 05/31/20 12:00, Chapman Flack wrote:
>> - Messages from the JVM itself that are emitted through a 'vfprintf hook'
>>   are caught by PL/Java's hook [1] and injected into PostgreSQL's
>>   elog system at a selectable level, usually INFO. (That is probably
>>   the path being followed by the -Xcheck:jni messages, explaining
>>   why they have INFO: in front.)

Exploiting that fact, I decided to just condense out those messages (only
the ones coming from Java's own JMX classes, outside our power to fix)
right at the source, by putting a small state machine in the vfprintf hook
to recognize and not log them. That's in PR #276.

Also in that PR, changes to fix the remaining -Xcheck:jni messages that
were really coming from PL/Java. For the moment, in my environment,
with PR #275 and #276 applied, it builds and tests -Xcheck:jni clean.

That should eliminate any need for doing fancy stuff with those messages
in the CI configuration. Any new issues that crop up in development later
ought to produce a smaller volume of messages that we want to know about.

Regards,
-Chap

---

From: Kartik Ohri <[email-masked]>
Date: Mon, 1 Jun 2020 14:58:11 +0530

> ought to produce a smaller volume of messages that we want to know about.
So we do not need a custom log filter now, right ?

---

From: Chapman Flack <[email-masked]>
Date: Mon, 1 Jun 2020 07:44:16 -0400

> So we do not need a custom log filter now, right ?

Right. And it's probably ok to include -Xcheck:jni in all configurations
(once PRs #275 and #276 are merged, of course).

Regards,
-Chap

---

From: Kartik Ohri <[email-masked]>
Date: Mon, 1 Jun 2020 17:34:47 +0530

Great! So, I'll begin work on setting up CI for Windows. Once, these PRs
are merged I'll update my repository and run a build with -Xcheck:jni in
all configurations to a full test for Linux and macOS.

---

From: Kartik Ohri <[email-masked]>
Date: Wed, 3 Jun 2020 22:54:24 +0530

Hi! I have begun the setup on Windows. Here, is the log
https://ci.appveyor.com/project/amCap1712/pljava.I get the error Command
executed with exception: ERROR: ALTER SYSTEM cannot run inside a
transaction bloc at psql -c "SET pljava.libjvm_location to '$libjvm'; ALTER
SYSTEM SET pljava.libjvm_location to '$libjvm'; SELECT pg_reload-conf();"
-U postgres . Can you help with this?

---

From: Chapman Flack <[email-masked]>
Date: Wed, 3 Jun 2020 14:09:15 -0400

On 06/03/20 13:24, Kartik Ohri wrote:
> ERROR: ALTER SYSTEM cannot run inside a
> -U postgres .

Hmm. psql does have a --single-transaction mode that, if enabled, makes
psql send a BEGIN before your first command and a COMMIT after the last
one, so that if you have multiple commands, they all happen in one
transaction. [1]

If one of your commands is something that's not allowed inside
a transaction block, that of course will be a problem.

What surprises me is that you see this happening on Windows only.
I wonder if psql is built with different option defaults there? Or could
there be a .psqlrc (or whatever it is called in Windows) that is setting
that option? Those are the kind of things I would look for.

That said, there are alternatives to the whole ALTER SYSTEM / pg_reload_conf
business. If you are going to include all of the testing commands within
one psql connection, there isn't any need for the ALTER SYSTEM or reload,
you can just send the SET commands first and then the others in the same
session. Or, if you do want to save the settings for later reconnection,
you could also do that with ALTER DATABASE or ALTER USER, and I think
both of those will work inside transactions.

Another alternative is to simply pass the settings with -c when
starting the PostgreSQL server [2]:

postgres -c pljava.libjvm_location="$libjvm" \
  -c pljava.vmoptions='-ea:org.postgresql.pljava... -Xcheck:jni'

(Note I'm writing Unixy syntax here; I don't use Windows enough to be
sure of the right way to quote that, but it gives you the idea.)

Those settings will apply to all sessions started in that run of the server.
It would be odd to run a production server that way, but one that is just
being started to run some tests, why not?

And still another alternative is just to send the options as part of the
psql connection string [3]:

psql "dbname=postgres options='-c pljava.libjvm_location=$libjvm
  -c pljava.vmoptions=-ea:org.postgresql.pljava...\\\ -Xcheck:jni'"

... where again, I'm showing Unixy syntax. The tricky bit is getting a
space inside a single option setting, which you can see takes three
backslashes in the above. I have no idea what would be correct in Windows
but if you figure it out, it's good practice for the Maven plugin work.

Settings sent with the psql connection of course are in effect just for
that session.

Regards,
-Chap

[1] https://www.postgresql.org/docs/13/app-psql.html#R1-APP-PSQL-3

[2] https://www.postgresql.org/docs/13/app-postgres.html#id-1.9.5.14.6.3

[3] https://www.postgresql.org/docs/13/libpq-connect.html#LIBPQ-CONNSTRING

---

From: Kartik Ohri <[email-masked]>
Date: Fri, 5 Jun 2020 19:14:25 +0530

Hi! I think ALTER USER will do the job but I am again facing an issue with
the pljava.libjvm_location. This works fine on Linux and macOS but not on
Windows.  I have tried escaping the / and used \ as well but none seems to
be working.

Here are the pljava variables.

select name, source, setting from pg_settings where name like 'pljava.%';
                name                              | source  |
                                                      setting

----------------------------------------------+---------+-------------------------------------------------------------------------------------------------------------------------------------------------
 pljava.debug                                  | default | off
 pljava.enable                                 | default | on
 pljava.implementors                      | default | postgresql
 pljava.java_thread_pg_entry         | default | allow
 pljava.libjvm_location                    | session | C:/Program
Files/Java/jdk14/bin/server/jvm.dll
 pljava.module_path                       | default | C:/Program
Files/PostgreSQL/12/share/pljava/pljava-1.6.0-SNAPSHOT.jar;C:/Program
Files/PostgreSQL/12/share/pljava/pljava-api-1.6.0-SNAPSHOT.jar
 pljava.release_lingering_savepoints | default | off
 pljava.statement_cache_size        | default | 11
 pljava.vmoptions                           | default |

And here is the psql log,

2020-06-05 13:25:06.558 UTC [5652] LOG:  database system was shut down at
2020-05-27 01:18:49 UTC
2020-06-05 13:25:06.585 UTC [1056] LOG:  database system is ready to accept
connections
2020-06-05 13:27:20.207 UTC [1056] LOG:  received SIGHUP, reloading
configuration files
2020-06-05 13:27:20.385 UTC [3096] WARNING:  Java virtual machine not yet
loaded
2020-06-05 13:27:20.385 UTC [3096] DETAIL:  Access is denied.

2020-06-05 13:27:20.385 UTC [3096] HINT:  SET pljava.libjvm_location TO the
correct path to the jvm library (libjvm.so or jvm.dll, etc.)
2020-06-05 13:27:20.395 UTC [3096] ERROR:  cannot use PL/Java before
successfully completing its setup
2020-06-05 13:27:20.395 UTC [3096] HINT:  Check the log for messages
closely preceding this one, detailing what step of setup failed and what
will be needed, probably setting one of the "pljava." configuration
variables, to complete the setup. If there is not enough help in the log,
try again with different settings for "log_min_messages" or
"log_error_verbosity".
2020-06-05 13:27:20.395 UTC [3096] STATEMENT:  CREATE EXTENSION pljava;
2020-06-05 13:36:27.036 UTC [5152] ERROR:  unrecognized configuration
parameter "client_min_messgaes"
2020-06-05 13:36:27.036 UTC [5152] STATEMENT:  set client_min_messgaes to
DEBUG1;
2020-06-05 13:36:32.400 UTC [5152] ERROR:  unrecognized configuration
parameter "client_min_messgages"
2020-06-05 13:36:32.400 UTC [5152] STATEMENT:  set client_min_messgages to
DEBUG1;
2020-06-05 13:37:58.758 UTC [5152] WARNING:  Java virtual machine not yet
loaded
2020-06-05 13:37:58.758 UTC [5152] DETAIL:  Access is denied.

2020-06-05 13:37:58.758 UTC [5152] HINT:  SET pljava.libjvm_location TO the
correct path to the jvm library (libjvm.so or jvm.dll, etc.)
2020-06-05 13:37:58.762 UTC [5152] ERROR:  cannot use PL/Java before
successfully completing its setup
2020-06-05 13:37:58.762 UTC [5152] HINT:  Check the log for messages
closely preceding this one, detailing what step of setup failed and what
will be needed, probably setting one of the "pljava." configuration
variables, to complete the setup. If there is not enough help in the log,
try again with different settings for "log_min_messages" or
"log_error_verbosity".
2020-06-05 13:37:58.762 UTC [5152] STATEMENT:  CREATE EXTENSION pljava;
2020-06-05 13:38:50.100 UTC [5152] WARNING:  Java virtual machine not yet
loaded
2020-06-05 13:38:50.100 UTC [5152] DETAIL:  Access is denied.

2020-06-05 13:38:50.100 UTC [5152] HINT:  SET pljava.libjvm_location TO the
correct path to the jvm library (libjvm.so or jvm.dll, etc.)
2020-06-05 13:39:26.369 UTC [5152] WARNING:  Java virtual machine not yet
loaded
2020-06-05 13:39:26.369 UTC [5152] DETAIL:  Access is denied.

2020-06-05 13:39:26.369 UTC [5152] HINT:  SET pljava.libjvm_location TO the
correct path to the jvm library (libjvm.so or jvm.dll, etc.)
2020-06-05 13:39:26.374 UTC [5152] ERROR:  cannot use PL/Java before
successfully completing its setup
2020-06-05 13:39:26.374 UTC [5152] HINT:  Check the log for messages
closely preceding this one, detailing what step of setup failed and what
will be needed, probably setting one of the "pljava." configuration
variables, to complete the setup. If there is not enough help in the log,
try again with different settings for "log_min_messages" or
"log_error_verbosity".

I hope you can shed some light on this.

---

From: Chapman Flack <[email-masked]>
Date: Fri, 5 Jun 2020 10:32:14 -0400

On 06/05/20 09:44, Kartik Ohri wrote:
> ...

My attention is drawn most strongly to the "Access is denied." That isn't
a familiar error to me from non-Windows environments, so it might indicate
some Windows-specific access control policy that is preventing a file
being opened.

I would not read too much into what the HINT says; it is just a constant
string in the PL/Java source. It mentions libjvm_location because that's
a usual suspect, but it doesn't really know why the dlopen failed. The
DETAIL is the best information for that.

It might help some to say

  \set VERBOSITY verbose

(a local command in psql, from the department of redundancy department),
and

  set client_min_messages to debug2;  -- it has to be spelled right :)

or even to experiment with pg_ls_dir(), pg_stat_file(),
pg_read_binary_file() [1] with the name of the JVM DLL or the directory
it lives in, to see if PostgreSQL always gets "Access is denied".
Presumably, if you spell the name wrong on purpose, you might get a
different error for that, and then you'll know this isn't a matter
of spelling the name right.

I found a StackOverflow thread [2] about some Windows settings that
might be relevant. Most of the answers there suggest changing settings
through the GUI (file/directory properties or service properties). I
didn't see anything about how to change from a script, though I assume
with PowerShell there is probably a way.

If this turns out to be a necessary step it will be good to get the
details into the PL/Java documentation for Windows, because I'm
pretty sure there's nothing about it in there now.

One more thing about setting client_min_messages; PL/Java only looks once,
just when starting the JVM, so for that setting to affect PL/Java, it has
to come before the first statement that causes PL/Java to load,

Regards,
-Chap

[1]
https://www.postgresql.org/docs/12/functions-admin.html#FUNCTIONS-ADMIN-GENFILE

[2]
https://stackoverflow.com/questions/4267051/error-5-access-denied-when-starting-windows-service

---

From: Kartik Ohri <[email-masked]>
Date: Sat, 6 Jun 2020 18:10:26 +0530

> it lives in, to see if PostgreSQL always gets "Access is denied".

I have played around a bit using your suggestions. It seems this is indeed
a permissions issue. The good thing is I was able to find a solution to it.

> pretty sure there's nothing about it in there now.

The solution is to use *pg_ctl* to start the postgres server from a shell
with appropriate permissions. Earlier, I was trying to start the postgres
server using *net start *in CMD and *Start-Service* in PowerShell. Even
with privileged shells, postgres created using these commands do not have
the necessary rights to some directories (including the Java directory,
hence, the libjvm location error). I was unable to figure out why this
happens.

I am now using pg_ctl to start the database. This is causing some issues in
the normal mode. The build stalls right after postgres starts. This does
not happen when the server is started using the other 2 options. I am
working on fixing this. However, in the interactive debug mode, I have been
able to use pg_ctl with success. I have run all the usual tests as on other
platforms except the -Xcheck:jni and all of these work.

Once this issue is fixed we will have a fully functional MSVC Windows
build. I'll let you know once this is done. Thanks.

---

From: Chapman Flack <[email-masked]>
Date: Sat, 6 Jun 2020 10:19:05 -0400

> hence, the libjvm location error). I was unable to figure out why this

Interesting! There are a bunch of #ifdef WIN32 sections in pg_ctl.c [1],
maybe some useful info in there.

Regards,
-Chap

[1]
https://git.postgresql.org/gitweb/?p=postgresql.git;a=blob;f=src/bin/pg_ctl/pg_ctl.c;hb=HEAD

---

From: Kartik Ohri <[email-masked]>
Date: Mon, 15 Jun 2020 22:17:09 +0530

The pg_ctl stall issue has been fixed.

I am now trying to work around the Powershell streams. It seems psql writes
all its output including info, debug to error stream whereas Powershell has
a different stream for each of errors, warnings, debug and info. I have
been able to redirect all the output to stdout instead. However, this
redirects errors as well without failing the build. I am thinking of
manually processing the output to catch errors and manually terminate the
build in such a case. However, I think there might be a simpler solution to
use psql in quiet mode. What are your views on this ?

---

From: Chapman Flack <[email-masked]>
Date: Mon, 15 Jun 2020 13:23:24 -0400

> use psql in quiet mode. What are your views on this ?

I was thinking about that last night again. The current "tests" in the
examples are written to report most problems they test for as warnings
and not errors, because the first error would stop everything, and I would
rather see as many issues as possible. (Of course you still get an error
if the test itself actually breaks, which also counts as finding a problem!
You just don't find the ones after it then.)

I was looking through psql's docs again last night because I really wish
it had some kind of

\if any-warnings-happened
  \quit status=1
\endif

which seem like surprising things for it to be missing, but as far as
I can tell, it really is missing both: no way to test for warnings (it's
clearly possible, because server responses do include them), and no way
to specify an exit status to \quit. Weird.

(You might check the psql docs yourself to be sure I haven't missed
something. It seems very weird for those abilities to be lacking, but
I'm honestly not seeing them.)

In part, this is because the tests in the examples DDR aren't written
with a more conventional testing framework. What I would really like to
do is move to a combination of pgTAP in the database and tap4j in Java,
and integrate them by providing a method maybe executeQueryAsTAP(qry),
where you give it an SQL qry that produces pgTAP output, and the method
knows that and parses the results and gives you a tap4j results object.
Also the inverse, a PL/Java function that returns a tap4j object as a
TAP-formatted query result.

That's a longer-term goal; there were a couple prerequisites needing
to happen in tap4j, and I sent one and they merged it [1], and I suggested
another and didn't have time to do it myself and no one else has yet [2],
though I see they recently added a "good first issue" tag for it.

Of course assuming the tap4j prerequisites get implemented, there'd
still be the matter of rewriting all the PL/Java tests as tap. Which is
definitely what I want, but it would still be work.

So for now we're stuck with test failures as warnings, and psql having
(unless I missed something) no good way to tell you there were warnings.
That leads down the ugly road of you trying to parse log text and look
for warnings, and not make the nar-maven-plugin-like mistake of thinking
later lines of warnings are errors, or be fooled by keywords inside other
messages, and all of those well-known horrors.

Unless ... we use something besides psql to connect to the database and
run the tests. For example, make travis_setup_pljava and travis_test_pljava
into a Python script using one of Python's pgsql drivers (there are a few
choices) to connect to the server and run the SET, CREATE EXTENSION, and
SELECT commands, and notice any warnings. Or the same in another language
if you prefer it to Python.

(I'm not very familiar with Powershell but I was thinking it sounded
powerful enough it might directly support ODBC connections or something,
and sure enough, [3].)

Then it's a simple matter of using a real API to do the queries and
retrieve whatever warnings came back.

It is worth checking that the choice of script language and PostgreSQL
driver will actually support retrieving warnings! They all should,
but apparently there's a driver for Go at least [4] that doesn't. That
would be frustrating, especially to find out after doing some work.

A final issue is whether to try to do this in one common script language
across platforms. It might be natural to do something in Python on Linux
and in Powershell on Windows (though I guess you can get Python on Windows
or even Powershell on Linux, but relying on an ODBC driver binary could
add complexity).

One language guaranteed to be present on any platform for testing PL/Java
will be ... Java, and that comes with jshell starting with Java 9, so
there could be that argument for writing a jshell script and using pgjdbc.
Fewer other dependencies then, and no native binary ones.

Regards,
-Chap

[1] https://github.com/tupilabs/tap4j/pull/38
[2] https://github.com/tupilabs/tap4j/issues/39
[3]
https://stackoverflow.com/questions/9217650/connect-to-remote-postgresql-database-using-powershell
[4]
https://stackoverflow.com/questions/42070023/how-to-get-postgresql-procedure-warning-messages

---

From: Kartik Ohri <[email-masked]>
Date: Wed, 17 Jun 2020 22:45:29 +0530

> and sure enough, [3].)

Out of the box, Powershell only supports MySQL server. ODBC drivers will be
required to be built from source for use with Powershell which I think will
not be worth the complexity. JDBC driver for Java or using python may be a
viable option. I haven't had much success with the python version though.

In any case, a very simple query suppressed INFO outputs SET
client_min_messages TO WARNING. What should be the behaviour if warnings
are found ? Should be marked as failed or passing.

---

From: Kartik Ohri <[email-masked]>
Date: Thu, 18 Jun 2020 00:06:07 +0530

Also, I saw that pljava was failing on Java 15 early builds. Doing some
digging, I found Nashorn Engine is proposed to be removed from Java 15. So,
I think it might be the probable cause.

---

From: Chapman Flack <[email-masked]>
Date: Wed, 17 Jun 2020 15:25:32 -0400

> I think it might be the probable cause.

Good catch. So it's gone in 15, eh? I was hoping it might be around
a little longer.

That means it'll be time to add a profile with build-time dependencies
on org.graalvm.js:js and org.graalvm.js:js-scriptengine. That ought to
handle the uses of JS in the build itself (modulo whatever Nashorn ->
GraalJS compatibility kinks we discover).

I see no reason to also depend on the Graal compiler as described in [1];
what JS is being used for in the build doesn't require speed, so we can
keep it simple.

That will leave the problem of the snippets of JS in the self-installer
jar. I really want that jar to Just Work when run on anybody's Java runtime,
without making them download other stuff first.

Copying/shading js.jar and js-scriptengine.jar into it might work, modulo
licensing issues, but it would grow what's now about a 1 MB jar to 16 MB.
Ouch!

Probably time to work on the self-installer Java code and just build in
the work that is currently done in the included JS. That makes it less
transparent and configurable, but it hasn't needed to change much in
several years so it's probably ok.

I wonder if there is some very small, BSD-licensed javax.script engine
for some other language that would work.

Regards,
-Chap

---

From: Chapman Flack <[email-masked]>
Date: Wed, 17 Jun 2020 15:31:17 -0400

On 06/17/20 13:15, Kartik Ohri wrote:
> not be worth the complexity.

Agreed.

> JDBC driver for Java or using python may be a

I was looking at the various Python driver choices myself the other night
and it wasn't as promising as I thought. pg8000 was the pure-Python one
that I remembered, but can't access warnings (as far as I can tell).
py-postgresql looks quite promising and has message hooks, but even it
has native components, as do several others. I stopped before completing
an exhaustive survey.

There's a lot to like about a Java and JDBC approach. It shouldn't be hard
to get off the ground, JDBC has a nice standard API for finding out about
warnings (and reading through the pgJDBC code, I believe it is implemented,
though I haven't done a test yet).

Later on, as the prerequisites for migrating to TAP get accomplished,
that code could just be evolved to match.

> are found ? Should be marked as failed or passing.

My convention in the existing tests has been to use INFO to report
success, and WARNING to report failure. The idea would be to let the
test progress as far as it can, even complete if possible, but give an
overall not-passing result if there were warnings.

Regards,
-Chap

---

From: Chapman Flack <[email-masked]>
Date: Wed, 17 Jun 2020 15:33:12 -0400

> I see no reason to also depend on the Graal compiler as described in [1];

oops.

[1]
https://medium.com/graalvm/graalvms-javascript-engine-on-jdk11-with-high-performance-3e79f968a819

---

From: Kartik Ohri <[email-masked]>
Date: Thu, 18 Jun 2020 01:12:41 +0530

> that code could just be evolved to match.
As of now, setting client log level seems to work very well. Should I try
to work out this approach or go ahead?

---

From: Chapman Flack <[email-masked]>
Date: Wed, 17 Jun 2020 15:53:59 -0400

> to work out this approach or go ahead?

That sounds adequate for now. Especially if log_min_messages is still
at INFO (or could even go finer than that), and you can limit just the
client messages to warning, use that to detect success/failure, but put
the backend log into the build log.

Regards,
-Chap

---

From: Kartik Ohri <[email-masked]>
Date: Fri, 19 Jun 2020 21:24:31 +0530

On windows, do we want to have all combinations of Java and PostgreSQL with
both mingw and MSVC?

---

From: Chapman Flack <[email-masked]>
Date: Fri, 19 Jun 2020 12:16:55 -0400

> On windows, do we want to have all combinations of Java and PostgreSQL with

As an ultimate goal, I think that's desirable.

If it comes to a matter of managing time, I could see starting with
a few points in the space (say, Java 9 because it's intended to be the
earliest supported in 1.6, Java 11 because it's LTS, the latest Java
that currently works, to catch regressions), with intermediate points
filled in later.

It is good to test all targeted PG versions, at least for MSVC, because
there have been recurring issues with different PG globals lacking the
PGDLLIMPORT declaration in different versions.

That may be less important with Mingw-w64 which (I think) does not rely
on the PGDLLIMPORT declarations.

Regards,
-Chap

---

From: Kartik Ohri <[email-masked]>
Date: Sat, 20 Jun 2020 11:30:29 +0530

Hi,
I am having trouble with building pljava in mingw. I have attached the log
as well. gcc, g++, java, maven and pg_config are all present and working.
OpenSSL is also installed using pacman prior to building. I think there is
something wrong with the include paths and other system variables. Can you
take a look at the log and let me know your views about the probable cause ?

---

From: Chapman Flack <[email-masked]>
Date: Sun, 21 Jun 2020 16:08:43 -0400

Hi,

About the large attachment that just went to the list: it was held
for moderation because of the size, and we shared the attachment more
efficiently as a github issue and solved it.

But I had not yet used my moderator powers to make the held message
disappear, and it seems it got released from the mod queue anyway.
I wasn't expecting that.

Sorry for the size.

Regards,
-Chap

---

From: Chapman Flack <[email-masked]>
Date: Sun, 21 Jun 2020 18:42:05 -0400

> GraalJS compatibility kinks we discover).

I have pushed a branch chore/master/denashorn that does that exactly,
when Java 15 or later is detected. It also sets the system property
polyglot.js.nashorn-compat, which lets the existing JavaScript bits
continue to work without any changes.

> several years so it's probably ok.

I ended up refactoring it just slightly so it can be usefully subclassed,
then writing a PL/Java-specific subclass that handles the pathname mapping.
So now it's a self-extracting jar made by adding two extra .class files
rather than just one. Relatively painless.

Kartik, could you check that the branch works with Java 15 in CI?

Now that there is a small PL/Java-specific class file inside the
installer jar, I may go ahead and give it a few extra small methods
useful for testing, just so they could be run from jshell with a
classpath pointing to the jar. It would be a sensible place to put them.

Regards,
-Chap

---

From: Kartik Ohri <[email-masked]>
Date: Mon, 22 Jun 2020 09:46:21 +0530

> Kartik, could you check that the branch works with Java 15 in CI?
Hi! The branch passes the CI with both Java 14 & Java 15.

---

From: Kartik Ohri <[email-masked]>
Date: Wed, 24 Jun 2020 02:09:17 +0530

The mingw build is working but it spins off its new console which makes it
impossible to detect errors and retrieve logs. It even stalls the build. I
have been trying to figure a way to get past this but have been unable to
do so till now. Do you have any suggestions on how to fix this ?

---

From: Chapman Flack <[email-masked]>
Date: Tue, 23 Jun 2020 16:56:39 -0400

> do so till now. Do you have any suggestions on how to fix this ?

I have never experienced a Mingw-w64 build personally, so I don't have
much insight into the details. I understand that it involves starting
a shell/console within which the environment is set up for doing the build.

I've always assumed that isn't necessary for running the built artifacts.
Can the Mingw-w64-built PostgreSQL server be run as an ordinary executable?
If it can, then the test run after building should work in the usual way.

I assume that when inside the msys2 shell, the filesystem you see is still
the same filesystem, so redirecting output or error output of the commands
into a file ought to work, and the file could be read outside of msys2.

According to a MinGW mailing list thread [1], it looks possible to
redirect output and error output, either using the familiar > and 2>
or by using a 'redir' command in msys (which mustn't be confused with
a standard Microsoft "network redirector" command of the same name).

I also gather from that thread that there's an "echo %ERRORLEVEL%" to
show the exit status of a command, so that could be used after the mvn
command to get an indication of whether Maven thought it had succeeded.

If these are things you've already tried and there were issues, some more
details of what seems to go wrong where might help.

Regards,
-Chap

[1] https://sourceforge.net/p/mingw/mailman/message/16209368/

---

From: Kartik Ohri <[email-masked]>
Date: Wed, 24 Jun 2020 22:44:10 +0530

Hi! I was able to fix the mingw issue through another way. The build matrix
is here https://ci.appveyor.com/project/amCap1712/pljava/builds/33715210/.
Appveyor does not have JDK 9 and 10. I have written a script for those
versions to install them. As you can see, Java 9, PG9 and PG10 are failing.
I am investigating these. Also, it seems that msys2 only has postgres12
pre-built packages. If we want to test for other versions, we may need to
build from source. What are your thoughts on this?

---

From: Chapman Flack <[email-masked]>
Date: Wed, 24 Jun 2020 13:23:37 -0400

> build from source. What are your thoughts on this?

I think that's great news and when I get time later today I will look at
the failing Java 9, PG9, PG10 logs.

As for the msys2 coverage of PostgreSQL versions, that sounds like the
kind of gap we might eventually want to return to and fill in, but it
doesn't have to be right now. Having this much of a CI matrix is worlds
better than before, and this might be a good time to shift more (not
necessarily all, but more) attention to the remaining parts of the GSoC
project.

Regards,
-Chap

---

From: Kartik Ohri <[email-masked]>
Date: Thu, 25 Jun 2020 01:01:19 +0530

> the failing Java 9, PG9, PG10 logs.

For Java 9 it is a bit subtle, the tests don't fail but every time a JNI
refs warning is emitted. This raises a false alarm and the build fails. In
the case of PG9 and PG10 with MSVC, a test in examples.jar fails
probably. org.postgresql.pljava.example.LoggerTest
SetOfRecordTest not ok

> necessarily all, but more) attention to the remaining parts of the GSoC

 Yeah, sure. I am exploring the maven plugin and have been facing some
issues. I'll try to debug it for another day or two before asking for help.
Meanwhile, could you make the introductions with the pgJDBC maintainer of
org.postgresql endpoint at maven so that we could start work on the next
part?

---

From: Chapman Flack <[email-masked]>
Date: Wed, 24 Jun 2020 19:41:20 -0400

On 06/24/20 15:31, Kartik Ohri wrote:

> refs warning is emitted. This raises a false alarm and the build fails.

I see that. It appears to be coming from java.lang.System.initProperties,
not anywhere we can fix, so it would be reasonable to just filter that out
if possible.

> In the case of PG9 and PG10 with MSVC, a test in examples.jar fails

I was just looking at this matrix again, and I realized there might be
something basic I forgot to mention because it's kind of insider trivia:
PostgreSQL changed to a shorter version number scheme starting with 10.
Before that, they were three components and a major release was a bump
in the middle one.

So the three latest major versions have been 12, 11, and 10, and the one
before that was 9.6, and before that was 9.5, and so on.

So, I am not certain, even looking at appveyor.yml, what version of
PostgreSQL it is using in the "PG: 9" case. 9.6, 9.5, 9.4, 9.3, 9.2,
9.1, 9.0 are all different major releases. If that's building against
9.0, that's farther back than I intend PL/Java 1.6 to support. (I'm
not absolutely decided what the cutoff version will be, but 9.3 to
9.5 would be in the ballpark.)

It would not be a bad thing to get the exact versions of PostgreSQL,
Java, etc., into these build logs for easy reference when something
goes wrong. The initial banner that PL/Java outputs when log level
is DEBUG1 would do the trick.

> issues. I'll try to debug it for another day or two before asking for help.

That doesn't sound like much fun. Feel free to say something about the
issues, or check in some code on a branch where I can try building it;
I could try to get some ideas in parallel with you.

This is the 'stretch' part of the project; I haven't built a Maven plugin
either, though I may have spent more time studying existing ones. An
earlier look gives me more of a chance to sound like I know what I'm
talking about when it's time to try to help. :)

Regards,
-Chap

---

From: Chapman Flack <[email-masked]>
Date: Wed, 8 Jul 2020 17:37:10 -0400

> that I remembered, but can't access warnings (as far as I can tell).

For the record, so no future reader of this thread is misinformed, I was
wrong about that. pg8000 does make notices/warnings available, using the
'notices' property of a pg8000 Connection. It is a deque that will hold
the most recent not-more-than-100 that were received. popleft() will
retrieve them in arrival order.

Each item is a dict that presents all of the fields of the PostgreSQL
notice, keyed by their single-character codes:

https://www.postgresql.org/docs/9.6/protocol-error-fields.html

So, not only was I mistaken to say "can't access", it in fact provides
better access than many other PG drivers.

Errors are a bit botched: those raise exceptions of course, and the
exception will have a tuple of values. Those are the values of all
the fields received from PostgreSQL, just as in the warning case, but
with all the keys thrown away, leaving you to guess which field is which.

-Chap