Subject: Renovating the PL/Java build process

From: Kartik Ohri <[email-masked]>
Date: Mon, 31 Aug 2020 13:42:30 +0530

It is the time of the summer when GSoC participants look back on the work
done in the past few months and so here are we. The original project idea
and the project proposal had three major components. Each of these has been
discussed in detail below.

1) Continuous Integration

Expected project outcome: CI builds and reports occur on pull requests, for
multiple versions of PostgreSQL and Java, and on Linux, macOS, and Windows.

Linked Pull Request: https://github.com/tada/pljava/pull/289

State of the work: The work has been largely successful. PL/Java now has a
CI pipeline built using Travis CI and AppVeyor CI. In a repository where
these CI providers have the required access, every commit and PR are built
and tested on over 30 combinations of PostgreSQL and Java versions across
Linux, Mac OS X and Windows (MSVC and MinGW). This has already begun to
bear fruit as can be seen from https://github.com/tada/pljava/issues/282
 and https://github.com/tada/pljava/issues/274.

Scope for future work: However, there is scope of several improvements. The
CI infrastructure can be moved to Github Actions. The motive is to
eliminate unnecessary access to external providers. Also, the test suite
can be made more comprehensive by adding builds from PostgreSQL source on
Mac OS X and Windows platform as well.

2) PL/Java API Artifact

Expected project outcome: pljava-api jar artifact deployed to Maven Central
when a new release is made.

State of the Work: The component has not been satisfied yet. However, the
mentor is satisfied with the progress since there is no new artifact ready
for release yet. It is our consensus that the introduction of PL/Java PGXS
Plugin and CI service providers has gone a long way in this respect. The
task should be relatively easy to accomplish now.

Scope for future work: Add a deploy phase to either of the CI providers to
publish the pljava-api artifact.

3) Replacing nar-maven-plugin

Expected project outcome: Build system no longer uses nar-maven-plugin,
correctly uses build settings from pg_config, and is tested on at least
Linux, macOS, and Windows (with Visual Studio and with MinGW).

Linked Pull Request:

https://github.com/tada/pljava/pull/288

https://github.com/tada/pljava/pull/292 (commit SHA #1df123b
<https://github.com/tada/pljava/pull/292/commits/1df123b8208272b54ca6c62978fe0c7acc17a3da>
)

State of the Work: The PL/Java PGXS plugin has successfully retired
nar-maven-plugin.
The CI builds for the project being built using the PGXS plugin are
passing. The PGXS plugin also queries pg_config to obtain various flags and
passes them on to the compiler and the linker. Overall, the plugin is in a
functional state and works just fine. Several useful utilities are also
exposed by the plugin for using in the scripts.

PL/Java PGXS is a complete rework of the earlier build process in a true
sense. It addresses many pain points that the nar maven plugin had
introduced. The biggest issue posed by nar maven was highly noisy builds.
Finding the actual errors among the tons of false positives was no easier
than finding a needle in a haystack. To make matters worse, the debugging
information provided by the nar plugin was average at best. For instance,
many times the build would fail with an error in include paths. Ok fine, let's
see why the path was incorrect. But no, you can only see an id assigned to
the include path by the JVM. Good luck debugging with that.

Another pressing issue is that nar tries to abstract a lot of details,
hence compromising code readability. The actual arguments and configuration
details passed to the compiler and linker are hidden deep inside nar maven's
source code. On the other hand, the pljava-so pom.xml file had a huge block
of javascript to just quote a string correctly for passing to the C code. The
point is not to berate nar maven plugin. It is a great plugin, but it is
just not suited for our simple use case.

Enter PL/Java PGXS. PL/Java PGXS turns this completely inside out. It hides
the uninteresting bland details and puts the important configuration
details in the pom. PL/Java PGXS is specifically geared towards building
PL/Java, hence unnecessary abstractions can be done away with.

The PGXS plugin has also retired the troublesome maven-javadoc-plugin and
supports building reports and scripting during the site phase.
maven-javadoc-plugin
tries to micromanage a lot of stuff when just running the javadoc on a set
of files is enough to do the right job. The current implementation using
PGXS plugin just does that by utilizing the APIs exposed by Java itself.

Scope for Future Work: Since PGXS plugin supports scripting, there is no
need for using maven antrun plugin for the same. The necessary code can be
ported from using maven antrun plugin to use PGXS instead. Further, the
exported symbols can be filtered to expose only PL/Java entry points. In
addition, currently the output produced by the compiler and linker is
redirected to standard output and standard error streams. It would be
better if the output was rather piped through the plugin in which case it
could be filtered and utilised as desired.

---

From: Chapman Flack <[email-masked]>
Date: Tue, 1 Sep 2020 08:13:03 -0400

> 3) Replacing nar-maven-plugin

A hearty thank you to Kartik, whose work has provided a renovated build
process that removes many sources of pain in the old one, and should also,
I think, be much more approachable for any newcomer trying to see how
the native code is being built. The plugins formerly used were configured
'declaratively', which can be elegant in simple cases, but once the question
becomes "what do I 'declare' to trick this plugin into doing what it wasn't
made for?", the simple approach here becomes easier to follow.

The new process also picks up the same compiler options that were used
building PostgreSQL, addressing our issue #152, and allows the
-Dpljava.libjvmlocation option to be given at build time even in Windows
(our issue #190) to build a package that will not need that setting
configured at install time when the packaged location for Java is known.

Another thank you to Thomas, who has enabled the main tada/pljava repo
for the Travis-CI and AppVeyor services Kartik used, so that new pull
requests on the master branch (leading toward the 1.6 release) are
automatically checked.

None of this work applies to the currently-released, 1.5.x branch, but
the benefits will appear when the first 1.6 release happens. That is
also when item #2, a pljava-api artifact deployed to Maven Central,
should be expected.

There is one unintended consequence of the current work. It now relies
on the compiling information reported by pg_config as used to build
PostgreSQL (as requested in issue #152), but that information is not
recorded if PostgreSQL was built with MSVC on Windows. Therefore, when
building with MSVC, we use hardcoded options, as before.

A consequence is that building PL/Java with MinGW-w64 against a
PostgreSQL instance built with MSVC is not now possible, because our
MinGW build process relies on pg_config information that the MSVC build
did not record.

I would probably never have dared to try building PL/Java with MinGW
when PostgreSQL was built with MSVC, but it happened by accident for
a while as we worked on CI this summer, and it did in fact work. So,
perhaps, someone may be relying on it.

If so, it will be simple to add back into the new process; it would
just be another set of rules for Windows-MinGW-PGMSVC, and would have
to hardcode the compiling options as the MSVC rules do.

Again, thank you to Kartik. It has been a pleasure collaborating on
this work.

Regards,
-Chap

---

From: Chapman Flack <[email-masked]>
Date: Tue, 1 Sep 2020 12:07:15 -0400

> Another thank you to Thomas, who has enabled the main tada/pljava repo

And lest I forget, thanks of course to Google Summer of Code for supporting
Kartik's work on this project.

Regards,
-Chap

---

From: Thomas Hallgren <[email-masked]>
Date: Tue, 1 Sep 2020 22:28:07 +0200

And I'd like to give a heartly thanks to you Chap, for all the hard work
you do managing this project. A project that I once started but then more
or less abandoned several years ago due to lack of time. You've managed to
drive this forward far better than I could ever have hoped to do. A truly
huge effort and excellent work throughout. I couldn't have wished for a
better successor to manage the project.

Regards,
Thomas