public inbox for [email protected]help / color / mirror / Atom feed
Renovating the PL/Java build process 4+ messages / 3 participants [nested] [flat]
* Renovating the PL/Java build process @ 2020-08-31 08:12 Kartik Ohri <[email protected]> 0 siblings, 1 reply; 4+ messages in thread From: Kartik Ohri @ 2020-08-31 08:12 UTC (permalink / raw) To: [email protected] It is the time of the summer when GSoC participants look back on the work done in the past few months and so here are we. The original project idea and the project proposal had three major components. Each of these has been discussed in detail below. 1) Continuous Integration Expected project outcome: CI builds and reports occur on pull requests, for multiple versions of PostgreSQL and Java, and on Linux, macOS, and Windows. Linked Pull Request: https://github.com/tada/pljava/pull/289 State of the work: The work has been largely successful. PL/Java now has a CI pipeline built using Travis CI and AppVeyor CI. In a repository where these CI providers have the required access, every commit and PR are built and tested on over 30 combinations of PostgreSQL and Java versions across Linux, Mac OS X and Windows (MSVC and MinGW). This has already begun to bear fruit as can be seen from https://github.com/tada/pljava/issues/282 and https://github.com/tada/pljava/issues/274. Scope for future work: However, there is scope of several improvements. The CI infrastructure can be moved to Github Actions. The motive is to eliminate unnecessary access to external providers. Also, the test suite can be made more comprehensive by adding builds from PostgreSQL source on Mac OS X and Windows platform as well. 2) PL/Java API Artifact Expected project outcome: pljava-api jar artifact deployed to Maven Central when a new release is made. State of the Work: The component has not been satisfied yet. However, the mentor is satisfied with the progress since there is no new artifact ready for release yet. It is our consensus that the introduction of PL/Java PGXS Plugin and CI service providers has gone a long way in this respect. The task should be relatively easy to accomplish now. Scope for future work: Add a deploy phase to either of the CI providers to publish the pljava-api artifact. 3) Replacing nar-maven-plugin Expected project outcome: Build system no longer uses nar-maven-plugin, correctly uses build settings from pg_config, and is tested on at least Linux, macOS, and Windows (with Visual Studio and with MinGW). Linked Pull Request: https://github.com/tada/pljava/pull/288 https://github.com/tada/pljava/pull/292 (commit SHA #1df123b <https://github.com/tada/pljava/pull/292/commits/1df123b8208272b54ca6c62978fe0c7acc17a3da; ) State of the Work: The PL/Java PGXS plugin has successfully retired nar-maven-plugin. The CI builds for the project being built using the PGXS plugin are passing. The PGXS plugin also queries pg_config to obtain various flags and passes them on to the compiler and the linker. Overall, the plugin is in a functional state and works just fine. Several useful utilities are also exposed by the plugin for using in the scripts. PL/Java PGXS is a complete rework of the earlier build process in a true sense. It addresses many pain points that the nar maven plugin had introduced. The biggest issue posed by nar maven was highly noisy builds. Finding the actual errors among the tons of false positives was no easier than finding a needle in a haystack. To make matters worse, the debugging information provided by the nar plugin was average at best. For instance, many times the build would fail with an error in include paths. Ok fine, let's see why the path was incorrect. But no, you can only see an id assigned to the include path by the JVM. Good luck debugging with that. Another pressing issue is that nar tries to abstract a lot of details, hence compromising code readability. The actual arguments and configuration details passed to the compiler and linker are hidden deep inside nar maven's source code. On the other hand, the pljava-so pom.xml file had a huge block of javascript to just quote a string correctly for passing to the C code. The point is not to berate nar maven plugin. It is a great plugin, but it is just not suited for our simple use case. Enter PL/Java PGXS. PL/Java PGXS turns this completely inside out. It hides the uninteresting bland details and puts the important configuration details in the pom. PL/Java PGXS is specifically geared towards building PL/Java, hence unnecessary abstractions can be done away with. The PGXS plugin has also retired the troublesome maven-javadoc-plugin and supports building reports and scripting during the site phase. maven-javadoc-plugin tries to micromanage a lot of stuff when just running the javadoc on a set of files is enough to do the right job. The current implementation using PGXS plugin just does that by utilizing the APIs exposed by Java itself. Scope for Future Work: Since PGXS plugin supports scripting, there is no need for using maven antrun plugin for the same. The necessary code can be ported from using maven antrun plugin to use PGXS instead. Further, the exported symbols can be filtered to expose only PL/Java entry points. In addition, currently the output produced by the compiler and linker is redirected to standard output and standard error streams. It would be better if the output was rather piped through the plugin in which case it could be filtered and utilised as desired. ^ permalink raw reply [nested|flat] 4+ messages in thread
* Re: Renovating the PL/Java build process @ 2020-09-01 12:13 Chapman Flack <[email protected]> parent: Kartik Ohri <[email protected]> 0 siblings, 1 reply; 4+ messages in thread From: Chapman Flack @ 2020-09-01 12:13 UTC (permalink / raw) To: Kartik Ohri <[email protected]>; [email protected] On 08/31/20 04:12, Kartik Ohri wrote: > It is the time of the summer when GSoC participants look back on the work > done in the past few months and so here are we. The original project idea > and the project proposal had three major components. Each of these has been > discussed in detail below. > > 1) Continuous Integration > 2) PL/Java API Artifact > 3) Replacing nar-maven-plugin A hearty thank you to Kartik, whose work has provided a renovated build process that removes many sources of pain in the old one, and should also, I think, be much more approachable for any newcomer trying to see how the native code is being built. The plugins formerly used were configured 'declaratively', which can be elegant in simple cases, but once the question becomes "what do I 'declare' to trick this plugin into doing what it wasn't made for?", the simple approach here becomes easier to follow. The new process also picks up the same compiler options that were used building PostgreSQL, addressing our issue #152, and allows the -Dpljava.libjvmlocation option to be given at build time even in Windows (our issue #190) to build a package that will not need that setting configured at install time when the packaged location for Java is known. Another thank you to Thomas, who has enabled the main tada/pljava repo for the Travis-CI and AppVeyor services Kartik used, so that new pull requests on the master branch (leading toward the 1.6 release) are automatically checked. None of this work applies to the currently-released, 1.5.x branch, but the benefits will appear when the first 1.6 release happens. That is also when item #2, a pljava-api artifact deployed to Maven Central, should be expected. There is one unintended consequence of the current work. It now relies on the compiling information reported by pg_config as used to build PostgreSQL (as requested in issue #152), but that information is not recorded if PostgreSQL was built with MSVC on Windows. Therefore, when building with MSVC, we use hardcoded options, as before. A consequence is that building PL/Java with MinGW-w64 against a PostgreSQL instance built with MSVC is not now possible, because our MinGW build process relies on pg_config information that the MSVC build did not record. I would probably never have dared to try building PL/Java with MinGW when PostgreSQL was built with MSVC, but it happened by accident for a while as we worked on CI this summer, and it did in fact work. So, perhaps, someone may be relying on it. If so, it will be simple to add back into the new process; it would just be another set of rules for Windows-MinGW-PGMSVC, and would have to hardcode the compiling options as the MSVC rules do. Again, thank you to Kartik. It has been a pleasure collaborating on this work. Regards, -Chap ^ permalink raw reply [nested|flat] 4+ messages in thread
* Re: Renovating the PL/Java build process @ 2020-09-01 16:07 Chapman Flack <[email protected]> parent: Chapman Flack <[email protected]> 0 siblings, 1 reply; 4+ messages in thread From: Chapman Flack @ 2020-09-01 16:07 UTC (permalink / raw) To: [email protected] On 09/01/20 08:13, Chapman Flack wrote: > A hearty thank you to Kartik, whose work has provided a renovated build > > Another thank you to Thomas, who has enabled the main tada/pljava repo And lest I forget, thanks of course to Google Summer of Code for supporting Kartik's work on this project. Regards, -Chap ^ permalink raw reply [nested|flat] 4+ messages in thread
* Re: Renovating the PL/Java build process @ 2020-09-01 20:28 Thomas Hallgren <[email protected]> parent: Chapman Flack <[email protected]> 0 siblings, 0 replies; 4+ messages in thread From: Thomas Hallgren @ 2020-09-01 20:28 UTC (permalink / raw) To: Chapman Flack <[email protected]>; +Cc: [email protected] And I'd like to give a heartly thanks to you Chap, for all the hard work you do managing this project. A project that I once started but then more or less abandoned several years ago due to lack of time. You've managed to drive this forward far better than I could ever have hoped to do. A truly huge effort and excellent work throughout. I couldn't have wished for a better successor to manage the project. Regards, Thomas On Tue, 1 Sep 2020 at 18:07, Chapman Flack <[email protected]> wrote: > On 09/01/20 08:13, Chapman Flack wrote: > > A hearty thank you to Kartik, whose work has provided a renovated build > > > > Another thank you to Thomas, who has enabled the main tada/pljava repo > > And lest I forget, thanks of course to Google Summer of Code for supporting > Kartik's work on this project. > > Regards, > -Chap > > > ^ permalink raw reply [nested|flat] 4+ messages in thread
end of thread, other threads:[~2020-09-01 20:28 UTC | newest] Thread overview: 4+ messages (download: mbox mbox.gz follow: Atom feed) -- links below jump to the message on this page -- 2020-08-31 08:12 Renovating the PL/Java build process Kartik Ohri <[email protected]> 2020-09-01 12:13 ` Chapman Flack <[email protected]> 2020-09-01 16:07 ` Chapman Flack <[email protected]> 2020-09-01 20:28 ` Thomas Hallgren <[email protected]>
This inbox is served by agora; see mirroring instructions for how to clone and mirror all data and code used for this inbox