Subject: [Pljava-dev] The pljava Project is dead (LONG but I hope interesting)

From: chap at anastigmatix.net (Chapman Flack)
Date: Sun, 19 Jul 2015 16:11:59 -0400

On 12/02/15 18:08, Jos? Carlos Mart?nez Llario wrote:
> Im very sorry to hear in other posts from potsgres community that
> pljava is almost dead.
> Is this true? hope not but im feeling like is already dead.
> Hope someone says the opposite.

I certainly hope it is not dead ... it's working too well for me, and
I've only recently regained enough time to contribute to it some more
myself.

Maybe it'll help to give some of my impressions of the condition of
the project, as a relative newcomer who wasn't around for the early
development, but successfully uses it and contributes to it.

MY CURRENT IMPRESSIONS:

I started with PL/Java just before the switch to maven-based building in
2013. I found the old build a bit fiddly, and then I grumbled a little
about having to learn this maven thing, but I have to say now that the
maven setup easily builds me a working pljava every time and seems quite
solid. Currently my platform is PG9.3 and Linux ... maybe other
platforms have build issues that could be improved.

Also, once built and installed in postgres, pljava seems to Just Work
and do what it says on the tin. For a new user, the process of getting
it set up in postgres, the first time, is more fiddly than it could be,
and that's where some valuable improvements could be made.

So, from where I stand, here's what I see:

 * It's usable now, for many realistic purposes

 * It has areas where valuable improvement could still be made
   (especially "around the edges", simplifying first-time installation,
   etc.)

 * The improvable bits tend to be small/meaty enough for one person, or
   just a few, to tackle and add noticeable value; that's kind of nice.

Thomas Hallgren seems to not currently have a lot of time to pound
on it himself, but he is clearly still paying attention, and responds
promptly when I make pull requests, usually in under a day.  I am not
sure how many other people have commit access.

THINGS HOLDING IT BACK:

Now, for the things I see holding it back, or being discouraging
to newcomers. You'll see how many I think could be easily addressed:

 * The wiki is way overdue for some love. For example, the maven-based
   build is working great as far as I can tell, and the makefiles are
   obsolete, but the wiki 'building' page doesn't even say maven yet
   (issue 18). I've been making code pull requests lately, but I can
   try to fit in some wiki attention too, if I can get edit permissions.

 * The javadocs aren't viewable on the github site. There is plenty of
   good javadoc in the code, and I've just messed around a little with
   the site-generating plugin for maven, and like magic produced a web
   site image with the javadocs included. I think it would be great to
   push that to the project's github-pages space, which is currently
   unused. (I can see that Thomas was experimenting with it in early
   2013, but that was just the same content that's now on the wiki.
   I think it could be quite helpful to keep the user-oriented texts
   on the wiki where they are now, but allow them to link to the
   detailed technical docs on github-pages.)

 * The issue tracker doesn't have a huge number of issues, but some are
   rather old. Also they are a mix of true distinct issues, and some
   duplicates or relateds, and some that should just have been list
   questions. Even just a small increase in attention to reviewing/
   categorizing/triaging issues when filed would probably go a long
   way to making the issue tracker seem less stagnant. I've been doing
   my part this week to try and resolve a few of 'em.

 * Issue 11, about regression testing, is super important, because a
   lack of good regression testing can have a chilling effect on
   participation. It is scary to dive in and start banging on code when
   the tests are sparse and fiddly to run. I agree with Thomas that it
   would certainly be a start to take the existing "examples" and
   scaffold them as tests, and I think I've seen an easy way to do it,
   that I'll work on. Once there's better testing, the project can be
   much more inviting to work on, because the fear-of-hosing-something
   factor is less.

TIME TO REVISIT BACKWARD COMPATIBILITY TARGETS ?

I think it would help for the project to publish an updated cutoff
for the oldest versions of PostgreSQL and Java that it actively means
to support. The latest I can find are in the old readme.html still
viewable in docs/ that says PostgreSQL >= 8.0.3 and Java >= 1.4.
According to 'git blame', both those lines were last touched
in 2005. :) The intro.html in the same directory says "primarily
targeted to the new 8.1 version", again as of 2005.

I know there's a cost to advancing the cutoff, in case there are
many PL/Java users still nursing along 8.0.3 databases, but there's
also a disincentive for interested contributors if we don't, because
it's really hard to develop new code with a hope of it also working
in ten-year-old EOLed environments, and it limits what can be aspired to.

As regards Java, we already have a component (the annotation-driven
SQL generator) that requires 1.6 or newer. 1.5 is pretty old now, and
before that, there were no generics. 1.7 brings try-with-resources,
which can make JDBC coding much less tedious.

In PostgreSQL:

8.1 added SET ROLE; PL/Java still only understands the three IDs
a session had before 8.1 (originally-authenticated ID, session ID,
and the effective one during a security-definer function), not
the four IDs there have been since 8.1.  That leads to strangeness
like:

You are connected as user "bob".
=> SET ROLE schemaowner;
SET ROLE
=> CREATE TABLE foostuff.test;
CREATE TABLE
=> SELECT sqlj.install_jar('foo.jar', 'foo', true);
ERROR:  permission denied for schema foostuff

... because PL/Java's install_jar (quite properly) dropped its own
privilege before running the deployment descriptor, but it dropped
back to "bob" instead of "schemaowner".

If we advance the backward-compatibility bar at least as far as 8.1,
we can fix that without having to special-case it.

8.2 added the RETURNING clause on DML statements. That is handy
because PL/Java's built-in JDBC implementation hasn't ever had
the special methods that retrieve generated IDs from a DML statement,
but as long as the backend groks RETURNING there's a perfectly usable
alternative (arguably even easier to use and to read).

8.4 added WITH, great for writing readable testcases!

WITH
 ref AS (VALUES ('adjective', 'avaricious'), ('noun', 'platypus')),
 test AS (SELECT * FROM javatest.propertyExampleAnno()),
 missing AS (TABLE ref EXCEPT TABLE test)
 SELECT 0 = count(*) FROM missing;

8.4 is also the latest to go EOL (about a year ago).

9.1 brought the extension-packaging system, which could offer a
target for PL/Java's various deployment methods to converge on.
One thing it can do is automatically select and run schema-updating
scripts, which is an outstanding PL/Java issue (#12).

9.1 goes EOL just over a year from now:
http://www.postgresql.org/support/versioning/

So, by biting the bullet and declaring some reasonable newer cutoffs
for back-compatibility, the project can become more contributor-
friendly.

STREAMLINING INSTALLATION/DEPLOYMENT

As PL/Java has been getting built, there is one part of installation
that doesn't quite fit with the others. *Almost* everything about
PL/Java gets configured using PostgreSQL's own mechanisms (in
postgresql.conf for dynamic_library_path and pljava.classpath, with
sqlj.set_classpath for user classes) but the location of libjvm itself
has not been. The native library has been built with a reference to
libjvm baked in, which the OS tries to find before PostgreSQL settings
can have any effect, and when it fails you have to pick one of the
solutions from the wiki that are all system-dependent and outside of
PostgreSQL (from the "installing on linux" page, edit LD_LIBRARY_PATH
in a PostgreSQL startup script, or tweak /etc/ld.conf.so, or, from
the "building pljava" page, figure out the right link options to build
the native library with an RPATH). The RPATH one is my favorite, but
only because it's the only one that isn't tweaking some shared system
state just to make one thing work. But it's still a pain, especially
rebuilding with a new RPATH if the JRE location changes.

So I've started a branch where you also set the libjvm location in
postgresql.conf like anything else, using pljava.libjvm_location.

https://github.com/jcflack/pljava/tree/feature/master/installhelp

I haven't made it a pull request yet because it's the sort of change
that might well be tested on a few other platforms besides mine.
I don't expect many portability problems with the dynamic loading,
because I used PostgreSQL's own pg_dlopen wrapper, which the main
project has already done the heavy lifting of getting to work on
at least nine different platforms. Any platform where pg_dlopen won't
work is one where PL/Java won't work anyway. I simply changed the maven
project to not bake the libjvm reference into the pljava object,
and pljava's native code can now get control and load libjvm using
the configured pljava.libjvm_location. If it fails, it can even give
a useful error with a hint to SET pljava.libjvm_location.

That's where it stands now but it doesn't have to stop there. With an
assign hook on the variable, it could even pick up where it left off
after a SET pljava.libjvm_location command. It could do the same with
pljava.classpath if the jar isn't found, and then even complete the SQL
installation steps if the objects aren't there yet. It could then
log those working settings and remind you to add them into
postgresql.conf, or (in 9.4+) even use ALTER SYSTEM to do that directly.

As a result, it could be possible to quickly try out PL/Java without
editing any configs at all: just start with

  LOAD /path/to/pljava.so;

and then issue SET pljava.libjvm_location and SET pljava.classpath
commands as the messages suggest, and watch it go.

Anyway, those are my current ideas on the state of PL/Java and the
parts of it I am working on.  I'm sorry it came out so long, but
I wanted to get several of the ideas out to the list for comment
(at least from those of you who'll read something this long. :)
For those who won't maybe I'll do some later messages taking up
smaller pieces individually.

Cheers,
-Chap

---

From: craig at 2ndquadrant.com (Craig Ringer)
Date: Mon, 20 Jul 2015 08:13:05 +0800

> for back-compatibility, the project can become more contributor-

Cutting heavily to focus on an extension-only version with
9.1-and-newer compatibility on JDK 6 (7?) seems sensible to me.

The PgJDBC project doesn't maintain compatibility anywhere near as far
back as PL/Java, and we're about to advance the BC stops even further.
People who want to use old databases can use old versions of the
driver.

The build system could also use some love.

-- 
 Craig Ringer                   http://www.2ndQuadrant.com/
 PostgreSQL Development, 24x7 Support, Training & Services

---

From: bgiles at coyotesong.com (Bear Giles)
Date: Sun, 19 Jul 2015 18:42:33 -0600

It's a big jump in scope but don't some databases have a native type for
java classes and a way to execute them? (It's not hard to imagine a UDT for
jar files as well.) It's not always possible to load jar files from the
operating system.

On Sun, Jul 19, 2015 at 6:13 PM, Craig Ringer <craig at 2ndquadrant.com> wrote:

> _______________________________________________
> Pljava-dev mailing list
> Pljava-dev at lists.pgfoundry.org
> http://lists.pgfoundry.org/mailman/listinfo/pljava-dev
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.pgfoundry.org/pipermail/pljava-dev/attachments/20150719/46f7d551/attachment.html>

---

From: craig at 2ndquadrant.com (Craig Ringer)
Date: Mon, 20 Jul 2015 10:10:02 +0800

> jar files as well.) It's not always possible to load jar files from the

Personally that fills me with horror, but each to their own.

You should be able to do this with PL/Java already if you really want
to. Just implement a custom classloader.

Anyway, that's a totally separate topic to how to make Pl/Java easier
to maintain, more attractive to use/work on, and more lively looking.

Personally I think the underlying impedence mismatch between
PostgreSQL shared-nothing-by-default multiprocessing architecture and
Java's shared-everything-by-default multithreading model limits the
appeal and utility of PL/Java as much as anything. Especially since
the JVM doesn't like being fork()ed and is designed around a
heavyweight startup long-running process approach. That said, Pl/Java
could expose the new dynamic shared memory features in PostgreSQL to
applications, which would permit apps to pass serialized objects
around via shared memory, and that'd meet most inter-process
communication needs.

---

From: chap at anastigmatix.net (Chapman Flack)
Date: Sat, 25 Jul 2015 12:15:36 -0400

I just noticed how this thread shows up in the archives, where the
repetition of "PL/Java is dead" gives a bit of the wrong impression. :)

> 9.1-and-newer compatibility on JDK 6 (7?) seems sensible to me.

I've glanced at how the extension machinery is implemented, and
it looks easy to make a streamlined installation that works
pre-extensions and also integrates with the extension system where
it's available.

It seems CREATE EXTENSION essentially sets a global flag that an
extension's being created, and then runs whatever script is supplied
to install the stuff. The registration of all the components as parts
of the extension happens automagically through a multitude of
event triggers that have been set up on every command that creates or
alters anything. So it doesn't look as if supporting extension-style
installation would require any great jump away from pre-extension-style
installation.

-Chap