Subject: Re: Starting build-system work (Windows/Mac users please speak up)
To: Kartik Ohri <kartikohri13@gmail.com>
References: <5EC17E17.7070002@anastigmatix.net>
 <CAASLQ4N0waxE9JKXjcLD=oAYz8NdgVAjBQd7FWGkuCOghFy8ng@mail.gmail.com>
 <5EC66741.4030208@anastigmatix.net>
 <CAASLQ4M4rMaL3gi23+jA-sfLU5DbhN+PTr9Jw3YxmTV_0juvbg@mail.gmail.com>
 <5EC6FB8B.7080806@anastigmatix.net>
 <CAASLQ4OB0SK1bLV=N8Sxuz2dDphkYjVA2007q351m1y8mn15Dg@mail.gmail.com>
Cc: pljava-dev@lists.postgresql.org
From: Chapman Flack <chap@anastigmatix.net>
Message-ID: <5EC82C8F.80606@anastigmatix.net>
Date: Fri, 22 May 2020 15:48:31 -0400
User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:38.0) Gecko/20100101
 Thunderbird/38.7.0
MIME-Version: 1.0
In-Reply-To: 
 <CAASLQ4OB0SK1bLV=N8Sxuz2dDphkYjVA2007q351m1y8mn15Dg@mail.gmail.com>
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
Precedence: bulk

On 05/22/20 14:24, Kartik Ohri wrote:

>  #77.8 and #77.17 illustrate this issue. In #77.8, macports is unable to
> install openssl which is installed properly in all of the remaining builds.
> #77.17 fails with some include path error due to nar-maven.

I have seen that in reported issues too. Inconvenient that the
nar-maven-plugin developers wrote an IncludePath object that doesn't
override toString(), so when it is printed in an error message it
just comes out IncludePath@hex-address.

Don't do that in your plugin. :)

I see Maven gives a hint there that if you run with -e it might
give a full stack trace of the errors. I haven't tried it, but
in a case like this I wonder if the full stack trace would include
a lower-level exception like FileNotFoundException that would
actually show the name that wasn't found!

My suspicion is that it also is related to some dependency like
openssl that intermittently isn't getting downloaded/installed properly.


Another thing I notice that could be annoying when going through
these logs is that the vast majority of the log is just download
progress of all the dependencies for the build.

It looks like that bulk can be reduced a lot by adding these Maven options:

--batch-mode
-Dorg.slf4j.simpleLogger.log.org.apache.maven.cli.transfer.Slf4jMavenTransferListener=warn

The first one just tones down some of the message bloat like color changes,
and setting the TransferListener logging level to 'warn' means we should see
if something goes wrong, but not fill the whole log with all the stuff
that went right.

Found here: https://stackoverflow.com/a/35653426


As another thing to think about, if every build is starting from a
clean slate, all of that stuff has to get re-downloaded every time.
If I start from a clean slate, build PL/Java once, then look at the
size of my local Maven repository:

$ du -sh ~/.m2/repository
72M	/var/tmp/testbuilds/.m2/repository

that's 72 megs of stuff getting downloaded ... each time.

If we had a way of doing multiple PG version / multiple Java version
test builds without completely resetting the environment every time,
that could save the Travis guys more than half a gig of bandwidth.

For every set of test runs.

What I do here is I have several different PG versions built and
installed in testbuilds/pg12 testbuilds/pg11 and so on, and likewise
for Java versions.

Then if I run Maven like so:

JAVA_HOME=/var/tmp/jdk-14+36/ mvn \
  -Dpljava.libjvmdefault=/var/tmp/jdk-14+36/lib/server/libjvm.so \
  -Dpgsql.pgconfig=/var/tmp/testbuilds/pg12/bin/pg_config \
  clean install

it builds for that Java version and that PG version:

- the JAVA_HOME in front controls what Java version will run Maven itself
- the -Dpljava.libjvmdefault builds in a default value of
  pljava.libjvm_location so it doesn't need to be specified later in PG
- the -Dpgsql.pgconfig controls which PG version to build against.


Then when it has created the pljava-packaging/target/pl*jar file, I can
run it like this:

/var/tmp/jdk-14+36/bin/java \
  -Dpgconfig=/var/tmp/testbuilds/pg12/bin/pg_config \
  -jar pljava-packaging/target/pl*jar

which makes sure to use the tested Java version to run the jar, and
again gives the pg_config location, which determines where the files
are installed.

This doesn't all have to be in place at once, if you are focused on
getting the basic functionality in place. But it would be a significant
optimization later.

Also, I suggest (as in the build docs) using these Maven options:

 -Dnar.cores=1 -Pwnosign

The -Pwnosign only works on platforms using gcc, so it would have to be
left off for others. It silences a whole category of sign-conversion
warnings that are only distracting.

Setting nar.cores to 1 makes it a lot easier to read error messages if
there are errors; otherwise, if the compilation uses a bunch of threads,
the messages all come out on top of each other and it is hard to see
where an error came from.

Also -Psaxon-examples is needed to build a set of XML examples that are
not built by default.

Regards,
-Chap