Received: from malur.postgresql.org ([217.196.149.56]) by arkaria.postgresql.org with esmtps (TLS1.3:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.92) (envelope-from ) id 1jcDuD-0006EJ-1M for pljava-dev@arkaria.postgresql.org; Fri, 22 May 2020 20:04:33 +0000 Received: from localhost ([127.0.0.1] helo=malur.postgresql.org) by malur.postgresql.org with esmtp (Exim 4.92) (envelope-from ) id 1jcDu9-0002C5-U5 for pljava-dev@arkaria.postgresql.org; Fri, 22 May 2020 20:04:29 +0000 Received: from makus.postgresql.org ([2001:4800:3e1:1::229]) by malur.postgresql.org with esmtps (TLS1.3:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.92) (envelope-from ) id 1jcDu9-0002By-Hs for pljava-dev@lists.postgresql.org; Fri, 22 May 2020 20:04:29 +0000 Received: from mail-ej1-x62b.google.com ([2a00:1450:4864:20::62b]) by makus.postgresql.org with esmtps (TLS1.3:ECDHE_RSA_AES_128_GCM_SHA256:128) (Exim 4.92) (envelope-from ) id 1jcDu2-0007m7-Kr for pljava-dev@lists.postgresql.org; Fri, 22 May 2020 20:04:28 +0000 Received: by mail-ej1-x62b.google.com with SMTP id se13so14352855ejb.9 for ; Fri, 22 May 2020 13:04:22 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=mime-version:references:in-reply-to:from:date:message-id:subject:to :cc; bh=IyhXSbtAMZRvz5onzdKVLV/74DWr2II9jYRIEH5opgw=; b=iJpC3njuPQsRH7bR6KyV4/tU0AltYarekOrXlYmWc7ov13dE8zOpNgG/JYhYlP0kuO nWByp5IDBxm2JrPz1Q7rRXadHFXyk6lja2zIowv3sgNxFWFHC+l77oPQbAzwO69vxpA5 YATdr8N/cq/ONHM92BacF9OoQon7bdu9atn57fzHf5gH/mriInf4K67UBUShicElF20R VXfiixBOBftwNgpmL3XKLOpo1iumSihwFK0TxIka4fJdeHxWBtk/QEdaL0g9jQU+5sul C7ExBGt53+Zo3rsEc2AfMUEKR/IwhT9pWYN2mcbrEzPI7gdAbKqT4iXjQmJZlBuSVh0c ZvZQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:mime-version:references:in-reply-to:from:date :message-id:subject:to:cc; bh=IyhXSbtAMZRvz5onzdKVLV/74DWr2II9jYRIEH5opgw=; b=A+yQ8UcXcTwqJ3kRP7hOFPRyVFJb52K2M63F8GOA3lQXWQ1pTZDEbiCfqbr9YRS864 3J2hbs1VjQmozDUrQsYvkDGfUO6CVVyU8dsghwRv7NWmKuqC66tV0CjKZ7y8bjpi98Hp 2gSqAQO8KR+6KEIcNHVAx3yJ41zhdLVa3xH/2OnSY7GI4ArN7Ol3TryJt+fwa03En+TL j4plribz0TC07uiAT0eYf6XVGXUEaULgwAIgHBEKk3NUZaPdrPzs4SsxKLyS0maHgByU p9vs9ZNybSrYmo9YSoeydCazXdV9wOXbnKIvAc7Zeq77Wld/uTMCMYGNnEzE79EcilOp rFMQ== X-Gm-Message-State: AOAM533ufzYuPLDyLzZ3d8/6cjzphR8XbKCrUUuEQ1tbGqUbUpU6nPt7 8Dc/pxovtT5Rt4msVkDfDJyaSD4sWRr041lK7yR290yBEc0= X-Google-Smtp-Source: ABdhPJyZJids/UHfgd1mFerkDIjB9SEUFqftJcSQ8uzzA0nHqR108vFeuqCYywnqnOwNFBk5EGAfNfXKJmnZ6bEGoeQ= X-Received: by 2002:a17:906:9709:: with SMTP id k9mr9592771ejx.48.1590177859363; Fri, 22 May 2020 13:04:19 -0700 (PDT) MIME-Version: 1.0 References: <5EC17E17.7070002@anastigmatix.net> <5EC66741.4030208@anastigmatix.net> <5EC6FB8B.7080806@anastigmatix.net> <5EC82C8F.80606@anastigmatix.net> In-Reply-To: <5EC82C8F.80606@anastigmatix.net> From: Kartik Ohri Date: Sat, 23 May 2020 01:34:08 +0530 Message-ID: Subject: Re: Starting build-system work (Windows/Mac users please speak up) To: Chapman Flack Cc: pljava-dev@lists.postgresql.org Content-Type: multipart/alternative; boundary="0000000000006fe9c605a6422203" List-Id: List-Help: List-Subscribe: List-Post: List-Owner: List-Archive: Precedence: bulk --0000000000006fe9c605a6422203 Content-Type: text/plain; charset="UTF-8" On Sat, May 23, 2020 at 1:18 AM Chapman Flack wrote: > On 05/22/20 14:24, Kartik Ohri wrote: > > > #77.8 and #77.17 illustrate this issue. In #77.8, macports is unable to > > install openssl which is installed properly in all of the remaining > builds. > > #77.17 fails with some include path error due to nar-maven. > > I have seen that in reported issues too. Inconvenient that the > nar-maven-plugin developers wrote an IncludePath object that doesn't > override toString(), so when it is printed in an error message it > just comes out IncludePath@hex-address. > > Don't do that in your plugin. :) :+1: > > I see Maven gives a hint there that if you run with -e it might > give a full stack trace of the errors. I haven't tried it, but > in a case like this I wonder if the full stack trace would include > a lower-level exception like FileNotFoundException that would > actually show the name that wasn't found! > > My suspicion is that it also is related to some dependency like > openssl that intermittently isn't getting downloaded/installed properly. > > > > Another thing I notice that could be annoying when going through > these logs is that the vast majority of the log is just download > progress of all the dependencies for the build. > > It looks like that bulk can be reduced a lot by adding these Maven options: > > --batch-mode > > -Dorg.slf4j.simpleLogger.log.org.apache.maven.cli.transfer.Slf4jMavenTransferListener=warn > > The first one just tones down some of the message bloat like color changes, > and setting the TransferListener logging level to 'warn' means we should > see > if something goes wrong, but not fill the whole log with all the stuff > that went right. > > Found here: https://stackoverflow.com/a/35653426 I will make these changes and try to reduce any other unnecessary logs as well. > As another thing to think about, if every build is starting from a > clean slate, all of that stuff has to get re-downloaded every time. > If I start from a clean slate, build PL/Java once, then look at the > size of my local Maven repository: > > $ du -sh ~/.m2/repository > 72M /var/tmp/testbuilds/.m2/repository > > that's 72 megs of stuff getting downloaded ... each time. > > If we had a way of doing multiple PG version / multiple Java version > test builds without completely resetting the environment every time, > that could save the Travis guys more than half a gig of bandwidth. > > For every set of test runs. > > What I do here is I have several different PG versions built and > installed in testbuilds/pg12 testbuilds/pg11 and so on, and likewise > for Java versions. > > Then if I run Maven like so: > > JAVA_HOME=/var/tmp/jdk-14+36/ mvn \ > -Dpljava.libjvmdefault=/var/tmp/jdk-14+36/lib/server/libjvm.so \ > -Dpgsql.pgconfig=/var/tmp/testbuilds/pg12/bin/pg_config \ > clean install > > it builds for that Java version and that PG version: > > - the JAVA_HOME in front controls what Java version will run Maven itself > - the -Dpljava.libjvmdefault builds in a default value of > pljava.libjvm_location so it doesn't need to be specified later in PG > - the -Dpgsql.pgconfig controls which PG version to build against. > > > Then when it has created the pljava-packaging/target/pl*jar file, I can > run it like this: > > /var/tmp/jdk-14+36/bin/java \ > -Dpgconfig=/var/tmp/testbuilds/pg12/bin/pg_config \ > -jar pljava-packaging/target/pl*jar > > which makes sure to use the tested Java version to run the jar, and > again gives the pg_config location, which determines where the files > are installed. > > This doesn't all have to be in place at once, if you are focused on > getting the basic functionality in place. But it would be a significant > optimization later. > This would be worthwhile but as mentioned at https://docs.travis-ci.com/user/build-stages/#data-persistence-between-stages-and-jobs , state is not persisted between jobs. Every build job compulsorily starts as a clean state. We can though cache dependencies to speed up the build and save bandwidth though to some extent using https://docs.travis-ci.com/user/caching/ . > Also, I suggest (as in the build docs) using these Maven options: > > -Dnar.cores=1 -Pwnosign > > The -Pwnosign only works on platforms using gcc, so it would have to be > left off for others. It silences a whole category of sign-conversion > warnings that are only distracting. > > Setting nar.cores to 1 makes it a lot easier to read error messages if > there are errors; otherwise, if the compilation uses a bunch of threads, > the messages all come out on top of each other and it is hard to see > where an error came from. > > Also -Psaxon-examples is needed to build a set of XML examples that are > not built by default. > > I will incorporate these changes into the builds. > Regards, > -Chap > Thanks for the detailed feedback. I will work on this and get back to you soon. --0000000000006fe9c605a6422203 Content-Type: text/html; charset="UTF-8" Content-Transfer-Encoding: quoted-printable
On Sat, May 23, 2020 at 1:18 AM Chapman F= lack <chap@anastigmatix.net= > wrote:
On 05/22/20 14:24, Kartik Ohri wrote:

>=C2=A0 #77.8 and #77.17 illustrate this issue. In #77.8, macports is un= able to
> install openssl which is installed properly in all of the remaining bu= ilds.
> #77.17 fails with some include path error due to nar-maven.

I have seen that in reported issues too. Inconvenient that the
nar-maven-plugin developers wrote an IncludePath object that doesn't override toString(), so when it is printed in an error message it
just comes out IncludePath@hex-address.

Don't do that in your plugin. :)
=C2=A0:+1:
=C2=A0
I see Maven gives a hint there that if you run with -e it might
give a full stack trace of the errors. I haven't tried it, but
in a case like this I wonder if the full stack trace would include
a lower-level exception like FileNotFoundException that would
actually show the name that wasn't found!

My suspicion is that it also is related to some dependency like
openssl that intermittently isn't getting downloaded/installed properly= .



Another thing I notice that could be annoying when going through
these logs is that the vast majority of the log is just download
progress of all the dependencies for the build.

It looks like that bulk can be reduced a lot by adding these Maven options:=

--batch-mode
-Dorg.slf4j.simpleLogger.log.org.apache.maven.cli.transfer.Slf4jMavenTransf= erListener=3Dwarn

The first one just tones down some of the message bloat like color changes,=
and setting the TransferListener logging level to 'warn' means we s= hould see
if something goes wrong, but not fill the whole log with all the stuff
that went right.

Found here: https://stackoverflow.com/a/35653426
I will make these changes and try to reduce any other unnecessary log= s as well.=C2=A0
As another thing to think about, if every build is starting from a
clean slate, all of that stuff has to get re-downloaded every time.
If I start from a clean slate, build PL/Java once, then look at the
size of my local Maven repository:

$ du -sh ~/.m2/repository
72M=C2=A0 =C2=A0 =C2=A0/var/tmp/testbuilds/.m2/repository

that's 72 megs of stuff getting downloaded ... each time.

If we had a way of doing multiple PG version / multiple Java version
test builds without completely resetting the environment every time,
that could save the Travis guys more than half a gig of bandwidth.

For every set of test runs.

What I do here is I have several different PG versions built and
installed in testbuilds/pg12 testbuilds/pg11 and so on, and likewise
for Java versions.

Then if I run Maven like so:

JAVA_HOME=3D/var/tmp/jdk-14+36/ mvn \
=C2=A0 -Dpljava.libjvmdefault=3D/var/tmp/jdk-14+36/lib/server/libjvm.so \ =C2=A0 -Dpgsql.pgconfig=3D/var/tmp/testbuilds/pg12/bin/pg_config \
=C2=A0 clean install

it builds for that Java version and that PG version:

- the JAVA_HOME in front controls what Java version will run Maven itself - the -Dpljava.libjvmdefault builds in a default value of
=C2=A0 pljava.libjvm_location so it doesn't need to be specified later = in PG
- the -Dpgsql.pgconfig controls which PG version to build against.


Then when it has created the pljava-packaging/target/pl*jar file, I can
run it like this:

/var/tmp/jdk-14+36/bin/java \
=C2=A0 -Dpgconfig=3D/var/tmp/testbuilds/pg12/bin/pg_config \
=C2=A0 -jar pljava-packaging/target/pl*jar

which makes sure to use the tested Java version to run the jar, and
again gives the pg_config location, which determines where the files
are installed.

This doesn't all have to be in place at once, if you are focused on
getting the basic functionality in place. But it would be a significant
optimization later.
This would be worthwhile but as me= ntioned at https://docs.travis-ci.com/user/build-st= ages/#data-persistence-between-stages-and-jobs , state is not persisted= between jobs. Every build job compulsorily=C2=A0starts as a clean state. W= e can though cache dependencies to speed up the build and save bandwidth th= ough to some extent using=C2=A0https://docs.travis-ci.com/user/caching/=C2=A0.
Also, I suggest (as in the build docs) using these Maven options:

=C2=A0-Dnar.cores=3D1 -Pwnosign

The -Pwnosign only works on platforms using gcc, so it would have to be
left off for others. It silences a whole category of sign-conversion
warnings that are only distracting.

Setting nar.cores to 1 makes it a lot easier to read error messages if
there are errors; otherwise, if the compilation uses a bunch of threads, the messages all come out on top of each other and it is hard to see
where an error came from.

Also -Psaxon-examples is needed to build a set of XML examples that are
not built by default.

I will incorporate these changes into the builds.=C2= =A0
Regards,
-Chap
Thanks for the detailed feedback. I will work on= this and get back to you soon.=C2=A0
--0000000000006fe9c605a6422203--