Received: from malur.postgresql.org ([217.196.149.56]) by arkaria.postgresql.org with esmtps (TLS1.3:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.92) (envelope-from ) id 1kCevr-0000nE-Om for pljava-dev@arkaria.postgresql.org; Mon, 31 Aug 2020 08:12:52 +0000 Received: from localhost ([127.0.0.1] helo=malur.postgresql.org) by malur.postgresql.org with esmtp (Exim 4.92) (envelope-from ) id 1kCevp-0001Oq-LX for pljava-dev@arkaria.postgresql.org; Mon, 31 Aug 2020 08:12:49 +0000 Received: from makus.postgresql.org ([2001:4800:3e1:1::229]) by malur.postgresql.org with esmtps (TLS1.3:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.92) (envelope-from ) id 1kCevp-0001ON-1R for pljava-dev@lists.postgresql.org; Mon, 31 Aug 2020 08:12:49 +0000 Received: from mail-ej1-x62f.google.com ([2a00:1450:4864:20::62f]) by makus.postgresql.org with esmtps (TLS1.3:ECDHE_RSA_AES_128_GCM_SHA256:128) (Exim 4.92) (envelope-from ) id 1kCevm-000121-1c for pljava-dev@lists.postgresql.org; Mon, 31 Aug 2020 08:12:47 +0000 Received: by mail-ej1-x62f.google.com with SMTP id i26so6926055ejb.12 for ; Mon, 31 Aug 2020 01:12:45 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=mime-version:from:date:message-id:subject:to; bh=pYWrfJf01cp1EzwVca7kgDVURlemoWv+OWwkyiMk11M=; b=Cc2ZNkYPVsPX8TCei78/IRxMlJjzvtU0dLc/Fd6VuAo6uYCFXHNVc1eyJzWd9X2WLY Jgt4pPfDdWb3m1Ll6lyhP8iTWhV4hQczJOJaOWiD337sjUpiBu9ymABF5r+qdUrHN+VT 1qyL/ZhDBYnR7eBijs4Ohg/XUkZwxYs19kqR5Fawf3ZKfL4hJPaJMwkkW70Jl+k/bswy wvV1jeVySHcqH12omaiVCfT/TrXWL6xHnL2TqNxqTPd8xn5dif0JpZWPkORn7xxAyRaC iK0m7wTnQ5hQhUOuFzvJFDDCxIbG7KHL1mgImi1A0LHWQOGBiMny1mvJP6rCtociMcTJ 3whA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:mime-version:from:date:message-id:subject:to; bh=pYWrfJf01cp1EzwVca7kgDVURlemoWv+OWwkyiMk11M=; b=gcQr201H95IKPzk2F3MYohY4XTfOXMTsT2QJQIzuVwxcMmMXtrM4NmeDYLQmwwvTz1 ob86AuPpiR/tDkiJNw5j/jo/KVf5SmIkt9VuAZ5/11Lh6Ld6o6s732Tma+ctWdMJqybq 77HgS4OYG2TP/9k248WjYQZ3vNCws8mvJcVhsvkFl4kRD0y5WzpJUDSfOTP2kizLA4/S bFX1vstqaZtz0T2z7/lTWhDFfCK0zYZZr55+e7Gxmi/v9uAz9K0D+ABUlVaToFGX0x95 4vr2/+5lSllp2+78iXX6tYv2IDa8kF5Ji30dmeu9UnHjw1z6qBSKRtybtgWl5Z3kkjL3 FIOA== X-Gm-Message-State: AOAM533A0ZIiinnLXiNYlhxLQ6slflUBCPsZLnLJQKKFuOK8I+dAeYUm YMMZmn6qWQlpy5UPZef7BHHCItiLUMqhbK92uuCx99aLjPTQIw== X-Google-Smtp-Source: ABdhPJxbAjzjerKT2nU3pX3awVOrR8/qYQnAMMOD45TNCW6X8FkppUF5WzQE/Mk3frVi4dMqfC3MkTh+nIK2OCxi62Y= X-Received: by 2002:a17:906:2296:: with SMTP id p22mr51581eja.510.1598861563870; Mon, 31 Aug 2020 01:12:43 -0700 (PDT) MIME-Version: 1.0 From: Kartik Ohri Date: Mon, 31 Aug 2020 13:42:30 +0530 Message-ID: Subject: Renovating the PL/Java build process To: pljava-dev@lists.postgresql.org Content-Type: multipart/alternative; boundary="0000000000008f41eb05ae27f71c" List-Id: List-Help: List-Subscribe: List-Post: List-Owner: List-Archive: Precedence: bulk --0000000000008f41eb05ae27f71c Content-Type: text/plain; charset="UTF-8" It is the time of the summer when GSoC participants look back on the work done in the past few months and so here are we. The original project idea and the project proposal had three major components. Each of these has been discussed in detail below. 1) Continuous Integration Expected project outcome: CI builds and reports occur on pull requests, for multiple versions of PostgreSQL and Java, and on Linux, macOS, and Windows. Linked Pull Request: https://github.com/tada/pljava/pull/289 State of the work: The work has been largely successful. PL/Java now has a CI pipeline built using Travis CI and AppVeyor CI. In a repository where these CI providers have the required access, every commit and PR are built and tested on over 30 combinations of PostgreSQL and Java versions across Linux, Mac OS X and Windows (MSVC and MinGW). This has already begun to bear fruit as can be seen from https://github.com/tada/pljava/issues/282 and https://github.com/tada/pljava/issues/274. Scope for future work: However, there is scope of several improvements. The CI infrastructure can be moved to Github Actions. The motive is to eliminate unnecessary access to external providers. Also, the test suite can be made more comprehensive by adding builds from PostgreSQL source on Mac OS X and Windows platform as well. 2) PL/Java API Artifact Expected project outcome: pljava-api jar artifact deployed to Maven Central when a new release is made. State of the Work: The component has not been satisfied yet. However, the mentor is satisfied with the progress since there is no new artifact ready for release yet. It is our consensus that the introduction of PL/Java PGXS Plugin and CI service providers has gone a long way in this respect. The task should be relatively easy to accomplish now. Scope for future work: Add a deploy phase to either of the CI providers to publish the pljava-api artifact. 3) Replacing nar-maven-plugin Expected project outcome: Build system no longer uses nar-maven-plugin, correctly uses build settings from pg_config, and is tested on at least Linux, macOS, and Windows (with Visual Studio and with MinGW). Linked Pull Request: https://github.com/tada/pljava/pull/288 https://github.com/tada/pljava/pull/292 (commit SHA #1df123b ) State of the Work: The PL/Java PGXS plugin has successfully retired nar-maven-plugin. The CI builds for the project being built using the PGXS plugin are passing. The PGXS plugin also queries pg_config to obtain various flags and passes them on to the compiler and the linker. Overall, the plugin is in a functional state and works just fine. Several useful utilities are also exposed by the plugin for using in the scripts. PL/Java PGXS is a complete rework of the earlier build process in a true sense. It addresses many pain points that the nar maven plugin had introduced. The biggest issue posed by nar maven was highly noisy builds. Finding the actual errors among the tons of false positives was no easier than finding a needle in a haystack. To make matters worse, the debugging information provided by the nar plugin was average at best. For instance, many times the build would fail with an error in include paths. Ok fine, let's see why the path was incorrect. But no, you can only see an id assigned to the include path by the JVM. Good luck debugging with that. Another pressing issue is that nar tries to abstract a lot of details, hence compromising code readability. The actual arguments and configuration details passed to the compiler and linker are hidden deep inside nar maven's source code. On the other hand, the pljava-so pom.xml file had a huge block of javascript to just quote a string correctly for passing to the C code. The point is not to berate nar maven plugin. It is a great plugin, but it is just not suited for our simple use case. Enter PL/Java PGXS. PL/Java PGXS turns this completely inside out. It hides the uninteresting bland details and puts the important configuration details in the pom. PL/Java PGXS is specifically geared towards building PL/Java, hence unnecessary abstractions can be done away with. The PGXS plugin has also retired the troublesome maven-javadoc-plugin and supports building reports and scripting during the site phase. maven-javadoc-plugin tries to micromanage a lot of stuff when just running the javadoc on a set of files is enough to do the right job. The current implementation using PGXS plugin just does that by utilizing the APIs exposed by Java itself. Scope for Future Work: Since PGXS plugin supports scripting, there is no need for using maven antrun plugin for the same. The necessary code can be ported from using maven antrun plugin to use PGXS instead. Further, the exported symbols can be filtered to expose only PL/Java entry points. In addition, currently the output produced by the compiler and linker is redirected to standard output and standard error streams. It would be better if the output was rather piped through the plugin in which case it could be filtered and utilised as desired. --0000000000008f41eb05ae27f71c Content-Type: text/html; charset="UTF-8" Content-Transfer-Encoding: quoted-printable

It is the time of the summer when=C2=A0= GSoC=C2= =A0participants look back on the work done in the past few months and so he= re are we. The original project idea and the project proposal had three maj= or components. Each of these has been discussed in detail below.=C2=A0

= =C2=A0

=

1) Continuous Integration=C2=A0

= =C2=A0

<= p class=3D"gmail-Paragraph gmail-BCX0 gmail-SCXW154559427" lang=3D"EN-US" s= tyle=3D"margin:0px;padding:0px;vertical-align:baseline;background-color:tra= nsparent;color:windowtext">Expected proj= ect outcome: CI builds and reports occur on pull requests, for multiple ver= sions of PostgreSQL and Java, and on Linux, macOS, and Windows.=C2=A0

<= span class=3D"gmail-TextRun gmail-EmptyTextRun gmail-BCX0 gmail-SCXW1545594= 27" lang=3D"EN-US" style=3D"margin:0px;padding:0px;font-kerning:none;line-h= eight:19.425px;font-variant-ligatures:none">= =C2=A0
Linked Pull Request:=C2=A0https://github.com/tada/pljava/pull/289=C2=A0

=C2=A0
State of the wor= k: The work has been largely successful. PL/Java now has a CI pipeline buil= t using Travis CI and=C2=A0AppVeyor=C2=A0CI. In a repository where these CI provid= ers have the required access, every commit and PR=C2=A0are=C2=A0built and tested on over 30 combinations of PostgreSQL a= nd Java versions across Linux, Mac OS X and Windows (MSVC and MinGW). This = has already begun to bear fruit as can be seen from=C2=A0https= ://github.com/tada/pljava/issues/282=C2=A0and=C2=A0https://github.com/tada/pljava/issues/274.=C2= =A0=C2=A0

=C2=A0
Scope for future wor= k: However, there is scope of several improvements. The CI infrastructure c= an be moved to=C2=A0Gi= thub=C2=A0Actions. The motive is to eliminate unnecessary ac= cess to external providers. Also, the test suite can be made more comprehen= sive by adding builds from PostgreSQL source on Mac OS X and Windows platfo= rm as well.=C2=A0<= /font>

= =C2=A0

2) PL/Java API Artifact=C2=A0
=C2=A0

Expected project outcome:=C2=A0pljava-api=C2=A0jar artifact deployed to Maven Central when a new release is= made.=C2=A0

=C2=A0
State of the Work= : The component has not been satisfied yet. However, the mentor is satisfie= d with the progress since there is no new artifact ready for release yet. I= t is our consensus that the introduction of PL/Java PGXS Plugin and CI serv= ice providers has gone a long way in this respect. The task should be relat= ively easy to accomplish=C2=A0now.=C2=A0

=C2=A0
Scope for future work: Add a dep= loy phase to either of the CI providers to publish the=C2=A0<= span class=3D"gmail-TextRun gmail-BCX0 gmail-SCXW154559427" lang=3D"EN-US" = style=3D"margin:0px;padding:0px;font-kerning:none;line-height:19.425px;font= -variant-ligatures:none">pljava-api=C2=A0arti= fact.=C2=A0=

=C2=A0

3) Replacing=C2=A0nar-maven-plugin<= /span>=C2=A0

=C2=A0
=C2=A0

Expected project outcome: Build system no longer uses=C2= =A0nar-maven-plugin, correctly uses build settings from=C2=A0pg_config,= and is tested on at least Linux, macOS, and Windows (with Visual Studio an= d with MinGW).=C2=A0

=C2=A0
Linked Pu= ll Request:=C2=A0=C2=A0

https://git= hub.com/tada/pljava/pull/288=C2=A0=C2=A0=C2=A0=C2=A0=C2= =A0=C2=A0

https://github.com/tada/plj= ava/pull/292=C2=A0(commit SHA #1df123b)=C2=A0

=C2=A0
State of the Work: The= PL/Java PGXS plugin has successfully retired=C2=A0nar-maven-plugin. The CI builds= for the project being built using the PGXS plugin are passing. The PGXS pl= ugin also queries=C2=A0pg_config=C2=A0to obtain various flags and passes = them on to the compiler and the linker. Overall, the plugin is in a functio= nal state and works just fine. Several useful utilities are also exposed by= the plugin for using in the scripts.=C2=A0=C2=A0

=C2=A0
PL/Java PGXS is a complete rework of the=C2=A0earlier=C2=A0build process in a true sense. It address= es=C2=A0many pain points that=C2=A0the=C2=A0nar=C2=A0maven plugin=C2=A0had introduced. The biggest issue posed by=C2=A0nar=C2=A0maven was highly no= isy builds. Finding the actual errors among the tons of false positives was= no easier than finding a needle in a haystack. To make matters worse, the = debugging information provided by the=C2=A0nar=C2=A0plugin was average at best. Fo= r instance, many times the build would fail with an error in include paths.= Ok fine,=C2=A0let's see why the path was= incorrect. But=C2=A0no,=C2=A0you can only se= e an id assigned to the include=C2=A0path=C2= =A0by the JVM. Good luck debugging with that.=C2=A0=C2=A0
=C2=A0
Another pressing issue is that=C2=A0nar=C2=A0tries to abstrac= t a lot of details, hence compromising code readability. The actual argumen= ts and configuration details passed to the compiler and linker=C2=A0are hidden=C2=A0deep inside=C2=A0nar=C2=A0maven's sour= ce code. On the other hand, the=C2=A0pljava-so pom.xml file had a huge block of=C2= =A0javascript=C2=A0to just quote a string correctly for passing to the C code.=C2=A0The point is not to berate=C2=A0nar=C2=A0maven pl= ugin. It is a great=C2=A0plugin,=C2=A0but it = is just not suited for our simple use case.=C2=A0=C2=A0

<= p class=3D"gmail-Paragraph gmail-BCX0 gmail-SCXW154559427" lang=3D"EN-US" s= tyle=3D"margin:0px;padding:0px;vertical-align:baseline;background-color:tra= nsparent;color:windowtext">=C2=A0

<= span class=3D"gmail-TextRun gmail-BCX0 gmail-SCXW154559427" lang=3D"EN-US" = style=3D"margin:0px;padding:0px;font-kerning:none;line-height:19.425px;font= -variant-ligatures:none">= Enter PL/Java PGXS. PL/Java PGXS turns this completely inside out. It hides= the uninteresting bland details and puts the important configuration detai= ls in the pom. PL/Java PGXS is specifically geared towards building PL/Java= , hence unnecessary abstractions can be done away with.=C2=A0

=C2=A0
The PGXS plugin has also retired the troubl= esome maven-javadoc-plugin and supports building reports and scripting during the = site phase. maven-java= doc-plugin tries to=C2=A0micromanage=C2=A0a lot of stuff when just running the=C2=A0javadoc=C2=A0on a set of fi= les is enough to do the right job. The current implementation= =C2=A0using PGXS plugin=C2=A0just does that by=C2=A0= utilizing=C2=A0the APIs exposed by Java itself.=C2=A0

=

=C2=A0
Scope for Future = Work: Since=C2=A0PGXS plugin supports scripti= ng, there is no need for using maven=C2=A0antrun=C2= =A0plugin for the same.=C2=A0The necessary code can be porte= d from using maven=C2=A0=C2=A0plugin to use PGXS instead.=C2=A0Further, the exported symbols can be filtered to expose only PL/Java entr= y points.=C2=A0In addition, currently the out= put=C2=A0produced by the compiler and linker= =C2=A0is redirected to standard output and standard error streams. It would= be better if the output was rather piped through the plugin in which case = it could be filtered and=C2=A0utilised=C2=A0as desired.=C2=A0

=
--0000000000008f41eb05ae27f71c--