Subject: Moving to HTTPS

From: Andrew Dunstan <[email-masked]>
Date: Sat, 7 Jan 2017 16:37:47 -0500

In common with the rest of the PostgreSQL infrastructure, we will 
shortly be moving the buildfarm server to supporting HTTPS, and 
eventually will stop support for plain HTTP. Many installations on 
buildfarm animals will not need any great adjustment, others will need 
some extra modules installed. Here's a simple test that should help tell 
if your animal needs some extra stuff installed. Run something like this 
on your animal, making sure that the perl in your path is the one used 
to run the buildfarm client:

    perl -MLWP::Simple -e 'head(q{https://www.postgresql.org});

If this doesn't complain you should be good. If it does, you probably 
need to install the LWP::Protocol::https module, and anything it depends 
on (e.g. crypto libraries).

This isn't enabled yet, I'm just being proactive so you don't get caught 
having to adjust in a hurry.

Windows users probably don't need to worry - this works on every 
installation of AS perl I have.

cheers

andrew

-- 
Sent via buildfarm-members mailing list ([email-masked])
To make changes to your subscription:
http://www.postgresql.org/mailpref/buildfarm-members

---

From: Noah Misch <[email-masked]>
Date: Sat, 7 Jan 2017 16:55:23 -0500

On Sat, Jan 07, 2017 at 04:37:47PM -0500, Andrew Dunstan wrote:
> Here's a simple test that should help tell if your animal needs some extra
> stuff installed. Run something like this on your animal, making sure that
> the perl in your path is the one used to run the buildfarm client:
> 
> 

I get no complaint from that, but I also get no complaint from these:

  perl -MLWP::Simple -e 'head(q{https://nonsense.postgresql.org});'; echo $?
  perl -MLWP::Simple -e 'head(q{nonsense://nonsense});'; echo $?

Maybe use this:

  perl -MData::Dumper -MLWP::Simple -e 'print Dumper [head(q{https://www.postgresql.org}) || "BAD"]'

---

From: Andrew Dunstan <[email-masked]>
Date: Sat, 7 Jan 2017 17:08:01 -0500

>    perl -MData::Dumper -MLWP::Simple -e 'print Dumper [head(q{https://www.postgresql.org}) || "BAD"]'

Yeah, you're right. Or perhaps even more simply we could do:

    	perl -MLWP::Simple -e 'print qq{BAD\n} unless head(q{https://www.postgresql.org});'

cheers

andrew

-- 
Andrew Dunstan                https://www.2ndQuadrant.com
PostgreSQL Development, 24x7 Support, Remote DBA, Training & Services

---

From: Tom Lane <[email-masked]>
Date: Sat, 07 Jan 2017 19:12:07 -0500

>     	perl -MLWP::Simple -e 'print qq{BAD\n} unless head(q{https://www.postgresql.org});'

Hmm, so that reports BAD on all four of my buildfarm critters :-(.

On longfin, which is running a current macOS release, "cpan install
LWP::Protocol::https" seems to be enough to fix it; but on the other
three, I get various flavors of failure.  It looks like the root cause is
that the versions of the relevant modules that are currently available
from CPAN assume newer versions of OpenSSL and/or other Perl modules than
these machines have installed.

I could embark on a massive update binge, but that would sort of defeat
the purpose of testing against old Perl versions.  Thoughts?

			regards, tom lane

---

From: Andrew Dunstan <[email-masked]>
Date: Sat, 7 Jan 2017 21:37:12 -0500

> the purpose of testing against old Perl versions.  Thoughts?

Ugh. I was afraid of something like that. We might need to look at 
providing a proxy or something.

cheers

andrew

--

---

From: David Fetter <[email-masked]>
Date: Sat, 7 Jan 2017 22:01:40 -0800

> the purpose of testing against old Perl versions.  Thoughts?

You could use install a more modern Perl version just for the
buildfarm client while keeping the system perl marooned however far in
the past we promise to support.  Don't know just how painful that is
on those machines...

Best,
David.
-- 
David Fetter <david(at)fetter(dot)org> http://fetter.org/
Phone: +1 415 235 3778  AIM: dfetter666  Yahoo!: dfetter
Skype: davidfetter      XMPP: david(dot)fetter(at)gmail(dot)com

Remember to vote!
Consider donating to Postgres: http://www.postgresql.org/about/donate

---

From: Dave Page <[email-masked]>
Date: Sun, 8 Jan 2017 11:00:15 +0000

>> the purpose of testing against old Perl versions.  Thoughts?
I get the same on the machine hosting casteroides and protociurus:

-bash-3.00$ /opt/csw/bin/perl -MLWP::Simple -e 'print qq{BAD\n} unless
head(q{https://www.postgresql.org});'
BAD

installing LWP::Protocol::https failed miserably with both Sun Studio and
GCC :-(

> providing a proxy or something.

That would kinda defeat the point of us trying to secure everything :-(

-- 
Dave Page
Blog: http://pgsnake.blogspot.com
Twitter: @pgsnake

EnterpriseDB UK: http://www.enterprisedb.com
The Enterprise PostgreSQL Company

---

From: Tom Lane <[email-masked]>
Date: Sun, 08 Jan 2017 10:34:33 -0500

> the past we promise to support.  Don't know just how painful that is

Yeah, I was thinking about the same: build a modern perl somewhere off to
the side (not in the normal PATH) and run the buildfarm script using that
while the build per se continues to use the system Perl.  The pain here
is going to mostly come from the buildfarm script itself, I'm afraid.
One thing I noticed right away is it's got /usr/bin/perl hardwired into
all its shebang lines.  Don't know what other gotchas there are.

			regards, tom lane

---

From: Andrew Dunstan <[email-masked]>
Date: Sun, 8 Jan 2017 11:16:34 -0500

> all its shebang lines.  Don't know what other gotchas there are.

That's more or less how the Msys stuff works. The buildfarm scripts run 
using the Msys DTK client (so we get the virtual paths right) but builds 
against AS perl.

And in fact there is provision for running the web transaction using a 
different perl. Right now it's only used in very limited circumstances 
(older Msys where LWP isn't supported at all in the DTK perl) but we 
could expand it. For example, we could run it any time aux_path is 
actually set. See the code starting around line 2097 of run_build.pl.

There's no great urgency about this. And I want to do it with minimal 
disturbance, so let's get it right.

cheers

andrew

---

From: Tom Lane <[email-masked]>
Date: Sun, 15 Jan 2017 03:39:42 -0500

> providing a proxy or something.

After further fooling with this, it seems the root cause is that you
need Perl (specifically Net::SSLeay) to be linked to OpenSSL 0.9.8
or later.  Since we require at least 0.9.8 for PG itself, this does
not seem like an unreasonably heavy lift.

I found that some of the involved modules, particularly IO::Socket::SSL
and LWP::Protocol::https, fail a few of their regression tests even
with 0.9.8.  But if you tell cpan to ignore that and install them
anyway, it seems like things work; at least, the simple test proposed
above now works on all four of my buildfarm critters.  It will be
interesting to see whether that equates to "buildfarm script can talk
to https server".  Do we have any more-thorough test case available?

BTW, if anyone else is like me and has a more modern openssl installed
in a non-default location on an old machine, the secret sauce for getting
Net::SSLeay to use that is to set environment variable OPENSSL_PREFIX
to the openssl install prefix path before building Net::SSLeay.

			regards, tom lane

---

From: Andrew Dunstan <[email-masked]>
Date: Sun, 15 Jan 2017 08:59:28 -0500

> to https server".  Do we have any more-thorough test case available?

The test should be thorough enough. If you want to try it for real and 
you're using the same that you built against, just change the URLs in 
your config file, i.e.:

     sed -i -e 
s,http://www.pgbuildfarm.org/,https://buildfarm.postgresql.org/, 
your-config-file

cheers

andrew

---

From: Mikael Kjellström <[email-masked]>
Date: Sun, 15 Jan 2017 15:09:50 +0100

> s,http://www.pgbuildfarm.org/,https://buildfarm.postgresql.org/,

I tried to surf to:

https://pgbuildfarm.org/

and I am getting that the certificate is invalid.  I see that you are 
using let's encrypt service for the SSL-certificate.  Just configure it 
to include all the DNS-names that you are using.   Currently the 
installed certificate is configured for the DNS-name:

buildfarm.postgresql.org

and with the SAN:

DNS Name: brentalia.postgresql.org
DNS Name: buildfarm.postgresql.org

I would suggest adding pgbuildfarm.org, www.pgbuildfarm.org etc also as 
SAN:s.

It's a pity that they don't support wildcard-certs though.

/Mikael

---

From: Tom Lane <[email-masked]>
Date: Sun, 15 Jan 2017 16:53:21 -0500

Andrew Dunstan <[email-masked]> writes:
>> ... it seems like things work; at least, the simple test proposed

> s,http://www.pgbuildfarm.org/,https://buildfarm.postgresql.org/, 

Done, and all of my machines have successfully reported builds using
this URL.  I'll leave them like that unless you say I shouldn't.

			regards, tom lane

---

From: Stefan Kaltenbrunner <[email-masked]>
Date: Mon, 16 Jan 2017 10:24:58 +0100

> I would suggest adding pgbuildfarm.org, www.pgbuildfarm.org etc also as

Thanks for the suggestion, but that is not easily doable in our current 
infrastructure - our letsencrypt deployment is fully automated and 
completely integrated with our internal config management and nameserver 
infrastructure and therefor only works with domains we also host and 
manage through/on pginfra.

regards

Stefan

---

From: Mikael Kjellström <[email-masked]>
Date: Mon, 16 Jan 2017 12:21:38 +0100

> manage through/on pginfra.

Ok, I see.   It was only a suggestion.  Maybe it isn't that important to 
use HTTPS on the pgbuildfarm-domain.

/Mikael