Subject: many animals are running old clients

From: Andrew Dunstan <[email-masked]>
Date: Fri, 17 Aug 2018 17:27:42 -0400

Animal owners,

The majority of buildfarm animals are not on the latest client code 
release. Version 8 was released more than 3 months ago. There are 26 
animals on release 7 and 43 on even older releases, in one case on a 
release going back 4 years. See below for the lists of delinquents.

If this situation doesn't improve I'm going to have to reinstitute the 
server filters which will reject very old clients.

I do my best to keep things backwards compatible, so that usually an 
upgrade is simply a matter of unpacking the release and you're done. I 
don't think asking people to do that once every few months is terribly 
onerous. It's a five minute job.

cheers

andrew

    sysname    |         owner_email         | script_version
--------------+-----------------------------+----------------
  alewife      | [email-masked]        | 007000
  conchuela    | [email-masked] | 007000
  desmoxytes   | [email-masked]          | 007000
  dory         | [email-masked]   | 007000
  dragonet     | [email-masked]          | 007000
  flaviventris | [email-masked]          | 007000
  grison       | [email-masked] | 007000
  guaibasaurus | [email-masked]     | 007000
  hamerkop     | [email-masked]      | 007000
  hornet       | [email-masked]           | 007000
  idiacanthus  | [email-masked]          | 007000
  komodoensis  | [email-masked]          | 007000
  lapwing      | [email-masked]       | 007000
  loach        | [email-masked] | 007000
  macaque      | [email-masked]             | 007000
  mandrill     | [email-masked]           | 007000
  mayfly       | [email-masked]        | 007000
  petalura     | [email-masked]          | 007000
  phycodurus   | [email-masked]          | 007000
  pogona       | [email-masked]          | 007000
  serinus      | [email-masked]          | 007000
  sidewinder   | [email-masked] | 007000
  sungazer     | [email-masked]           | 007000
  tern         | [email-masked]           | 007000
  topminnow    | [email-masked]           | 007000
  xenodermus   | [email-masked]          | 007000
(26 rows)

     sysname    |         owner_email         | script_version
---------------+-----------------------------+----------------
  chipmunk      | [email-masked]   | 004013
  gharial       | [email-masked]         | 004015
  dunlin        | [email-masked]      | 004015
  protosciurus  | [email-masked]           | 004015
  anole         | [email-masked]         | 004015
  koreaceratops | [email-masked]        | 004015
  damselfly     | [email-masked]              | 004015
  castoroides   | [email-masked]           | 004015
  clam          | [email-masked]         | 004016
  quokka        | [email-masked]         | 004016
  grouse        | [email-masked]        | 004017
  coypu         | [email-masked]           | 004017
  spurfowl      | [email-masked]          | 004018
  curculio      | [email-masked] | 004018
  rover_firefly | [email-masked]            | 004019
  thrips        | [email-masked]        | 004019
  jaguarundi    | [email-masked]        | 005000
  caiman        | [email-masked]     | 005000
  calliphoridae | [email-masked]          | 005000
  chub          | [email-masked]    | 005000
  culicidae     | [email-masked]          | 005000
  dangomushi    | [email-masked]          | 005000
  francolin     | [email-masked]          | 005000
  gull          | [email-masked]        | 005000
  handfish      | [email-masked]     | 005000
  hyrax         | [email-masked]          | 005000
  mantid        | [email-masked] | 005000
  mereswine     | [email-masked]        | 005000
  mylodon       | [email-masked]          | 005000
  piculet       | [email-masked]          | 005000
  skink         | [email-masked]          | 005000
  whelk         | [email-masked]        | 005000
  woodlouse     | [email-masked]        | 005000
  rhinoceros    | [email-masked]          | 006000
  vulpes        | [email-masked]        | 006001
  sprat         | [email-masked]        | 006001
  dhole         | [email-masked]        | 006001
  takin         | [email-masked]        | 006001
  locust        | [email-masked]           | 006001
  urocryon      | [email-masked]        | 006001
  nudibranch    | [email-masked]        | 006001
  mule          | [email-masked]       | 006001
  cuon          | [email-masked]        | 006001
(43 rows)

-- 
Andrew Dunstan                https://www.2ndQuadrant.com
PostgreSQL Development, 24x7 Support, Remote DBA, Training & Services

---

From: Tom Lane <[email-masked]>
Date: Fri, 17 Aug 2018 17:51:13 -0400

> release going back 4 years. See below for the lists of delinquents.

There's also a depressingly large number of animals that still aren't
building the REL_11_STABLE branch.  Some also have odd gaps in their
back-branch coverage.  If you don't use run_branches.pl to automate
branch selection, please also take the time to check that you are
building the right set of branches.

> server filters which will reject very old clients.

While I'd certainly encourage people to update, I'm not sure why
that's necessary?

			regards, tom lane

---

From: Andrew Dunstan <[email-masked]>
Date: Fri, 17 Aug 2018 19:57:07 -0400

> While I'd certainly encourage people to update, I'm not sure why

Example: Some animals are still getting the Pre_run_port_check error. 
That doesn't even exist any more, it disappeared when the security 
enhancements went in.

cheers

andrew

---

From: Mikael Kjellström <[email-masked]>
Date: Sat, 18 Aug 2018 12:48:12 +0200

> release going back 4 years. See below for the lists of delinquents.

I've updated all my animals below to version 8.

>   curculio      | [email-masked] | 004018

It might be worth to send out a email directly to the animal owners that 
a new version is out.  It's easy to miss if you don't follow the mailing 
lists.

Just a suggestion.

And regarding:

"I do my best to keep things backwards compatible, so that usually an 
upgrade is simply a matter of unpacking the release and you're done. I 
don't think asking people to do that once every few months is terribly 
onerous. It's a five minute job."

Every time I upgrade I need to go through all the .pl-files and change 
the path to where perl is located as different os/distros has perl at 
different paths and also it's not even consistent within the shipped files.

For example:

% head -n 1 *.pl
==> run_branches.pl <==
#!/usr/bin/perl

==> run_build.pl <==
#!/usr/bin/perl

==> run_web_txn.pl <==
#!/c/perl/bin/perl

==> setnotes.pl <==
#!/usr/bin/perl

==> update_personality.pl <==
#!/usr/bin/perl

I also go through the config-file line by line to see if anything is 
changed or added.

It's not a big deal though.  Keep up the good work!

/Mikael

---

From: Andrew Dunstan <[email-masked]>
Date: Sat, 18 Aug 2018 08:12:04 -0400

>>   curculio      | [email-masked] | 004018

Thanks.

>
> It might be worth to send out a email directly to the animal owners 
> that a new version is out.  It's easy to miss if you don't follow the 
> mailing lists.

Mail goes to the buildfarm-members list. That should be sufficiently 
specific. Having to email owners individually would be unpleasant.

> It's not a big deal though.  Keep up the good work!

run_web_txn.pl is only ever used on Windows/msys, that's why its path is 
different. If you're not using one of those systems you can safely 
ignore it.

Would it help if I added a script to set the perl locations? 
Alternatively, we could switch to using /usr/bin/env, but I'm not sure 
how universal it is. And if you want to use a non-distro perl it 
possibly won't help anyway.

cheers

andrew

---

From: Tom Lane <[email-masked]>
Date: Sat, 18 Aug 2018 11:12:03 -0400

> Would it help if I added a script to set the perl locations? 

Is it really necessary to touch the shebang lines at all?
I thought if you invoked the top script with
	perl-of-choice /path/to/run_build.pl ... args ...
then you'd be good.  If that's not so, maybe we could make it work?

>> I also go through the config-file line by line to see if anything is 

Yeah, this bit is the most time-consuming for me.  Not sure what's to
be done about it.

			regards, tom lane

---

From: Andrew Dunstan <[email-masked]>
Date: Sat, 18 Aug 2018 12:03:49 -0400

> then you'd be good.  If that's not so, maybe we could make it work?

Sadly it's not quite that simple. run_branches has this code:

    sub run_branch
    {
         my $branch = shift;
         my @args = ($run_build,
    PGBuild::Options::standard_option_list(), $branch);

         # Explicitly use perl from the path (and not this perl, so
    don't use $^X)
         # This script needs to run on Cygwin with non-cygwin perl if
    it's running
         # in tandem with AS/MinGW perl, since Cygwin perl doesn't honor
    locks
         # the samne way, and the global lock fails. But the build
    script needs
         # to run with the native perl, even on Cygwin, which it picks
    up from
         # the path. (Head exploding yet?).
         system("perl", @args);
         return;
    }

I could probably reasonably limit that so that everywhere but Cygwin we 
use the called perl. Windows is really the only place where we have to 
wrestle with multiple perls.

> Yeah, this bit is the most time-consuming for me.  Not sure what's to

It's in my list of things to consider. One of the things on the roadmap 
in my head is a possible switch to a YAML config file.

Perhaps I should publish in the release notes a diff of the sample 
config file against the previous release.

cheers

andrew

---

From: Mikael Kjellström <[email-masked]>
Date: Sun, 19 Aug 2018 11:04:54 +0200

> specific. Having to email owners individually would be unpleasant.

OK, yes I agree.

> different. If you're not using one of those systems you can safely 

OK, that makes sense.

> possibly won't help anyway.

Well, it's not that hard to go through all the .pl-files manually and 
change the path to where perl is located.

Would be nice to have a script that automatically detected the perl 
binary through using which or something like that but that wouldn't be 
portable and work on other OS:es like Windows, I guess.

I did get a problem with my NetBSD 7 animal not accepting the 
SSL-certificate (Let's Encrypt) that https://git.postgresql.org uses.

I got the error:
Git mirror failure:
Cloning into bare repository '/home/pgbf/buildroot/pgmirror.git'...
fatal: unable to access 
'https://git.postgresql.org/git/postgresql.git/': SSL certificate 
problem: unable to get local issuer certificate

I managed to solve that in NetBSD but following the following guide:

https://www.cambus.net/installing-ca-certificates-on-netbsd/

what that does is basically installing the mozilla-rootscerts on to the 
system and using it as default.

After that it worked fine again.  But I can imagine on older OS:es 
where you can't easily upgrade the CA-certs on the system it could be a 
big problem. HTTPS and the cert business is so broken it's not even 
funny imho.

/Mikael

---

From: Andrew Dunstan <[email-masked]>
Date: Sun, 19 Aug 2018 15:04:58 -0400

On 08/18/2018 12:22 PM, Noah Misch wrote (offlist):
> On Sat, Aug 18, 2018 at 08:17:17AM -0400, Andrew Dunstan wrote:
>>
>> Is there anything I can do to make upgrade easier?
> The show_log.pl web page could highlight a particular run's config change,
> including version number changes, when there is such a change compared to the
> previous run.  If it's clear enough that folks are unlikely to confuse an
> upgrade-induced failure with a PG-commit-induced failure, I could stop checking
> for post-upgrade buildfarm failures.

What I've done is add this to the database in a way that makes it easily 
accessible to the web app, and added that info to the history page.

I might break that out into a separate history at some stage.

>
> It wouldn't make an upgrade easier, but the "Flags" column of show_status.pl
> could give an extra badge for being on the latest version.  That would motivate
> some folks.

Yes, I'll look at this, although I'm reluctant to make the dashboard any 
more busy.

cheers

andrew

---

From: Gael Le Mignot <[email-masked]>
Date: Mon, 20 Aug 2018 11:01:02 +0200

Hello,

I updated mine (mule), sorry for the lag on the update.

Regards,
-- 
Gaël Le Mignot - [email-masked]
Pilot Systems - 82, rue de Pixérécourt - 75020 Paris
Tel : +33 1 44 53 05 55 - www.pilot-systems.net
Découvrez notre offre Cloud privé 100% infogéré - www.pilotsystems.net/cloud/

---

From: Sandeep Thakkar <[email-masked]>
Date: Mon, 20 Aug 2018 15:55:55 +0530

Hi Andrew

Sorry about that. I've updated all the below animals to version 8.
- anole
- gharial
- clam
- quokka

I copied the buildfarm.conf from the previous client installation directory
and to the new one. Hope that should be OK.

-- 
Sandeep Thakkar

---

From: Sandeep Thakkar <[email-masked]>
Date: Mon, 20 Aug 2018 16:02:17 +0530

Hi Tom,

On Sat, Aug 18, 2018 at 3:21 AM, Tom Lane <[email-masked]> wrote:

> All my animals use run_branches.pl except 'clam', which is a PPC64LE
machine. and when we setup this animal we found 9.3 was failing with
"error: cannot guess build type; you must specify one". This was discussed
in the thread with sub: "Include ppc64le build type for back branches".
Therefore, I explicitly run run_build.pl for all branches (I added for
REL_11_STABLE as it was missing). Once we disable 9.3 runs, I'll move to
using run_branhces.pl on 'clam' as well.

-- 
Sandeep Thakkar

---

From: Noah Misch <[email-masked]>
Date: Mon, 20 Aug 2018 20:24:40 -0700

> accessible to the web app, and added that info to the history page.

I think that wastes prime screen real estate.  If show_history.pl is going to
spend that space on something, I would choose commit hash.  But I liked it
better with nothing there.

The concept I had in mind was a "Config changed since last success" block
adjacent to the "Files changed since last success" block in show_log.pl.  It
might look like this:

- 'config_env' => { 'CC' => 'gcc' },
+ 'config_env' => { 'CC' => 'gcc -mips32r2' },
- 'script_version' => 'REL_7',
+ 'script_version' => 'REL_8',

Script version is a minor aspect, worth little by itself.  If you diff the
entire config block, though, that amounts to a meaningful feature.

---

From: Andrew Dunstan <[email-masked]>
Date: Tue, 21 Aug 2018 06:58:01 -0400

> entire config block, though, that amounts to a meaningful feature.

It's easy to revert the display change, and there is utility in having 
the script version easily available in the database without having to 
extract it in each query.

Getting the config summary from the previous build is possibly a bit 
less simple.

I'll take a look when I get a bit more time.

cheers

andrew

---

From: Bernd Helmle <[email-masked]>
Date: Wed, 22 Aug 2018 17:05:18 +0200

>   chub          | [email-masked]    | 005000

chub should be up to date now and builds REL_11_STABLE, too.

---

From: Dave Page <[email-masked]>
Date: Wed, 29 Aug 2018 15:35:40 +0100

> onerous. It's a five minute job.

It's very far from a 5 minute job on ancient Solaris boxes :-(

In my case, I've spent an hour or more trying to get Digest::SHA to
install, and have failed miserably as it fails to compile for me with both
Sun Studio and GCC. In the case of Sun Studio, CPAN seems insistent on
trying to use the compiler in a way that makes it react like the crippled
default compiler on the system, and it ignores variables like CC, and
aliases and so-on, and insists on trying to run "cc" regardless. I'm going
to have to give up for now.

-- 
Dave Page
Blog: http://pgsnake.blogspot.com
Twitter: @pgsnake

EnterpriseDB UK: http://www.enterprisedb.com
The Enterprise PostgreSQL Company

---

From: Andrew Dunstan <[email-masked]>
Date: Wed, 29 Aug 2018 14:14:42 -0400

On 08/29/2018 10:35 AM, Dave Page wrote:
>
>
> On Fri, Aug 17, 2018 at 10:27 PM, Andrew Dunstan 
> <[email-masked] 
> <mailto:[email-masked]>> wrote:
>
>
>     Animal owners,
>
>
>     The majority of buildfarm animals are not on the latest client
>     code release. Version 8 was released more than 3 months ago. There
>     are 26 animals on release 7 and 43 on even older releases, in one
>     case on a release going back 4 years. See below for the lists of
>     delinquents.
>
>
>     If this situation doesn't improve I'm going to have to reinstitute
>     the server filters which will reject very old clients.
>
>
>     I do my best to keep things backwards compatible, so that usually
>     an upgrade is simply a matter of unpacking the release and you're
>     done. I don't think asking people to do that once every few months
>     is terribly onerous. It's a five minute job.
>
>
> install, and have failed miserably as it fails to compile for me with 
> both Sun Studio and GCC. In the case of Sun Studio, CPAN seems 
> insistent on trying to use the compiler in a way that makes it react 
> like the crippled default compiler on the system, and it ignores 
> variables like CC, and aliases and so-on, and insists on trying to run 
> "cc" regardless. I'm going to have to give up for now.
>
>

eep!

Please let me know when issues like this arise.

Note BTW that we switched from Digest::SHA1 to Digest::SHA back in 2013, 
because it's more commonly available, and I never heard of a problem 
with that until now, AFAIR. Assuming you have Digest::SHA1 available, I 
could probably save you all this grief by trying to use one and then the 
other.

More generally, if people have difficulties I try to work out a way 
around them. So please do sing out when that happens.

cheers

andrew

---

From: Tom Lane <[email-masked]>
Date: Wed, 29 Aug 2018 18:17:23 -0400

> Please let me know when issues like this arise.

One thing of the same ilk is that a few months ago I had to give up on
using https reporting on gaur/pademelon, and switch back to http.
I don't think that was your fault, because it was not associated with a
buildfarm client upgrade.  It appeared to me that the pginfra folk had
changed something to require newer TLS versions in https connections to
the buildfarm server.  Like Dave, I spent a fair while trying to install
new-enough Perl modules to fix that, without a lot of success: newer
Perl code doesn't compile on that platform.

It's debatable of course how much anyone cares about buildfarm animals
that are this old.  I've already shut down pademelon since its compiler
never heard of C99, and I don't see much point in running it on just
the back branches.

			regards, tom lane

---

From: Dave Page <[email-masked]>
Date: Thu, 30 Aug 2018 11:32:30 +0100

> never heard of C99, and I don't see much point in running it on just

Right - I do wonder that about this machine (whether it's worth supporting,
not whether its compiler supports C99). Here's what we're talking about:

-bash-3.00$ cat /etc/release
                       Solaris 10 11/06 s10s_u3wos_10 SPARC
           Copyright 2006 Sun Microsystems, Inc.  All Rights Reserved.
                        Use is subject to license terms.
                           Assembled 14 November 2006
-bash-3.00$ prtconf -b
name:  SUNW,Sun-Blade-1000
model:  SUNW,501-6230
banner-name:  SUNW,Sun-Blade-2000 (2 X UltraSPARC-III+)

---

From: Darcy Buskermolen <[email-masked]>
Date: Thu, 30 Aug 2018 05:49:36 -0500

That particular era of equipment is still quite prominent in telecom central office environments, and I know of several telcom nms solutions that use PostgreSQL.  However having said that it's of little likelyhood that those are being updated to use newer versions of postgres or even wether or not they receive security/bugfixes....

Sent on the go, from somewhere other than here.
-------- Original message --------From: Dave Page <[email-masked]> Date: 2018-08-30  05:32  (GMT-06:00) To: Tom Lane <[email-masked]> Cc: Andrew Dunstan <[email-masked]>, [email-masked] Subject: Re: many animals are running old clients 

the back branches.

Right - I do wonder that about this machine (whether it's worth supporting, not whether its compiler supports C99). Here's what we're talking about:
-bash-3.00$ cat /etc/release                       Solaris 10 11/06 s10s_u3wos_10 SPARC           Copyright 2006 Sun Microsystems, Inc.  All Rights Reserved.                        Use is subject to license terms.                           Assembled 14 November 2006-bash-3.00$ prtconf -bname:  SUNW,Sun-Blade-1000model:  SUNW,501-6230banner-name:  SUNW,Sun-Blade-2000 (2 X UltraSPARC-III+)
-- 
Dave Page
Blog: http://pgsnake.blogspot.com
Twitter: @pgsnake

---

From: Dave Page <[email-masked]>
Date: Thu, 30 Aug 2018 12:00:54 +0100

On Thu, Aug 30, 2018 at 11:49 AM, Darcy Buskermolen <[email-masked]>
wrote:

> That particular era of equipment is still quite prominent in telecom
> central office environments, and I know of several telcom nms solutions
> that use PostgreSQL.  However having said that it's of little likelyhood
> that those are being updated to use newer versions of postgres or even
> wether or not they receive security/bugfixes....
>

Right - we have some customers in that category; they're usually the ones
asking for extended support on ancient releases.

> -------- Original message --------
> From: Dave Page <[email-masked]>
> Date: 2018-08-30 05:32 (GMT-06:00)
> To: Tom Lane <[email-masked]>
> Cc: Andrew Dunstan <[email-masked]>,
> [email-masked]
> Subject: Re: many animals are running old clients
>
>
>
> Right - I do wonder that about this machine (whether it's worth
> supporting, not whether its compiler supports C99). Here's what we're
> talking about:
>

---

From: Noah Misch <[email-masked]>
Date: Thu, 30 Aug 2018 06:58:31 -0700

>                            Assembled 14 November 2006

The lack of any other Solaris animal makes it more valuable than its age alone
would imply.