Subject: New and unified 9.5 spec file is in git

From: Devrim GÜNDÜZ <[email-masked]>
Date: Tue, 19 Jan 2016 23:44:41 +0200

Hi,

After various attempts to make RHEL 5 happy, I pushed unified
PostgreSQL 9.5 spec file to git, which is supposed to work for all
distros. 

The spec file also adds plpython3 subpackage (on supported distros),
including patches from Fedora for Python 3.5 tests (on supported
distros), adds dtrace support (on supported distros), and some cosmetic
cleanup.

I'm about to push packages to repo. I tested them, hopefully I did not
miss anything :-)

If anyone could take a look at it, it would be awesome.

Regards,
-- 
Devrim GÜNDÜZ
Principal Systems Engineer @ EnterpriseDB: http://www.enterprisedb.com
PostgreSQL Danışmanı/Consultant, Red Hat Certified Engineer
Twitter: @DevrimGunduz , @DevrimGunduzTR

---

From: Craig Ringer <[email-masked]>
Date: Wed, 20 Jan 2016 10:15:21 +0800

> PostgreSQL 9.5 spec file to git, which is supposed to work for all

Great! That's wonderful news.

I hope you got some use out of the work I sent you a while ago on that.
Apologies that I didn't find the time to follow up on it.

> If anyone could take a look at it, it would be awesome.

I'll look right now.

-- 
 Craig Ringer                   http://www.2ndQuadrant.com/
 PostgreSQL Development, 24x7 Support, Training & Services

---

From: Craig Ringer <[email-masked]>
Date: Wed, 20 Jan 2016 11:52:17 +0800

> If anyone could take a look at it, it would be awesome.

It built perfectly with mock on Fedora 21 for a F23 target. Brilliant!

To make it clear that they're the same is it worth having a common/
directory at the same level as EL-5, F-23, etc and symlinking the specfile
to the various subdirs from there, along with any identical patches etc?

Here's the full build process I used:

sudo yum -y install rpmdevtools mock
git clone git://git.postgresql.org/git/pgrpms.git
cd pgrpms/rpm/redhat/9.5/postgresql/F-23
spectool --get-files postgresql-9.5.spec
mock -r fedora-23-x86_64 --buildsrpm --sources . --spec postgresql-9.5.spec
cp
/var/lib/mock/fedora-23-x86_64/result/postgresql95-9.5.0-2PGDG.fc23.src.rpm
.
mock -r fedora-23-x86_64  --rebuild postgresql95-9.5.0-2PGDG.fc23.src.rpm

I'm really happy about this, since it means the build depends are now
correct and it can be built without needing a dedicated build machine using
nothing but mock. No need to manually set up dependencies anymore, set up a
build box, etc. No need to juggle x86_64 and i386 either, you can run
builds for *everything* from a single F23 box or VM or docker or whatever.

If the same process is followed for other packages then it'll become
possible to build the whole PGDG system with mockchain with no custom build
machines, building anywhere that mock is available. No accidental
undeclared build or runtime dependencies.  Since you seem to be using
Docker to prep build envs now it's less of an issue than when the build
boxes are hand-maintained, but the great thing about mock is that it gets
all the build depends info straight from the spec file.

I also rebuilt the F23 package for F21 just by changing the -r argument to
mock. Same for building it for i386 on an x86_64 host.

The EL-5 package built happily under mock for -r epel-5-i386 too. I didn't
test install it.

Thanks so much. This is great and I really hope you got some use out of my
prior work on it.

---

From: Craig Ringer <[email-masked]>
Date: Wed, 20 Jan 2016 12:22:57 +0800

I've uploaded a branch with a couple of small improvements. See
https://github.com/2ndQuadrant/pgdg-pgrpms .

git remote add 2ndq https://github.com/2ndQuadrant/pgdg-pgrpms.git
git fetch 2ndq
# view it
git log -p remotes/2ndq/add-builddepends
# merge it
git merge remotes/2ndq/add-builddepends

This is the main patch of interest:

https://github.com/2ndQuadrant/pgdg-pgrpms/commit/60524156129b27b9cf3836621a0d05afb658d60e

A further suggested improvement, which I haven't put in that tree but
wanted to flag as an idea - detect the target based on the directory name
so there's no need to have a different Makefile for each target dir and
they can all just be symlinks.

For each dist's Makefile:

-DIST=.rhel5
 SPECFILE="postgresql-9.5.spec"
+# This gibberish gets the immediate parent directory's name, prepends
RPMDIST_
+# and looks it up as a Make variable lazily on first evaluation. That way
we
+# don't require a new Makefile for each build directory.
+#
+# The RPMDIST_ vars are defined in Makefile.global
+#
+DIST=$(RPMDIST_$(shell basename $(dir $(abspath $(lastword
$(MAKEFILE_LIST))))))

then in Makefile.global:

+# These mappings could be automated with $(foreach ...) and $(eval ...)
+# but there hardly seems any point.
+RPMDIST_EL-5=.rhel5
+RPMDIST_EL-6=.rhel6
+RPMDIST_EL-7=.rhel7
+RPMDIST_F-21=.fc21
+RPMDIST_F-22=.fc22
+RPMDIST_F-23=.fc23

Then EL-5, EL-6, EL-7, F-22 and F-23 can be deleted and replaced with
common/ which is symlinks to each of them. That shouldn't upset the build
system; otherwise I'd suggest switching moving all files up a level,
removing the dist subtrees, and using "make DIST=EL-5" or whatever to
control the target.

Less duplication in the source tree, no need to remember to copy files
around, easier to track history.

---

From: Jeff Frost <[email-masked]>
Date: Tue, 19 Jan 2016 20:33:23 -0800

> On Jan 19, 2016, at 19:52, Craig Ringer <[email-masked]> wrote:
> 
> To make it clear that they're the same is it worth having a common/ directory at the same level as EL-5, F-23, etc and symlinking the specfile to the various subdirs from there, along with any identical patches etc?
> 
> cp /var/lib/mock/fedora-23-x86_64/result/postgresql95-9.5.0-2PGDG.fc23.src.rpm .
> I'm really happy about this, since it means the build depends are now correct and it can be built without needing a dedicated build machine using nothing but mock. No need to manually set up dependencies anymore, set up a build box, etc. No need to juggle x86_64 and i386 either, you can run builds for *everything* from a single F23 box or VM or docker or whatever.
> 
> If the same process is followed for other packages then it'll become possible to build the whole PGDG system with mockchain with no custom build machines, building anywhere that mock is available. No accidental undeclared build or runtime dependencies.  Since you seem to be using Docker to prep build envs now it's less of an issue than when the build boxes are hand-maintained, but the great thing about mock is that it gets all the build depends info straight from the spec file.
> 
> I also rebuilt the F23 package for F21 just by changing the -r argument to mock. Same for building it for i386 on an x86_64 host.
> 
> The EL-5 package built happily under mock for -r epel-5-i386 too. I didn't test install it.
> 
> Thanks so much. This is great and I really hope you got some use out of my prior work on it.
> 

This makes me so excited! I can't wait to try it and work on getting the other packages set up this way as well!

Great job you guys!

---

From: Devrim GÜNDÜZ <[email-masked]>
Date: Wed, 20 Jan 2016 22:11:48 +0200

Hi Craig,

On Wed, 2016-01-20 at 10:15 +0800, Craig Ringer wrote:

> I hope you got some use out of the work I sent you a while ago on
> that. Apologies that I didn't find the time to follow up on it.

Even though I did not use your actual work, I must admit that I learned
*a lot* from your spec. Apologies for not crediting you over there.
Thank you very much!

Regards,

---

From: Devrim GÜNDÜZ <[email-masked]>
Date: Wed, 20 Jan 2016 22:17:14 +0200

Hi,

> It built perfectly with mock on Fedora 21 for a F23 target.

Thanks for checking!

> directory at the same level as EL-5, F-23, etc and symlinking the
> specfile to the various subdirs from there, along with any identical
> patches etc?

That would be the next step. Let's finish the actual work first.

> correct and it can be built without needing a dedicated build machine
> using nothing but mock. No need to manually set up dependencies
> anymore, set up a build box, etc. No need to juggle x86_64 and i386
> either, you can run builds for *everything* from a single F23 box or
> VM or docker or whatever.

Well, we are building 100+ packages in 9.5 set. Setting up a new mock
environment everytime would slow us down, IMHO. What do you think?

> If the same process is followed for other packages 

*All* of the *recently added* spec files are unified nowadays :) This
also applies to some specs that I found time to rework on (like
postgis, repmgr, etc.). The repo is not fully unified yet, but we are
on it.

> Since you seem to be using Docker to prep build envs now

Err, no. We are using actual VMs (powered by libvirtd+qemu-kvm)

>  it's less of an issue than when the build boxes are hand-maintained, 
> but the great thing about mock is that it gets all the build depends
> info straight from the spec file.

All deps are already installed on our build servers. If not, we install
them before building a new package.

> The EL-5 package built happily under mock for -r epel-5-i386 too. I

This is great testing. Appreciated, thanks!

> Thanks so much. This is great and I really hope you got some use out

You are welcome. As I wrote a few mins ago, I really was inspired from
your work.

Regards,

---

From: Devrim GÜNDÜZ <[email-masked]>
Date: Wed, 20 Jan 2016 22:18:35 +0200

Hi,

> https://github.com/2ndQuadrant/pgdg-pgrpms/commit/60524156129b27b9cf3

Oh, why do we need -devel packages for postgresql95-devel package?

Regards,

---

From: Devrim GÜNDÜZ <[email-masked]>
Date: Wed, 20 Jan 2016 22:18:55 +0200

Hi Jeff,

On Tue, 2016-01-19 at 20:33 -0800, Jeff Frost wrote:
> This makes me so excited! I can't wait to try it and work on getting
> the other packages set up this way as well!
> 
> Great job you guys!

Thanks!

Regards,

---

From: Craig Ringer <[email-masked]>
Date: Thu, 21 Jan 2016 09:15:33 +0800

> Oh, why do we need -devel packages for postgresql95-devel package?
So that when you install the postgresql95-devel package it's reliably
usable.

If we don't depend on gettext-devel etc, then where PostgreSQL uses those
headers in its own headers you'll get errors at compile time unless they
happen to be installed because of the requirements of other unrelated
packages.

---

From: Craig Ringer <[email-masked]>
Date: Thu, 21 Jan 2016 09:21:14 +0800

> environment everytime would slow us down, IMHO. What do you think?

mock caches the build env so in practice it's very fast. mockchain helps
even more.

mock is what Koji uses and what's used to build all of Fedora and EPEL, so
it gets a fair bit of performance attention. There _is_ a performance cost
vs building bare on a preconfigured build box but it's (IMO) greatly offset
by the increased reliability and consistency of package builds. You don't
risk having undeclared dependencies, building against a version other than
the one declared in the spec file, etc.

> *All* of the *recently added* spec files are unified nowadays :)

Great!

> them before building a new package.

That can cause issues though. For example I think that's how the Java 8
problems a while ago happened. The spec file declared a dependency on
openjdk 7 but the rpms were built against what was locally installed, which
is openjdk 8.

The nice thing with mock is that the buildroot can contain only exactly the
required packages for a build. No extras. It provides a lot more confidence
that the spec matches the actual build and that anyone can reproduce the
package build consistently without a special hand-created build env.

I've been using it for the BDR and pglogical builds and it does a pretty
good job. It's helped me find a number of bugs and oversights too.