Subject: RPM Morgue

From: Craig Ringer <[email-masked]>
Date: Mon, 25 Jun 2018 09:42:28 +0800

Devrim, team:

The apt.postgresql.org crew have a package morgue for old versions at
atalia.postgresql.org/morgue/ . It's not a full repo, but you can fish out
needed packages manually and install them. This is a *lifesaver* when
trying to examine a core file a customer system where debuginfo wasn't
installed, or trying to reproduce a subtle version-specific issue.

Do you have anything like that? If not, do you have any interest in it? I'd
really like to get something like it going, and rather than creating one
in-house at 2ndQ where nobody else lands up benefiting, it might make sense
to help out with one for the community.

What I'm thinking of is a selective rsync to an archive repo, where
packages get copied but never get deleted, then createrepo_c runs in
incremental mode (--update --retain-old-md-by-age=5d) to index them.

So it doesn't upset https://yum.postgresql.org/, but
https://yum-archive.postgresql.org/ or whatever can keep a deep history of
packages.

An alternative is to ditch the repo indexing and use an AWS S3 bucket to
host an unindexed package slush pile, like the apt morgue. createrepo can't
run sensibly on s3-hosted files. But s3 is cheap - 2c/gb/month, or less if
you go for infrequent access mode.

Thoughts?

I can always set up my own mirror inhouse for 2ndQ, but I'd rather work
with you.

-- 
 Craig Ringer                   http://www.2ndQuadrant.com/
 PostgreSQL Development, 24x7 Support, Training & Services

---

From: Christoph Berg <[email-masked]>
Date: Mon, 25 Jun 2018 09:27:53 +0200

Re: Craig Ringer 2018-06-25 <[email-masked]>
> installed, or trying to reproduce a subtle version-specific issue.

Fwiw, the reason I set it up is because our repo software (reprepro)
doesn't support keeping the past N versions of every package around.
The yum repo keeps the last 3 versions in place, I believe. But of
course that doesn't help if $customer didn't upgrade for $long.

Christoph

---

From: Devrim Gündüz <[email-masked]>
Date: Wed, 27 Jun 2018 00:40:54 +0100

Hi Craig,

> you go for infrequent access mode.

I'm not against keeping the old package, but there are two things:

* We need to ask sysadmins team, because this means a lot of extra disk space.

* Building older packages is not that hard with the RPMs as you know -- just
change the version number, and run make rpm11 (or whatever). I'm not sure that
keeping the old packages are worth the hassle.

Again, I'm not against the idea as long as you can convince sysadmin team, and
also write scripts to pull the packages :)

Cheers,
-- 
Devrim Gündüz
EnterpriseDB: https://www.enterprisedb.com
PostgreSQL Consultant, Red Hat Certified Engineer
Twitter: @DevrimGunduz , @DevrimGunduzTR

---

From: Craig Ringer <[email-masked]>
Date: Wed, 27 Jun 2018 10:49:41 +0800

> * We need to ask sysadmins team, because this means a lot of extra disk

I'm not averse to hosting elsewhere, I just know Pg infra likes to keep
control where they can.

> keeping the old packages are worth the hassle.

Unfortunately that's not useful. You need *exactly matching* debuginfo
packages for the same build-IDs. You can't just rebuild the suite and
extract the debuginfo packages to install on another host.

---

From: Christoph Berg <[email-masked]>
Date: Wed, 27 Jun 2018 22:35:12 +0200

Re: Devrim Gündüz 2018-06-27 <[email-masked]>
> * We need to ask sysadmins team, because this means a lot of extra disk space.

The apt morgue is 74 GB atm. It's not mirrored anywhere - which isn't
ideal, but the at means we only need to keep an eye on the local disk
space, and not on N mirror hosts which all might have slightly
different limits.

The initial discussion with the sysadmins was that everyone kind of
agreed that the service is useful, but we couldn't settle for any
definition of what resources should be allocated for the next N years.
So in the end I just put it somewhere, and that's where it's living
now. (I don't blame them for that, it's difficult to work with
statements like "this could be any size, because we don't know how
many packages there will be in 3 years".)

Christoph

---

From: GOLLET Nicolas <[email-masked]>
Date: Thu, 28 Jun 2018 13:14:52 +0200

Hi Craig, I had also thought that setting up a public morgue would be a good thing, I recently set up an unofficial public morgue (WIP): http://pgyum-morgue.ng.pe/ (WIP) I'm missing some RPM, if you want us to work together on setting it up I'm available;) I had thought not to put all the packages in the morgue but the most important ones: - postgresql (RPMs but also an archive containing all RPMs (libs, contrib, debuginfo, ...) by version (WIP). - slony - postgis ... ? Classified by architecture and RHLE version... See you soon! ++ Nicolas ---- On mer., 27 juin 2018 01:40:54 +0200 Devrim Gündüz <[email-masked]> wrote ---- Hi Craig, On Mon, 2018-06-25 at 09:42 +0800, Craig Ringer wrote: > The apt.postgresql.org crew have a package morgue for old versions at > atalia.postgresql.org/morgue/ . It's not a full repo, but you can fish out > needed packages manually and install them. This is a *lifesaver* when > trying to examine a core file a customer system where debuginfo wasn't > installed, or trying to reproduce a subtle version-specific issue. > > Do you have anything like that? If not, do you have any interest in it? I'd > really like to get something like it going, and rather than creating one > in-house at 2ndQ where nobody else lands up benefiting, it might make sense > to help out with one for the community. > > What I'm thinking of is a selective rsync to an archive repo, where > packages get copied but never get deleted, then createrepo_c runs in > incremental mode (--update --retain-old-md-by-age=5d) to index them. > > So it doesn't upset https://yum.postgresql.org/, but > https://yum-archive.postgresql.org/ or whatever can keep a deep history of > packages. > > An alternative is to ditch the repo indexing and use an AWS S3 bucket to > host an unindexed package slush pile, like the apt morgue. createrepo can't > run sensibly on s3-hosted files. But s3 is cheap - 2c/gb/month, or less if > you go for infrequent access mode. > > Thoughts? I'm not against keeping the old package, but there are two things: * We need to ask sysadmins team, because this means a lot of extra disk space. * Building older packages is not that hard with the RPMs as you know -- just change the version number, and run make rpm11 (or whatever). I'm not sure that keeping the old packages are worth the hassle. Again, I'm not against the idea as long as you can convince sysadmin team, and also write scripts to pull the packages :) Cheers, -- Devrim Gündüz EnterpriseDB: https://www.enterprisedb.com PostgreSQL Consultant, Red Hat Certified Engineer Twitter: @DevrimGunduz , @DevrimGunduzTR

---

From: Christoph Berg <[email-masked]>
Date: Thu, 28 Jun 2018 13:31:30 +0200

Re: GOLLET Nicolas 2018-06-28 <[email-masked]>
> I had also thought that setting up a public morgue would be a good thing, I
> recently set up an unofficial public morgue (WIP): http://pgyum-morgue.ng.pe/
> (WIP)
> 
> I'm missing some RPM, if you want us to work together on setting it up I'm
> available;)
> I had thought not to put all the packages in the morgue but the most important
> ones:

Why not include everything? Filtering is extra effort, and even if you
get the filters right, they will soon become outdated.

Christoph

---

From: Craig Ringer <[email-masked]>
Date: Fri, 29 Jun 2018 09:38:37 +0800

> get the filters right, they will soon become outdated.
Right. Also, you're most likely to need postgres debuginfo, which is also
the biggest single package.