Subject: programmatic way to fetch latest release for a given major.minor version

From: "Andrew Hammond" <[email-masked]>
Date: 9 Apr 2007 14:47:20 -0700

I'm writing a script that wants to know the latest release for a given
major.minor version. Is there some better way than parsing
http://www.postgresql.org/ftp/source/ or trying to connect to ftp
(which is invariably timing out on me today. Is that box getting
hammered or something?) and doing the parsing that? Both approaches
feel quite awkward to me.

Andrew

---

From: "CAJ CAJ" <[email-masked]>
Date: Mon, 9 Apr 2007 15:26:12 -0700

> feel quite awkward to me.

Use wget to download via  HTTP (added recently).  Probably wise to add a
couple mirrors in your script.

---

From: "Andrew Hammond" <[email-masked]>
Date: Mon, 9 Apr 2007 15:34:02 -0700

On 4/9/07, CAJ CAJ <[email-masked]> wrote:
> On 9 Apr 2007 14:47:20 -0700, Andrew Hammond
> <[email-masked]> wrote:

I'm not asking how to download stuff. I'm asking how to figure out the
current release number for a given major.minor. I thought that was
clear in my original post, but I guess not. For example, how do I
determine (programmatically) the lastest version of 8.1.

I'm also interested in a clever way to select one of the "close"
mirrors at random for downloading via http. However I had planned to
hold that question until I'd solved the first issue.

Andrew

---

From: Alvaro Herrera <[email-masked]>
Date: Mon, 9 Apr 2007 18:54:45 -0400

> determine (programmatically) the lastest version of 8.1.

Maybe get configure.in from CVS:
cvs log configure.in
and parse the "symbolic names" list?

-- 
Alvaro Herrera                                http://www.CommandPrompt.com/
PostgreSQL Replication, Consulting, Custom Development, 24x7 support

---

From: Listmail <[email-masked]>
Date: Tue, 10 Apr 2007 01:36:02 +0200

Here you go.

	Fetches versions and prints most recent minor for each major
	Tests all mirrors for speed and prints out the 4 fastest (takes some time)
	http://www.crummy.com/software/BeautifulSoup/

	Have a nice day !

#! /bin/env python
# -*- coding: utf-8  -*-

import urllib, BeautifulSoup, re, time, sys

def get_all_versions():
	soup =  
BeautifulSoup.BeautifulSoup( urllib.urlopen( "http://ftp3.fr.postgresql.org/pub/postgresql/source/"  
).read() )
	for a in soup( 'a', {'href': re.compile( r"v\d+.\d+.\d+" ) } ):
		yield map( int, re.search( r"v(\d+)\.(\d+)\.(\d+)*", a['href']  
).groups() )

def get_latest_versions():
	lastversions = {}
	for a,b,c in sorted( get_all_versions() ):
		lastversions[ (a,b) ] = c
	return sorted( lastversions.items() )

def parse_query_string( url ):
	return dict( map( urllib.unquote_plus, pair.split('=',1) ) for pair in  
re.split( "&(?:amp;|)", urllib.splitquery( url )[1] ) )

def get_mirrors():
	soup =  
BeautifulSoup.BeautifulSoup( urllib.urlopen( "http://wwwmaster.postgresql.org/download/mirrors-ftp"  
).read() )
	for a in soup( 'a', {'href': re.compile( r"\?setmir=.*url=" ) } ):
		yield parse_query_string( a['href'] )['url']

def get_fastest_mirrors( urls, filename ):
	for url in urls:
		sys.stdout.write( "		%s\r" % url )
		t = time.time()
		try:
			urllib.urlopen( url + filename )
		except:
			pass
		d = time.time()-t
		print "%.02f s" % d
		yield d, url

for major, minor in get_latest_versions():
	print "%d.%d.%d" % (major[0], major[1], minor)

mirrors = get_mirrors()
fastest = sorted( get_fastest_mirrors( mirrors, "sync_timestamp" ))[:4]
for d, mirror in fastest:
	print "%.02f s	%s" % (d,mirror)

On Tue, 10 Apr 2007 00:34:02 +0200, Andrew Hammond  
<[email-masked]> wrote:

> ---------------------------(end of broadcast)---------------------------
> TIP 3: Have you checked our extensive FAQ?
>
>                http://www.postgresql.org/docs/faq

---

From: Dave Page <[email-masked]>
Date: Tue, 10 Apr 2007 08:36:10 +0100

> determine (programmatically) the lastest version of 8.1.

Currently you can't without parsing pages.

> hold that question until I'd solved the first issue.

You might find this of use: http://www.postgresql.org/mirrors.xml. It's
updated regularly, so the mirrors should all have test OK within the
last 48 hours.

Regards, Dave.

---

From: Magnus Hagander <[email-masked]>
Date: Tue, 10 Apr 2007 10:15:58 +0200

> Currently you can't without parsing pages.

I've been meaning to do this for a while now, and I guess this might be a
good motivator ;-) The idea would, of course, be to build this from a set of
common data with the frontpage of the website, so they're always in sync.
As I see it, there are basically three ways of doing it:

1) Fold it into mirrors.xml. That's going to be a bit hard given how the
schema is (since the root element is mirror), but it would work. Another
downside is that is that people who just want to check the latest version
will have to pull down the whole mirror list file. This file is also
rewritten often, so checks on If-Modified-Since will fail.

2) Create a new file with a specific schema. Something like:
<version>
 <version major="8.2" minor="8.2.3" />
 <version major="8.1" minor="8.1.8" />
</version>
This is the most lightweight solution.

3) Use RSS format. Not exactly sure how it would look, but it can certainly
be done in RSS...

I think I prefer option 2, but what do others think?

//Magnus

---

From: Dave Page <[email-masked]>
Date: Tue, 10 Apr 2007 09:52:52 +0100

> This is the most lightweight solution.

More like:

<versions>
  <version major="8" minor="2" revision="3" />
  <version major="8" minor="1" revision="8" />
</versions>

But I can't help thinking that we should have some additional values for
release notes, download sub-URLs (to be appended to the mirror roots) etc.

Regards, Dave.

---

From: Listmail <[email-masked]>
Date: Tue, 10 Apr 2007 11:09:50 +0200

I love Open Source XD

	http://ethan.tira-thompson.com/cvslog2web/

	Note that this is overkill (but it would look SEXY on the site).
	However, the original poster probably wants to know when to update his  
servers, so he won't care about CVS commits...
	If there was a RSS feed of new postgres versions I'd subscribe to it  
though.

---

From: Magnus Hagander <[email-masked]>
Date: Tue, 10 Apr 2007 12:18:52 +0200

>   <version major="8" minor="1" revision="8" />

But that doesn't reflect our terminology. Per our own terminology, bioth
8.1 and 8.2 are major versions. Not only is that what we've been saying for
years, we even documented it at
http://www.postgresql.org/support/versioning.

> release notes, download sub-URLs (to be appended to the mirror roots) etc.

Yeah, thus the X in XML :-) 
But given that we might want to add things like this, having our own custom
XML format certainly makes a bit more sense, it might be harder to try to
trick it into RSS.

//Magnus

---

From: Magnus Hagander <[email-masked]>
Date: Tue, 10 Apr 2007 12:20:45 +0200

> 	http://ethan.tira-thompson.com/cvslog2web/

Hmm. Don't tell people about my secret plans :)

(Though I hadn't looked at that piece of software in particylar)

> 	However, the original poster probably wants to know when to update 
> 	his  servers, so he won't care about CVS commits...

That's the argument for RSS - you acn subscribe to it from standard RSS
readers.
Perhaps we need to do both...

//Magnus

---

From: Dave Page <[email-masked]>
Date: Tue, 10 Apr 2007 12:03:44 +0100

> http://www.postgresql.org/support/versioning.

Yeah yeah, but terminology aside, having 2 or three digits in each
attribute is just wrong!

> Yeah, thus the X in XML :-) 

:-p

> XML format certainly makes a bit more sense, it might be harder to try to

Agreed.

/D

---

From: Magnus Hagander <[email-masked]>
Date: Tue, 10 Apr 2007 13:15:09 +0200

> Yeah yeah, but terminology aside, having 2 or three digits in each

Terminology aside, why? The unit is "8.1" not "8" and "1". It makes no
sense to say you're on version 8, in the given context, so why should the
XML data pretend there is?

//Magnus

---

From: Dave Page <[email-masked]>
Date: Tue, 10 Apr 2007 12:35:38 +0100

> XML data pretend there is?

Because serving the data in the decomposed format gives the consumer the
maximum flexibility to do as they wish with the data. I find it hard to
see why, as a relational database guy, you'd want to offer the data as
"8.1", "8.1.3" etc. when you can just give the three parts separately
and allow people to check whatever they need without having to chop up
strings!

Imagine wanting to display only the details of the 8.x releases on a
site for example. In your schema, you'd have to use a substring match on
an attribute value to filter out 6.x and 7.x.

Regards,D ave.

---

From: Listmail <[email-masked]>
Date: Tue, 10 Apr 2007 13:44:43 +0200

> XML data pretend there is?

	Just pretend that :

	- version = a tuple of integers (a, b, c, ...)
	- major = (a, b)
	- minor = (c, ...)

	Besides, that is sortable (unlike strings where 15 < 2) :

	latest minor for major :
	major, max(minor) where major = what you want

<pgversion><major><int value="8" /><int value="2" /></major><minor><int  
value="3" /></minor></pgversion>

 from BeautifulSoup import BeautifulSoup as Soup
s = Soup("""<pgversion><major><int value="8" /><int value="2"  
/></major><minor><int value="3" /></minor></pgversion>""" )

>>> v = s.find('pgversion')
>>> [int(x['value']) for x in v.find('major') ]
[8, 2]
>>> [int(x['value']) for x in v.find('minor') ]
[3]

---

From: Magnus Hagander <[email-masked]>
Date: Tue, 10 Apr 2007 14:27:08 +0200

> an attribute value to filter out 6.x and 7.x.

That is actually precisely my point. It makes *no sense* to filter based on
8.x. 8.0 is no more a major release than 7.4. And we'd encourage people to
do that.

//Magnus

---

From: Martijn van Oosterhout <[email-masked]>
Date: Tue, 10 Apr 2007 14:44:42 +0200

> XML format certainly makes a bit more sense, it might be harder to try to

I'd say do it in the format you're most comfortable with. Then some
XSLT wizard can convert it into RSS from there...

Have a nice day,
-- 
Martijn van Oosterhout   <[email-masked]>   http://svana.org/kleptog/
> From each according to his ability. To each according to his ability to litigate.

---

From: Magnus Hagander <[email-masked]>
Date: Tue, 10 Apr 2007 14:54:55 +0200

> XSLT wizard can convert it into RSS from there...

Too late, I already did the RSS. Which was very simple, because of the RSS
framework stuff already in place on the website.

*However*, that one doesn't contain all the information that a program
would want, and certainly not in an easily parseable format for them. It's
designed for people in the other end. So I intend to do a "proper XML
format" as well.

//Magnus

---

From: Peter Eisentraut <[email-masked]>
Date: Tue, 10 Apr 2007 14:59:12 +0200

> an attribute value to filter out 6.x and 7.x.

That use case is just as valid as wanting to show only releases >= 7.4.  The 
user could run an arithmetic expression to check for that and >= 8.0.

-- 
Peter Eisentraut
http://developer.postgresql.org/~petere/

---

From: Dave Page <[email-masked]>
Date: Tue, 10 Apr 2007 14:13:04 +0100

Magnus Hagander wrote:
> 8.x. 8.0 is no more a major release than 7.4. 

Yes it is - that's precisely why it was 8.0 and not 7.5.

/D

---

From: Alvaro Herrera <[email-masked]>
Date: Tue, 10 Apr 2007 09:57:08 -0400

> Yes it is - that's precisely why it was 8.0 and not 7.5.

That was merely a marketing artifact; it was called 7.5 until the very
end of the devel cycle.

-- 
Alvaro Herrera                                http://www.CommandPrompt.com/
The PostgreSQL Company - Command Prompt, Inc.

---

From: Dave Page <[email-masked]>
Date: Tue, 10 Apr 2007 15:12:31 +0100

> That was merely a marketing artifact; it was called 7.5 until the very

Yes, but marketing is one reason why someone might want to group 8.x,
7.x etc on a website which is exactly the sort of thing this code is for.

As others have said, yes, you could do it but looking at a substring of
the version, and yes, you could do it with mathematical comparisons on
major.minor (with limitations - what happens if we get to 8.10 ?), but
would we suggest people use those techniques for searching their
databases for matching records, or would we suggest storing the
interesting elements in different columns for ease of use, flexibility
and efficiency? How does this differ (aside from the obvious)?

Regards, Dave

---

From: Alvaro Herrera <[email-masked]>
Date: Tue, 10 Apr 2007 10:25:35 -0400

> 7.x etc on a website which is exactly the sort of thing this code is for.

Ah, but then it's not an decision to be made on arithmetics alone -- you
have to build a higher-level semantic comparison.  Because if you want
to group by something else, for example the quality of Windows support,
you surely don't want 8.0 nor 8.1, because they have unfixable problems
(the pgstat bug, autovacuum not working).  You need to include only 8.2
and higher.

> and efficiency? How does this differ (aside from the obvious)?

It makes sense to store things separately when they have a semantic
difference.  What we call "major" is the first two digits and dot.  We
call "minor" to the third digit, and that's all.  We don't have
"revisions".  This is how it has ever been and we even document it as
such.  Offering the first two digits separately would be a mistake
because it causes confusion over what's significant -- the first digit
by itself is not significant.

---

From: Tom Lane <[email-masked]>
Date: Tue, 10 Apr 2007 10:42:07 -0400

> by itself is not significant.

While I agree with this position, ISTM that an easy compromise is
available: put both formats into the data.

			regards, tom lane

---

From: Dave Page <[email-masked]>
Date: Tue, 10 Apr 2007 15:58:12 +0100

> by itself is not significant.

Yes, and for the most part it's true (certainly as far as the purpose of
that documentation is concerned), however it all falls apart next time
we have the 'are there enough new features in this release to bump the
first digit' discussion which, along with the fact that we have bumped
it twice now, is proof in itself that we do give it some meaning.

Anyhoo, I've said my piece now - I'll await the inevitable replies
telling me I'm a donut and then move on :-)

Regards, Dave

---

From: Jorge Godoy <[email-masked]>
Date: Tue, 10 Apr 2007 13:48:08 -0300

> 	Besides, that is sortable (unlike strings where 15 < 2) :

But then, floats are as sortable as integers and 8.3 < 15.1...

-- 
Jorge Godoy      <[email-masked]>

---

From: Peter Wilson <[email-masked]>
Date: Tue, 10 Apr 2007 18:00:59 +0100

> But then, floats are as sortable as integers and 8.3 < 15.1...

8.7, 8.9, 8.10 - oops. 8.10 < 8.9

The 'period' is effectively a field separator, with the unfortunate side-effect
that it looks like a number.

Unless of course, the version after 8.9 *will* be 9.0, in which case it is a
number and you're right.

Pete

---

From: Magnus Hagander <[email-masked]>
Date: Wed, 11 Apr 2007 18:20:46 +0200

> >   <version major="8" minor="1" revision="8" />

Ok, I've added one of these as well now, alongside the RSS feed. You can
get the pg specific XML file at:
http://www.postgresql.org/versions.xml

If someone wants the schema change, react *now*. Later on we can only
append to it, and not change it :)

//Magnus

---

From: Listmail <[email-masked]>
Date: Wed, 11 Apr 2007 18:49:18 +0200

> append to it, and not change it :)

	Since I like to complain...

<numericversion v1="8" v2="2" v3="3"/>

	Suppose you someday add another dot, or a "b" for beta, wouldn't it be  
better to have

<versionlist><item>8</item><item>2</item><item>3</item></versionlist>

	... or <item value="3" />

---

From: Magnus Hagander <[email-masked]>
Date: Wed, 11 Apr 2007 20:07:22 +0200

On Wed, Apr 11, 2007 at 06:49:18PM +0200, Listmail wrote:
> 	Suppose you someday add another dot, or a "b" for beta, wouldn't it 
> 	be  better to have
> 

IIRC, but not entirely sure, the order of items in XML is not guaranteed.
So you'd need something like 
<versionlist><item seq="1">8</item><item seq="2">2</item> etc etc

I'm not sure, but I have some kind of memory of that ;-)

As for beta, we're only going to be listing production versions in this
one. It's there to list the latest available version in each released
series.
And if we add another dot, we can just add a v4="7" attribute. Adding is
not a problem, only modifying.

//Magnus