public inbox for [email protected]
help / color / mirror / Atom feedFrom: Jonathan S. Katz <[email protected]>
To: Tom Lane <[email protected]>
Cc: [email protected]
Subject: Re: Release note trimming: another modest proposal
Date: Mon, 6 Aug 2018 17:17:41 -0400
Message-ID: <[email protected]> (raw)
In-Reply-To: <[email protected]>
References: <[email protected]>
<[email protected]>
<[email protected]>
<[email protected]>
<[email protected]>
<[email protected]>
<[email protected]>
<[email protected]>
<[email protected]>
<[email protected]>
<[email protected]>
<[email protected]>
<[email protected]>
<[email protected]>
> On Aug 6, 2018, at 3:37 PM, Tom Lane <[email protected]> wrote:
>
> "Jonathan S. Katz" <[email protected]> writes:
>>> On Aug 6, 2018, at 3:27 PM, Tom Lane <[email protected]> wrote:
>>> Actually, a concrete reason why that might not be good is that it results
>>> in having a single point of failure: once we remove branch N's relnotes
>>> from the active branches, the only copy of that data is the one in the
>>> archive table the docload script is filling. Given, say, a bug in the
>>> docload script that causes it to overwrite the wrong table entries,
>>> can we recover?
>
>> Well, the release notes are still in the git history as well as the tarballs.
>> One could always pull an older tarball of PostgreSQL with the full
>> release.sgml and load from there.
>
> True ... as long as those older tarballs represent data that our current
> workflow can process. For instance, if we did another documentation
> format change (from XML to something else), the older tarballs would
> perhaps no longer be useful for this purpose.
>
> On the other hand, it's hard to believe that we'd make such a conversion
> without tools to help. So probably if the situation came up, we could
> cobble together something that would allow ingesting the old format.
Attached is a (rough) working copy of the patch to pgweb. It can:
- Extract the release notes from the docload and puts them into their
own table
- Display the release notes via pgweb akin to earlier screenshots
It needs:
- The notes actually exposed in the navigation tree
- Review how some of the xrefs are translated (esp. non-release ones)
- Dependency on all major versions being cataloged in our “Version”
table on pgweb, which currently we do not do
- Magnus review, as to do this I introduced a new Python dependency
I was able to successfully load all of the release notes from the 10.4
tarball and spot checked view several different major/minor version
combinations.
It’s not near production ready, but wanted to demonstrate that it would
not be too hard to get this done.
Jonathan
Attachments:
[application/octet-stream] release-notes.patch (7.9K, 2-release-notes.patch)
download | inline diff:
diff --git a/pgweb/docs/migrations/0004_auto_20180806_1917.py b/pgweb/docs/migrations/0004_auto_20180806_1917.py
new file mode 100644
index 0000000..5d67c29
--- /dev/null
+++ b/pgweb/docs/migrations/0004_auto_20180806_1917.py
@@ -0,0 +1,31 @@
+# -*- coding: utf-8 -*-
+# Generated by Django 1.11.13 on 2018-08-06 19:17
+from __future__ import unicode_literals
+
+from django.db import migrations, models
+import django.db.models.deletion
+
+
+class Migration(migrations.Migration):
+
+ dependencies = [
+ ('core', '0001_initial'),
+ ('docs', '0003_docs_alias'),
+ ]
+
+ operations = [
+ migrations.CreateModel(
+ name='ReleaseNote',
+ fields=[
+ ('id', models.AutoField(auto_created=True, primary_key=True, serialize=False, verbose_name='ID')),
+ ('version', models.ForeignKey(on_delete=django.db.models.deletion.CASCADE, to='core.Version', db_column='version', to_field=b'tree')),
+ ('minor_version', models.IntegerField()),
+ ('title', models.CharField(max_length=256)),
+ ('content', models.TextField()),
+ ],
+ ),
+ migrations.AlterUniqueTogether(
+ name='releasenote',
+ unique_together=set([('version', 'minor_version')]),
+ ),
+ ]
diff --git a/pgweb/docs/models.py b/pgweb/docs/models.py
index a2754b6..73f46fb 100644
--- a/pgweb/docs/models.py
+++ b/pgweb/docs/models.py
@@ -31,3 +31,13 @@ class DocPageAlias(models.Model):
class Meta:
db_table = 'docsalias'
verbose_name_plural='Doc page aliases'
+
+class ReleaseNote(models.Model):
+ """Contains content for a release note"""
+ version = models.ForeignKey(Version, db_column='version', to_field='tree')
+ minor_version = models.IntegerField()
+ title = models.CharField(max_length=256)
+ content = models.TextField()
+
+ class Meta:
+ unique_together = [('version', 'minor_version')]
diff --git a/pgweb/docs/views.py b/pgweb/docs/views.py
index 7cc1692..49e3a60 100644
--- a/pgweb/docs/views.py
+++ b/pgweb/docs/views.py
@@ -14,7 +14,7 @@ from pgweb.util.misc import send_template_mail
from pgweb.core.models import Version
-from models import DocPage
+from models import DocPage, ReleaseNote
from forms import DocCommentForm
def docpage(request, version, typ, filename):
@@ -81,6 +81,21 @@ def docsrootpage(request, version, typ):
def redirect_root(request, version):
return HttpResponseRedirect("/docs/%s/static/" % version)
+def releasenotesarchive(request, *args):
+ """Have a page available to view release notes"""
+ # If the version is greater than 10, only use the first two arguments
+ if int(args[0]) >= 10:
+ major_version = int(args[0])
+ minor_version = int(args[2]) if args[2] else 0
+ else:
+ major_version = "%s.%s" % (args[0], args[2])
+ minor_version = int(args[4]) if args[4] else 0
+ page = get_object_or_404(ReleaseNote, version=major_version, minor_version=minor_version)
+ return render_pgweb(request, 'docs', 'docs/release-notes.html', {
+ 'page': page,
+ 'title': page.title,
+ })
+
def root(request):
versions = Version.objects.filter(Q(supported=True) | Q(testing__gt=0,tree__gt=0)).order_by('-tree')
return render_pgweb(request, 'docs', 'docs/index.html', {
diff --git a/pgweb/urls.py b/pgweb/urls.py
index 64caf1e..5d8748f 100644
--- a/pgweb/urls.py
+++ b/pgweb/urls.py
@@ -57,6 +57,7 @@ urlpatterns = [
url(r'^docs/$', pgweb.docs.views.root),
url(r'^docs/manuals/$', pgweb.docs.views.manuals),
url(r'^docs/manuals/archive/$', pgweb.docs.views.manualarchive),
+ url(r'^docs/release-notes/(\d+)(.(\d+)(.(\d+))?)?/$', pgweb.docs.views.releasenotesarchive),
url(r'^docs/(current|devel|\d+(?:\.\d)?)/(static|interactive)/(.*).html?$', pgweb.docs.views.docpage),
url(r'^docs/(current|devel|\d+(?:\.\d)?)/(static|interactive)/$', pgweb.docs.views.docsrootpage),
url(r'^docs/(current|devel|\d+(?:\.\d)?)/$', pgweb.docs.views.redirect_root),
diff --git a/templates/docs/release-notes.html b/templates/docs/release-notes.html
new file mode 100644
index 0000000..4b9096a
--- /dev/null
+++ b/templates/docs/release-notes.html
@@ -0,0 +1,7 @@
+{%extends "base/page.html"%}
+{%block title%}Release Notes - {{page.title}}{%endblock%}
+{%block contents%}
+ <div id="docContent">
+ {{ page.content|safe }}
+ </div>
+{% endblock contents %}
diff --git a/tools/docs/docload.py b/tools/docs/docload.py
index 2965f9c..5a6d178 100755
--- a/tools/docs/docload.py
+++ b/tools/docs/docload.py
@@ -10,6 +10,7 @@ import tidy
from optparse import OptionParser
from ConfigParser import ConfigParser
+import bs4
import psycopg2
pagecount = 0
@@ -62,9 +63,66 @@ def load_doc_file(filename, f):
't': title,
'c': str(s),
})
+ # If this is a release note, load the release note.
+ if filename.startswith('release-'):
+ load_release_note(filename, str(s))
global pagecount
pagecount += 1
+def load_release_note(filename, content):
+ """Load a release note into the system based on the filename"""
+ if not quiet: print "--- release note: %s" % filename
+ # Format the content for display in the release note page
+ parser = bs4.BeautifulSoup(content, "html.parser")
+ # Get the version that this release document is referencing
+ release_version = parser.find(class_="sect1")["id"].split("RELEASE-")[-1].replace('-', '.')
+ # Extract the versions
+ if float(release_version.split('.')[0]) >= 10 or float(release_version.split('.')[0]) <= 1:
+ try:
+ major_version, minor_version = release_version.split('.')
+ except ValueError:
+ major_version, minor_version = release_version, 0
+ else:
+ m = re.search(r'(\d+).(\d+)(.(\d+))?', release_version)
+ major_version = "%s.%s" % (m.groups()[0], m.groups()[1])
+ minor_version = m.groups()[3] if m.groups()[3] else 0
+ # Remove extraneous sections of the release notes
+ for class_name in ['navheader', 'navfooter', 'toc']:
+ tag = parser.find(class_=class_name)
+ if tag:
+ tag.decompose()
+ # Update the headers to the proper release version, and remove extra references
+ # from the documentation
+ release_title = "Release " + release_version
+ # For really old release of PostgreSQL there is a nested product name
+ tag = parser.find('h2', class_="title")
+ if not tag.string:
+ tag.replace_with('<h2 class="title" style="clear: both">%s</h2>' % release_title)
+ else:
+ tag.string.replace_with(release_title)
+ for tag in parser.find_all('h3', class_='title'):
+ if not tag.string: continue
+ m = re.search(r'(([A-Z0-9]+\.)+).(.*)$', tag.string)
+ if not m: continue
+ tag.string.replace_with(m.groups()[-1])
+ # Update the URLs to point to the release note archives
+ for tag in parser.find_all(class_="xref"):
+ release = tag['href'].split('.')[0].split("release-")[-1]
+ title = 'Release ' + ".".join(release.split('-'))
+ tag['href'] = "/docs/release-notes/archive/" + tag['href']
+ tag['title'] = title
+ tag.string.replace_with(title)
+ # Update the documentation links to point to the current docs.
+ for tag in parser.find_all(class_="link"):
+ tag['href'] = "/docs/current/static/" + tag['href']
+ # The content is now ready to be loaded into the database
+ curs.execute("""
+ INSERT INTO docs_releasenote (version, minor_version, title, content)
+ VALUES (%(v)s, %(m)s, %(t)s, %(c)s)
+ ON CONFLICT (version, minor_version) DO UPDATE
+ SET title = EXCLUDED.title, content = EXCLUDED.content""",
+ { 'v': major_version, 'm': minor_version, 't': release_title, 'c': parser.prettify() })
+
## Main execution
parser = OptionParser(usage="usage: %prog [options] <version> <tarfile>")
@@ -110,7 +168,7 @@ re_htmlfile = re.compile('[^/]*/doc/src/sgml/html/.*')
re_tarfile = re.compile('[^/]*/doc/postgres.tar.gz$')
for member in tf:
if re_htmlfile.match(member.name):
- load_doc_file(os.path.basename(member.name), tf.extractfile(member))
+ load_doc_file(filename, tf.extractfile(member))
if re_tarfile.match(member.name):
f = tf.extractfile(member)
inner_tar = tarfile.open(fileobj=f)
@@ -139,4 +197,3 @@ connection.commit()
connection.close()
if not quiet: print "Done (%i pages)." % pagecount
-
[application/pgp-signature] signature.asc (833B, 3-signature.asc)
download
view thread (58+ messages) latest in thread
reply
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Reply to all the recipients using the --to and --cc options:
reply via email
To: [email protected]
Cc: [email protected], [email protected], [email protected]
Subject: Re: Release note trimming: another modest proposal
In-Reply-To: <[email protected]>
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
This inbox is served by agora; see mirroring instructions
for how to clone and mirror all data and code used for this inbox