Received: from malur.postgresql.org ([2a02:16a8:dc51::56]) by arkaria.postgresql.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_CBC_SHA384:256) (Exim 4.89) (envelope-from ) id 1fmmtM-0003Jh-Rs for pgsql-docs@arkaria.postgresql.org; Mon, 06 Aug 2018 21:18:17 +0000 Received: from localhost ([127.0.0.1] helo=malur.postgresql.org) by malur.postgresql.org with esmtp (Exim 4.89) (envelope-from ) id 1fmmtK-0004YP-Ua for pgsql-docs@arkaria.postgresql.org; Mon, 06 Aug 2018 21:18:14 +0000 Received: from magus.postgresql.org ([2a02:c0:301:0:ffff::29]) by malur.postgresql.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_CBC_SHA384:256) (Exim 4.89) (envelope-from ) id 1fmmtK-0004YI-NW for pgsql-docs@lists.postgresql.org; Mon, 06 Aug 2018 21:18:14 +0000 Received: from meldrar.postgresql.org ([2a02:c0:301:0:ffff::31]) by magus.postgresql.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_CBC_SHA384:256) (Exim 4.89) (envelope-from ) id 1fmmt3-000607-2o for pgsql-docs@lists.postgresql.org; Mon, 06 Aug 2018 21:18:13 +0000 Received: from [144.121.72.194] (helo=[172.16.7.107]) by meldrar.postgresql.org with esmtpsa (TLS1.2:ECDHE_RSA_AES_256_CBC_SHA384:256) (Exim 4.89) (envelope-from ) id 1fmmsq-00081A-TP; Mon, 06 Aug 2018 21:17:56 +0000 From: "Jonathan S. Katz" Message-Id: <9FBD8BD2-F2F3-4618-A304-DEE93007EDBB@postgresql.org> Content-Type: multipart/signed; boundary="Apple-Mail=_6266A028-AEBD-46E8-A46C-0B91F154DB6E"; protocol="application/pgp-signature"; micalg=pgp-sha256 Mime-Version: 1.0 (Mac OS X Mail 10.3 \(3273\)) Subject: Re: Release note trimming: another modest proposal Date: Mon, 6 Aug 2018 17:17:41 -0400 In-Reply-To: <1620.1533584265@sss.pgh.pa.us> Cc: pgsql-docs@lists.postgresql.org To: Tom Lane References: <19252.1533509841@sss.pgh.pa.us> <37D00E58-A0F0-42E4-83F1-A124A282575D@postgresql.org> <18020.1533568149@sss.pgh.pa.us> <04F6EF85-C7B7-42F3-84BC-D5670C9D77E1@postgresql.org> <19825.1533570442@sss.pgh.pa.us> <3070E803-B5D5-46DA-80E5-47B43BE9B085@postgresql.org> <23297.1533574521@sss.pgh.pa.us> <660.1533583153@sss.pgh.pa.us> <1083.1533583655@sss.pgh.pa.us> <1F8EA64B-695C-4893-A436-B655C840788D@postgresql.org> <1620.1533584265@sss.pgh.pa.us> X-Mailer: Apple Mail (2.3273) X-Host-Lookup-Failed: Reverse DNS lookup failed for 144.121.72.194 (failed) List-Id: List-Help: List-Subscribe: List-Post: List-Owner: List-Archive: Precedence: bulk --Apple-Mail=_6266A028-AEBD-46E8-A46C-0B91F154DB6E Content-Type: multipart/mixed; boundary="Apple-Mail=_974930D2-E7E1-4F7A-AA75-654E4D34D515" --Apple-Mail=_974930D2-E7E1-4F7A-AA75-654E4D34D515 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset=utf-8 > On Aug 6, 2018, at 3:37 PM, Tom Lane wrote: >=20 > "Jonathan S. Katz" writes: >>> On Aug 6, 2018, at 3:27 PM, Tom Lane wrote: >>> Actually, a concrete reason why that might not be good is that it = results >>> in having a single point of failure: once we remove branch N's = relnotes >>> from the active branches, the only copy of that data is the one in = the >>> archive table the docload script is filling. Given, say, a bug in = the >>> docload script that causes it to overwrite the wrong table entries, >>> can we recover? >=20 >> Well, the release notes are still in the git history as well as the = tarballs. >> One could always pull an older tarball of PostgreSQL with the full >> release.sgml and load from there. >=20 > True ... as long as those older tarballs represent data that our = current > workflow can process. For instance, if we did another documentation > format change (from XML to something else), the older tarballs would > perhaps no longer be useful for this purpose. >=20 > On the other hand, it's hard to believe that we'd make such a = conversion > without tools to help. So probably if the situation came up, we could > cobble together something that would allow ingesting the old format. Attached is a (rough) working copy of the patch to pgweb. It can: - Extract the release notes from the docload and puts them into their own table - Display the release notes via pgweb akin to earlier screenshots It needs: - The notes actually exposed in the navigation tree - Review how some of the xrefs are translated (esp. non-release ones) - Dependency on all major versions being cataloged in our =E2=80=9CVersion= =E2=80=9D table on pgweb, which currently we do not do - Magnus review, as to do this I introduced a new Python dependency I was able to successfully load all of the release notes from the 10.4 tarball and spot checked view several different major/minor version combinations. It=E2=80=99s not near production ready, but wanted to demonstrate that = it would not be too hard to get this done. Jonathan --Apple-Mail=_974930D2-E7E1-4F7A-AA75-654E4D34D515 Content-Disposition: attachment; filename=release-notes.patch Content-Type: application/octet-stream; x-unix-mode=0644; name="release-notes.patch" Content-Transfer-Encoding: 7bit diff --git a/pgweb/docs/migrations/0004_auto_20180806_1917.py b/pgweb/docs/migrations/0004_auto_20180806_1917.py new file mode 100644 index 0000000..5d67c29 --- /dev/null +++ b/pgweb/docs/migrations/0004_auto_20180806_1917.py @@ -0,0 +1,31 @@ +# -*- coding: utf-8 -*- +# Generated by Django 1.11.13 on 2018-08-06 19:17 +from __future__ import unicode_literals + +from django.db import migrations, models +import django.db.models.deletion + + +class Migration(migrations.Migration): + + dependencies = [ + ('core', '0001_initial'), + ('docs', '0003_docs_alias'), + ] + + operations = [ + migrations.CreateModel( + name='ReleaseNote', + fields=[ + ('id', models.AutoField(auto_created=True, primary_key=True, serialize=False, verbose_name='ID')), + ('version', models.ForeignKey(on_delete=django.db.models.deletion.CASCADE, to='core.Version', db_column='version', to_field=b'tree')), + ('minor_version', models.IntegerField()), + ('title', models.CharField(max_length=256)), + ('content', models.TextField()), + ], + ), + migrations.AlterUniqueTogether( + name='releasenote', + unique_together=set([('version', 'minor_version')]), + ), + ] diff --git a/pgweb/docs/models.py b/pgweb/docs/models.py index a2754b6..73f46fb 100644 --- a/pgweb/docs/models.py +++ b/pgweb/docs/models.py @@ -31,3 +31,13 @@ class DocPageAlias(models.Model): class Meta: db_table = 'docsalias' verbose_name_plural='Doc page aliases' + +class ReleaseNote(models.Model): + """Contains content for a release note""" + version = models.ForeignKey(Version, db_column='version', to_field='tree') + minor_version = models.IntegerField() + title = models.CharField(max_length=256) + content = models.TextField() + + class Meta: + unique_together = [('version', 'minor_version')] diff --git a/pgweb/docs/views.py b/pgweb/docs/views.py index 7cc1692..49e3a60 100644 --- a/pgweb/docs/views.py +++ b/pgweb/docs/views.py @@ -14,7 +14,7 @@ from pgweb.util.misc import send_template_mail from pgweb.core.models import Version -from models import DocPage +from models import DocPage, ReleaseNote from forms import DocCommentForm def docpage(request, version, typ, filename): @@ -81,6 +81,21 @@ def docsrootpage(request, version, typ): def redirect_root(request, version): return HttpResponseRedirect("/docs/%s/static/" % version) +def releasenotesarchive(request, *args): + """Have a page available to view release notes""" + # If the version is greater than 10, only use the first two arguments + if int(args[0]) >= 10: + major_version = int(args[0]) + minor_version = int(args[2]) if args[2] else 0 + else: + major_version = "%s.%s" % (args[0], args[2]) + minor_version = int(args[4]) if args[4] else 0 + page = get_object_or_404(ReleaseNote, version=major_version, minor_version=minor_version) + return render_pgweb(request, 'docs', 'docs/release-notes.html', { + 'page': page, + 'title': page.title, + }) + def root(request): versions = Version.objects.filter(Q(supported=True) | Q(testing__gt=0,tree__gt=0)).order_by('-tree') return render_pgweb(request, 'docs', 'docs/index.html', { diff --git a/pgweb/urls.py b/pgweb/urls.py index 64caf1e..5d8748f 100644 --- a/pgweb/urls.py +++ b/pgweb/urls.py @@ -57,6 +57,7 @@ urlpatterns = [ url(r'^docs/$', pgweb.docs.views.root), url(r'^docs/manuals/$', pgweb.docs.views.manuals), url(r'^docs/manuals/archive/$', pgweb.docs.views.manualarchive), + url(r'^docs/release-notes/(\d+)(.(\d+)(.(\d+))?)?/$', pgweb.docs.views.releasenotesarchive), url(r'^docs/(current|devel|\d+(?:\.\d)?)/(static|interactive)/(.*).html?$', pgweb.docs.views.docpage), url(r'^docs/(current|devel|\d+(?:\.\d)?)/(static|interactive)/$', pgweb.docs.views.docsrootpage), url(r'^docs/(current|devel|\d+(?:\.\d)?)/$', pgweb.docs.views.redirect_root), diff --git a/templates/docs/release-notes.html b/templates/docs/release-notes.html new file mode 100644 index 0000000..4b9096a --- /dev/null +++ b/templates/docs/release-notes.html @@ -0,0 +1,7 @@ +{%extends "base/page.html"%} +{%block title%}Release Notes - {{page.title}}{%endblock%} +{%block contents%} +
+ {{ page.content|safe }} +
+{% endblock contents %} diff --git a/tools/docs/docload.py b/tools/docs/docload.py index 2965f9c..5a6d178 100755 --- a/tools/docs/docload.py +++ b/tools/docs/docload.py @@ -10,6 +10,7 @@ import tidy from optparse import OptionParser from ConfigParser import ConfigParser +import bs4 import psycopg2 pagecount = 0 @@ -62,9 +63,66 @@ def load_doc_file(filename, f): 't': title, 'c': str(s), }) + # If this is a release note, load the release note. + if filename.startswith('release-'): + load_release_note(filename, str(s)) global pagecount pagecount += 1 +def load_release_note(filename, content): + """Load a release note into the system based on the filename""" + if not quiet: print "--- release note: %s" % filename + # Format the content for display in the release note page + parser = bs4.BeautifulSoup(content, "html.parser") + # Get the version that this release document is referencing + release_version = parser.find(class_="sect1")["id"].split("RELEASE-")[-1].replace('-', '.') + # Extract the versions + if float(release_version.split('.')[0]) >= 10 or float(release_version.split('.')[0]) <= 1: + try: + major_version, minor_version = release_version.split('.') + except ValueError: + major_version, minor_version = release_version, 0 + else: + m = re.search(r'(\d+).(\d+)(.(\d+))?', release_version) + major_version = "%s.%s" % (m.groups()[0], m.groups()[1]) + minor_version = m.groups()[3] if m.groups()[3] else 0 + # Remove extraneous sections of the release notes + for class_name in ['navheader', 'navfooter', 'toc']: + tag = parser.find(class_=class_name) + if tag: + tag.decompose() + # Update the headers to the proper release version, and remove extra references + # from the documentation + release_title = "Release " + release_version + # For really old release of PostgreSQL there is a nested product name + tag = parser.find('h2', class_="title") + if not tag.string: + tag.replace_with('

%s

' % release_title) + else: + tag.string.replace_with(release_title) + for tag in parser.find_all('h3', class_='title'): + if not tag.string: continue + m = re.search(r'(([A-Z0-9]+\.)+).(.*)$', tag.string) + if not m: continue + tag.string.replace_with(m.groups()[-1]) + # Update the URLs to point to the release note archives + for tag in parser.find_all(class_="xref"): + release = tag['href'].split('.')[0].split("release-")[-1] + title = 'Release ' + ".".join(release.split('-')) + tag['href'] = "/docs/release-notes/archive/" + tag['href'] + tag['title'] = title + tag.string.replace_with(title) + # Update the documentation links to point to the current docs. + for tag in parser.find_all(class_="link"): + tag['href'] = "/docs/current/static/" + tag['href'] + # The content is now ready to be loaded into the database + curs.execute(""" + INSERT INTO docs_releasenote (version, minor_version, title, content) + VALUES (%(v)s, %(m)s, %(t)s, %(c)s) + ON CONFLICT (version, minor_version) DO UPDATE + SET title = EXCLUDED.title, content = EXCLUDED.content""", + { 'v': major_version, 'm': minor_version, 't': release_title, 'c': parser.prettify() }) + ## Main execution parser = OptionParser(usage="usage: %prog [options] ") @@ -110,7 +168,7 @@ re_htmlfile = re.compile('[^/]*/doc/src/sgml/html/.*') re_tarfile = re.compile('[^/]*/doc/postgres.tar.gz$') for member in tf: if re_htmlfile.match(member.name): - load_doc_file(os.path.basename(member.name), tf.extractfile(member)) + load_doc_file(filename, tf.extractfile(member)) if re_tarfile.match(member.name): f = tf.extractfile(member) inner_tar = tarfile.open(fileobj=f) @@ -139,4 +197,3 @@ connection.commit() connection.close() if not quiet: print "Done (%i pages)." % pagecount - --Apple-Mail=_974930D2-E7E1-4F7A-AA75-654E4D34D515-- --Apple-Mail=_6266A028-AEBD-46E8-A46C-0B91F154DB6E Content-Transfer-Encoding: 7bit Content-Disposition: attachment; filename=signature.asc Content-Type: application/pgp-signature; name=signature.asc Content-Description: Message signed with OpenPGP -----BEGIN PGP SIGNATURE----- iQIzBAEBCAAdFiEE+oS2la8r95ogZD/x8QSccp8cZScFAltouvUACgkQ8QSccp8c ZSdDYBAAllsHKiIl1jGce9IuM6wc0O4oy8Yb1Jo29znIOqGHEHLIr0HhfChIpUHk QRC7+IG9W1Hqxnd+mJnedkHFnyabBoSqBZ+auQRUt346OWPfiD7MskGoWoVN1Erz 98hk2VTVDFtKtr2N7EEjo//UlA5hk/TWsJYaGLsvyT4QDBoZP6FAy8IY1png8wXq dStyZyPjraqzWwV9cRcn+P4WjyEr0uCA4Hq3hgyfT+TOi0xjBrliPrx4lEBlBYu3 DFurruMlWalCUEhvRx36a6Tmp5UZbHAk7PwVCuCEI1Wf2yQmpbV+Ljs5wYo6nOfD WAIaz5DAY3fbZ4/UuGu2RgffiO1zhPkgvCxbiaY145ThAdDS9dKNsmQzve+N2o83 FMQIsGz4XxlrBSxknfb0N8UJxJdvd0t9UEKB3vKXGkWhPiGPrvi8PqXDFU4G5V3h iRu75n7af1JCG/ug0Vxkj5f0dcTVljOB0+zF7nEBt/P7JvzQ1FUVvOO6z2zLd3Vw tFfEgVySab+fFZOJvS8x5kmCUpJxDUaUHvG7O992eQC7qJ8diAtKj93bNUqCjXtK 09qPsc5kBIsmcMt6Ct7jH41YApHVmit4+OrvlNwbOiS6ac9deiWY8PWL+Z/fHTil JXeYT8o2/9t5zPiNmdsmLyJF6qXMsKEqzo7fKt/aL85zXsXCZyk= =a+8l -----END PGP SIGNATURE----- --Apple-Mail=_6266A028-AEBD-46E8-A46C-0B91F154DB6E--