Subject: Images in the official documentation

From: Daniel Westermann <[email-masked]>
Date: Fri, 23 Feb 2018 17:14:06 +0100 (CET)

Hi %, 

I am working with PostgreSQL documentation quite a few years now and I am almost happy. What I think is completely missing (especially if you compare to commercial product documentation) are pictures that illustrate a topic, e.g. the relation of instance->database->user/role->schema->objects. 

Is there an agreement not to include that pictures for any reason? I can not promise that I find time for that in the near future but if that will be appreciated I am willing to spend time on that to make the documentation even better. 

Regards 
Daniel

---

From: Tom Lane <[email-masked]>
Date: Fri, 23 Feb 2018 11:21:28 -0500

Daniel Westermann <[email-masked]> writes:
> Is there an agreement not to include that pictures for any reason?

This has been discussed repeatedly (see the archives).  We'd surely
like the ability to include figures, but we've not found any tools that
met the varying requirements people wanted to set --- mainly, that it
be possible to commit readable source text for figures into our git
repo and have updates that were reviewable (ie, didn't amount to a 100%
replacement of one set of gibberish with another one).  In the distant
past, as I recall, we had a GIF or two; but we abandoned that on the
grounds that it was unmaintainable and also incompatible with some
documentation output formats.  I'm not too sure what the state of
play is on the latter point, now that we've switched to XML.

			regards, tom lane

---

From: Peter Eisentraut <[email-masked]>
Date: Fri, 23 Feb 2018 20:04:26 -0500

On 2/23/18 11:21, Tom Lane wrote:
> In the distant

The complications with the image formats in the past were mainly around
what ((pdf)jade)tex would accept.  The tools have shifted a bit now, and
the zoo formats is a different one.  Nothing that a few make rules
couldn't address, though, I think.

The issue of how to manage the sources is still the same, though.

-- 
Peter Eisentraut              http://www.2ndQuadrant.com/
PostgreSQL Development, 24x7 Support, Remote DBA, Training & Services

---

From: Oleg Bartunov <[email-masked]>
Date: Sat, 24 Feb 2018 21:45:16 +0300

> The issue of how to manage the sources is still the same, though.

SVG format is ascii based vector format. We made experimental pdf with pictures
http://www.sai.msu.su/~megera/postgres/files/postgres-11-diagram.pdf
(GIN AM diagram, Appendix L).

Appendix L also demonstrates our sample database with step-by-step
introduction to Postgres for beginners.  We have a separate book for beginners,
which we released under BSD license and  it's available on
russian/english languages.
Our experience shows, that people really appreciate it. I hope we will
have time at PGCon
to discuss documentation somehow.

---

From: Tom Lane <[email-masked]>
Date: Sat, 24 Feb 2018 14:11:58 -0500

> SVG format is ascii based vector format.

Yeah.  I think where the last discussion left this was that we'd be
willing to accept SVG-format figures, but we were having a hard time
figuring out what tools to recommend for editing them, because available
editors have no respect for readability or preserving small deltas when
rewriting SVG files.

			regards, tom lane

---

From: Jürgen Purtz <[email-masked]>
Date: Sun, 25 Feb 2018 18:00:30 +0600

As an addition to my mail from January 2016 concerning graphics 
(https://www.postgresql.org/message-id/568A9148.30303%40purtz.de) I 
propose to use SVG (after switching to XML) - but not an SVG which is 
generated by Inkscape or similar tools. Those editors generate very ugly 
and chatty commands. This form is not easy to read or understand. 
Therefore we shall use nothing but a simple text editor and write every 
line by our self. The process is divided into two parts:

As a basis we shall develop an SVG library containing a bunch of 
"atomic" symbols of simple graphical elements (rectangle, arrow, ...) up 
to complex elements (magnetic disc, laptop, cloud, UML-elements, ...). 
The SVG routines creating those symbols shall accept parameters for 
position, size, rotation, colour, ... . This library shortens the 
individual SVG files, it ensures a consistent rendering of common 
graphical elements, it is diff-able, and it will reach a stable state - 
some day.

The real graphics shall use the elements of the library and add 
individual SVG elements. The rules for this part are the same as above: 
create SVG commands with vi (or similar), store it in git.

If such an approach works (we must distribute the docs across a wide 
range of different systems, a proof-of-system is necessary) and the 
community accepts my proposal, I would like to work on the library-part 
- starting after finishing my actual project in about 6 weeks from now. 
The attached file contains a very first draft as of Jan. 2016.

Kind regards
Jürgen Purtz

On 23.02.2018 22:14, Daniel Westermann wrote:
> Hi %,
>
> I am working with PostgreSQL documentation quite a few years now and I 
> am almost happy. What I think is completely missing (especially if you 
> compare to commercial product documentation) are pictures that 
> illustrate a topic, e.g. the relation of 
> instance->database->user/role->schema->objects.
>
> Is there an agreement not to include that pictures for any reason? I 
> can not promise that I find time for that in the near future but if 
> that will be appreciated I am willing to spend time on that to make 
> the documentation even better.
>
> Regards
> Daniel

---

From: Steve Atkins <[email-masked]>
Date: Sun, 25 Feb 2018 12:12:02 -0800

> On Feb 25, 2018, at 4:00 AM, Jürgen Purtz <[email-masked]> wrote:
> 
> As an addition to my mail from January 2016 concerning graphics (https://www.postgresql.org/message-id/568A9148.30303%40purtz.de) I propose to use SVG (after switching to XML) - but not an SVG which is generated by Inkscape or similar tools. Those editors generate very ugly and chatty commands. This form is not easy to read or understand. Therefore we shall use nothing but a simple text editor and write every line by our self. The process is divided into two parts:
> As a basis we shall develop an SVG library containing a bunch of "atomic" symbols of simple graphical elements (rectangle, arrow, ...) up to complex elements (magnetic disc, laptop, cloud, UML-elements, ...). The SVG routines creating those symbols shall accept parameters for position, size, rotation, colour, ... . This library shortens the individual SVG files, it ensures a consistent rendering of common graphical elements, it is diff-able, and it will reach a stable state - some day.
> 
> The real graphics shall use the elements of the library and add individual SVG elements. The rules for this part are the same as above: create SVG commands with vi (or similar), store it in git.
> If such an approach works (we must distribute the docs across a wide range of different systems, a proof-of-system is necessary) and the community accepts my proposal, I would like to work on the library-part - starting after finishing my actual project in about 6 weeks from now. The attached file contains a very first draft as of Jan. 2016.

Writing SVG by hand maybe doesn't seem the best idea.

I understand the attraction to people who want to store everything as diffable text, but images of this sort are unlikely to get updated by others, which means they're unlikely to be maintained as the things they're intended to document change. It also means that the people best suited to generating diagrams are the least likely to do so, and vice-versa.

Cheers,
  Steve

---

From: Craig Ringer <[email-masked]>
Date: Mon, 26 Feb 2018 10:45:10 +0800

On 26 February 2018 at 04:12, Steve Atkins <[email-masked]> wrote:

> I understand the attraction to people who want to store everything as
> diffable text, but images of this sort are unlikely to get updated by
> others, which means they're unlikely to be maintained as the things they're
> intended to document change. It also means that the people best suited to
> generating diagrams are the least likely to do so, and vice-versa.
>
>
>
Yeah, I think it'd just effectively preserve the status quo by rendering
anyone who's willing to add images and designs to the docs unable - or
unlikely to be willing - to do so.

-- 
 Craig Ringer                   http://www.2ndQuadrant.com/
 PostgreSQL Development, 24x7 Support, Training & Services

---

From: Steve Atkins <[email-masked]>
Date: Sun, 25 Feb 2018 19:18:50 -0800

> On Feb 25, 2018, at 6:45 PM, Craig Ringer <[email-masked]> wrote:
> 
> Yeah, I think it'd just effectively preserve the status quo by rendering anyone who's willing to add images and designs to the docs unable - or unlikely to be willing - to do so. 

Yup. I do think that graphics would be nice in a few places, and that SVG is likely the best format for them.

There are quite a few tools that could be integrated with varying amounts of effort into the documentation generation workflow.

# ASCII language or ascii art to SVG

# asciitosvg

https://github.com/dhobsd/asciitosvg

Inspired by ditaa, similar in functionality

## blockdiag, seqdiag, actdiag, nwdiag

http://blockdiag.com

Generates various box and arrow diagrams from a DOT-ish input language.

## ditaa

It takes ascii art of box and arrow diagrams and turns them into nice svg. Also supports boxes that look like "storage" cylinders, documents, clouds and computers.

## erd

https://github.com/BurntSushi/erd

Entity relationship diagrams, from a plain text input. Uses DOT and graphviz under the covers.

## Markdeep

http://casual-effects.com/markdeep/

In-browser javascript rendering of ascii art, particularly boxes and arrows.

Alternate implementation, https://github.com/blampe/goat, converts to SVG via CLI.

## Mermaid

Flowcharts and sequence diagrams from a markdown-esque input.

## mscgen

Message sequence chart inputs in a DOT-ish language to SVG

## plantuml

http://plantuml.com

It supports a human-editable descriptive text input language and generates from it:

  sequence diagrams
  various box + arrow style diagrams
  flowcharts
  state diagrams
  etc.

## shaape

https://github.com/christiangoltz/shaape

Converts ascii art to SVG. Rather nice.

## svgbob

https://github.com/ivanceras/svgbobrus

Ascii art to SVG. Likely good for boxes and arrows.

## syntrax

https://kevinpt.github.io/syntrax/

Railroad diagrams (like the ones SQLite docs are known for).

## umlet

http://www.umlet.com

Pointy-clicky editor, but also converts plain text to uml, sequence, activity diagrams

# Interactive editor

## SVG-Edit

https://github.com/SVG-Edit/svgedit

Open source, browser based interactive SVG editor. Seems to generate fairly clean SVG.

Cheers,
  Steve

---

From: Tom Lane <[email-masked]>
Date: Sun, 25 Feb 2018 23:16:49 -0500

> unlikely to be willing - to do so.

This is an entirely reasonable complaint.  But I don't see how we'd cope
with patches that rewrite an entire SVG file because the patch author's
WYSIWG editor emits its output in a style randomly different from the tool
the previous patch author used.  It seems like such patches would be
effectively unreviewable, and certainly incapable of being merged.

Maybe we could improve matters a bit by insisting that everyone use the
same version of the same SVG-editing tool.  But that's not too practical.
Worse, from what I've seen, even that would not really fix the problem.
The tools simply don't give a damn about comparability of their dump
files.  I don't blame their authors exactly (try diffing Postgres data
file changes :-() but that doesn't mean it isn't a problem for us.

How can we resolve these issues?

			regards, tom lane

---

From: Craig Ringer <[email-masked]>
Date: Mon, 26 Feb 2018 14:30:31 +0800

> effectively unreviewable, and certainly incapable of being merged.

Well, they definitely couldn't be merged in any situation entailing
conflicts, no.

Patch review would entail displaying the new and (if present) old SVGs,
possibly in the context of a build of the docs, or possibly standalone.

This is always going to be the case for anything but the most trivial SVG
changes anyway. After all, even small textual changes can cause elements to
overlap, break out of their expected bounaries, or otherwise look wrong.

So IMO whether it's SVG or a raster image format, the net effect isn't that
different: you have to review the rendered result not the source.
Personally I'd just mark svg as binary in .gitattributes to stop it from
spamming noise in diffs.

Github offers a cool tool for side-by-side compares of svg diffs (
https://github.com/blog/1902-svg-viewing-diffing) but that likely won't
help us much.

There's diffsvg (https://github.com/jrsmith3/diffsvg), which I haven't
tried but looks interesting. Combined with filters in .gitattributes this
might offer improved visibility into change history if we really need it.

Personally, I don't think images will be changing that often and they
should just be tracked as blobs.

> How can we resolve these issues?

Question the assumptions and requirements. Why do we actually _need_
diffable, mergeable images? Sure, it'd be *nice*, but what's the real world
impact if we don't have it?

---

From: Tom Lane <[email-masked]>
Date: Mon, 26 Feb 2018 14:23:14 -0500

> impact if we don't have it?

Well, I'll tell you exactly why I'm being sticky about this: we've been
down this road before.  We used to have some figures in .gif format,
and one of the problems with them was they were too hard to update.
I don't buy the "they won't need updates" argument for a second, either.
For instance, I recall that one of the images we had was a diagram of
the system catalog cross-references, and it was constantly out of date
because of the difficulty of updating it.

Admittedly, this was 15+ years ago.  Maybe the state of the art in
figure editors has advanced to the point where it won't be so hard.
But color me suspicious.

			regards, tom lane

---

From: Craig Ringer <[email-masked]>
Date: Tue, 27 Feb 2018 09:02:40 +0800

> because of the difficulty of updating it.

Yeah, that sounds painful. I can certainly see your objection when framed
in terms of things like illustrations of catalogs and catalog relationships.

If I were maintaining the docs in a vacuum, I'd use graphviz for something
like that, because it's a figure that does need regular updates and
changes. And because
 the list of fun things to do in my life definitely does not include
hand-writing SVG. Not that tweaking GraphViz .dot is fun, but it's the
default tool for a reason.

I'd be awfully tempted to generate the node-map part of the catalog
relationship .dot file from a query, too.

I still advocate for relaxing the policy for images that are *not* likely
to need frequent updates, but also for being conservative about how/when we
include images. Does this add real value to the docs, is it worth any
maintenance hassle? Then, for things that will change more, like catalogs,
using a tool like graphviz. If we object to adding a docs build-dependency,
we could always commit generated files like we already do for the
'configure' script, and make sure there's a committer/maintainer Make
target that warns if the sources are newer than the docs.

---

From: Oleg Bartunov <[email-masked]>
Date: Tue, 27 Feb 2018 11:52:45 +0300

> figure editors has advanced to the point where it won't be so hard.

In case you missed, a couple of years ago we discussed this on pgcon:

Heikki's version:
https://wiki.postgresql.org/wiki/Figures_%26_Pics_in_Docs

Emre suggested to use Markdeep (BSD license),
http://casual-effects.com/markdeep/
http://www.sai.msu.su/~megera/postgres/gin-ascii-v2.md.html

It looks good for small diagrams, but will not work for complex stuff,
such as pg_catalog structure.

---

From: Peter Eisentraut <[email-masked]>
Date: Tue, 27 Feb 2018 10:13:42 -0500

On 2/26/18 20:02, Craig Ringer wrote:
> If I were maintaining the docs in a vacuum, I'd use graphviz for
> something like that, because it's a figure that does need regular
> updates and changes. And because

I think graphviz would be a great fit for what we are discussing here.
Certainly more so then some-person-on-github's latest idea for how to
convert ASCII art into diagrams.

---

From: Alvaro Herrera <[email-masked]>
Date: Tue, 27 Feb 2018 12:19:14 -0300

> convert ASCII art into diagrams.

... particularly so if said idea involves PHP, Haskell, Go, Python, or
any other language that we don't currently have as requirement in our
build chain.

GraphViz gets my vote, too.  It may not produce the most elegant
diagrams in the universe, but the source format is as good as we can
get.

-- 
Álvaro Herrera                https://www.2ndQuadrant.com/
PostgreSQL Development, 24x7 Support, Remote DBA, Training & Services

---

From: Jürgen Purtz <[email-masked]>
Date: Wed, 11 Jul 2018 11:07:29 +0200

Our discussion about grafics in the documentation reached to the 
conclusion that we shall use SVG, the importance to 'diff-ability' is 
rated differently, and there is no consensus about tools.

To push the issue forward I modify my original proposal to use plain svg 
files in a standard editor as follows:

  * We define a 'Simplified SVG format' (SSVG)
  * We create libraries where complex elements are predefined and can be
    referenced
  * We write the source in ssvg-format
  * A compiler (written in bison or xslt) converts ssvg-files to svg-files
  * We extend the sgml-files to include the svg-files
  * The ssvg and svg-files are located in a new svg directory, Makefile
    copies them to sgml and html directory
  * A proof-of-concept is performed in 11beta2 for HTML and PDF generation.
  * The ssvg-format may be XML (as used in the examples), JSON, C-style
    function calls

PRO:

  * SVG 1.x has many restrictions and SVG 2.x does not make progress in
    the last years. Tools and Browsers support different ranges of the
    specification. The planned compiler cuts everything down to the
    basic language level, where a broad support is possible.
  * Predefinded elements and default values reduce the ssvg file to a
    small and clear source file.
  * You can embed original svg commands into ssvg files.
  * When you use an editor and a browser in parallel, you get the visual
    result with few clicks.
  * Everything is diff-able.
  * The Makefile needs only slightly amendments: additional cp commands
    and some target-dependencies. We need no new tool.

CON:

  * The development is done in a non-wysiwyg editor and without mouse.
  * You have to count pixel.

Example:
PageLayout.ssvg: written in the new language
PageLayout.svg: the generated svg file (actually by hand, the compiler 
is not yet implemented)
storage.sgml: an additional paragraph to refer to the svg-file

<para>
   <mediaobject id="PageLayoutSVG">
     <imageobject role="html">
       <imagedata fileref="PageLayout.svg" format="SVG"/>
     </imageobject>
     <imageobject role="fo">
       <imagedata fileref="PageLayout.svg" format="SVG" scalefit="1" 
width="100%" contentdepth="100%"/>
     </imageobject>
   </mediaobject>
</para>

PageLayoutHtml.png: the HTML result
PageLayoutPdf.png: the PDF result

A second example: pgDump.svg within backup.sgml

Kind regards, Jürgen Purtz

---

From: Pavel Golub <[email-masked]>
Date: Thu, 19 Jul 2018 15:06:08 +0300

Hello, Jürgen.

You wrote:

JP> Our discussion about grafics in the documentation reached to the 
JP> conclusion that we shall use SVG, the importance to 'diff-ability'
JP> is rated differently, and there is no consensus about tools.

I disagree. From what I heard, GraphViz is the winner for now. I can
give you my two cents: plantuml is another good choice

JP> To push the issue forward I modify my original proposal to use   
JP> plain svg files in a standard editor as follows:

JP>   We define a 'Simplified SVG format' (SSVG)  We create libraries
JP> where complex elements are predefined and         can be
JP> referenced  We write the source in ssvg-format  A compiler
JP> (written in bison or xslt) converts ssvg-files to        
JP> svg-files  We extend the sgml-files to include the svg-files  The
JP> ssvg and svg-files are located in a new svg directory,        
JP> Makefile copies them to sgml and html directory  A
JP> proof-of-concept is performed in 11beta2 for HTML and PDF        
JP> generation.  The ssvg-format may be XML (as used in the examples),
JP> JSON,         C-style function calls   
JP>   

JP>        PRO:
JP>   
JP>   
JP>   SVG 1.x has many restrictions and SVG 2.x does not make        
JP> progress in the last years. Tools and Browsers support different  
JP> ranges of the specification. The planned compiler cuts        
JP> everything down to the basic language level, where a broad        
JP> support is possible.  Predefinded elements and default values
JP> reduce the ssvg file         to a small and clear source file. 
JP> You can embed original svg commands into ssvg files.  When you use
JP> an editor and a browser in parallel, you get the         visual
JP> result with few clicks.  Everything is diff-able.  The Makefile
JP> needs only slightly amendments: additional cp         commands and
JP> some target-dependencies. We need no new tool.  
JP>   
JP> CON:
JP>   
JP>   
JP>   The development is done in a non-wysiwyg editor and without    
JP> mouse.  You have to count pixel.  
JP>   

JP>        Example:
JP>        PageLayout.ssvg: written in the new language
JP>        PageLayout.svg: the generated svg file (actually by hand,
JP> the       compiler is not yet implemented)
JP>        storage.sgml: an additional paragraph to refer to the svg-file
JP>   
JP>        <para>
JP>          <mediaobject id="PageLayoutSVG">
JP>            <imageobject role="html">
JP>              <imagedata fileref="PageLayout.svg" format="SVG"/>
JP>            </imageobject>
JP>            <imageobject role="fo">
JP>              <imagedata fileref="PageLayout.svg" format="SVG"    
JP> scalefit="1" width="100%" contentdepth="100%"/>
JP>            </imageobject>
JP>          </mediaobject>
JP>        </para>
JP>   
JP>        PageLayoutHtml.png: the HTML result
JP>        PageLayoutPdf.png: the PDF result
JP>   
JP>        A second example: pgDump.svg within backup.sgml
JP>   
JP>   
JP>        Kind regards, Jürgen Purtz
JP>   
JP>   

JP>   
JP>   

-- 
With best wishes,
 Pavel                          mailto:[email-masked]

---

From: Jürgen Purtz <[email-masked]>
Date: Thu, 19 Jul 2018 17:31:23 +0200

> give you my two cents: plantuml is another good choice

Ok, please give us an example - possibly the two previous graphics.

Kind regards, Jürgen

---

From: Pavel Golub <[email-masked]>
Date: Fri, 20 Jul 2018 19:14:13 +0300

Hello, Jürgen.

You wrote:

>> give you my two cents: plantuml is another good choice

JP> Ok, please give us an example - possibly the two previous graphics.

Fair enough. Let's try different formats. Sorry, what exactly previous
graphics we're talking about?

JP> Kind regards, Jürgen

---

From: Jürgen Purtz <[email-masked]>
Date: Sun, 22 Jul 2018 21:46:31 +0200

Because I strongly support the involvement of SVG into our 
documentation, I welcome every activity to establish this goal. The mail 
to which Pavel had replied 
(https://www.postgresql.org/message-id/4ea1bacb-02ca-e967-31d7-d2a6db30abff%40purtz.de) 
contained the two graphics /PageLayout.svg/ "Page Layout" and 
/pgDump.svg/ "pg_dump, psql, pg_restore". Actually they are *not* part 
of our documentation. They are only part of a proof-of-concept to 
generate SVG and include the result into HTML and PDF output. The two 
examples can be used as references for comparisons with any tool.

Kind regards, Jürgen

---

From: Jürgen Purtz <[email-masked]>
Date: Wed, 14 Nov 2018 14:36:09 +0100

After many mails, F2F-discussions and some experiments - everything 
without a concrete result - I hereby start a new initiative concerning 
SVG tools. The attached PDF file describes my recommendation. The two 
SVG files are examples for working in this sense.

Kind regards

Jürgen Purtz

---

From: "Jonathan S. Katz" <[email-masked]>
Date: Wed, 14 Nov 2018 09:39:49 -0500

Hi Jürgen,

> SVG files are examples for working in this sense.

It looks like you put a lot of effort into this, thanks! Those SVGs do
look great, and it's IMO it's a much needed feature to support images in
our documentation :) I will leave it to others to discuss if the
generated SVGs are a technical fit for the documentation code.

I would recommend that you transfer the contents of your proposal to the
PostgreSQL wiki (https://wiki.postgresql.org/) as it would make it
easier for others to collaborate on and contribute to it. If you need
help with it, please feel free to ping me.

The one thing that I see is if we would like to have a style guide,
primarily around colors, for images we put into the repo. I know that
somehow correlates with the definition of "bike shed" but it could also
help reduce that in the future. As we do have a PostgreSQL color
palette, I also see that as an opportunity to reduce potential
bikeshedding as well.

Thanks!

Jonathan