Subject: Cognitive dissonance

From: John Gage <[email-masked]>
Date: Tue, 8 Jun 2010 11:04:07 +0200

Unix is a text-based operating system with unbelievably helpful text  
manipulation tools.

Postgres is a creature of Unix which happens to have unbelievable text  
searching and manipulation tools.

Yet, the only one file edition of the Postgres documentation is  
in...pdf format.  Huh?

I know.  I know.  I have already brought this up.  And various ways of  
creating a one file text edition of the documentation have been  
proposed to me.  I know.

But either I am a visitor from the Crab Nebula, or there is someone  
else out there who would like to have a text file of the entire  
documentation.

Two examples from other applications.

I use Vim.  Vim's documentation is as easy to access as any  
documentation on earth...as long as you know exactly what you are  
looking for.  Otherwise, it is a tremendous pain.

I also use the National Library of Medicine's MeSH subject headings.   
25,000 descriptors with definitions, synonyms and a lot of other  
things.  They give it to you in single files either as text, xml, or  
other ways.  Big files.  Hundreds of megabytes.  That makes it so that  
you can do just about anything with it you want.  It is one of the  
seven wonders of the world.

I do suggest that a plain text file of the entire documentation be  
made part of the documentation armamentarium.

Respectfully,

John Gage

---

From: Lew <[email-masked]>
Date: Tue, 08 Jun 2010 08:07:13 -0400

John Gage wrote:
> 25,000 descriptors with definitions, synonyms and a lot of other things.
> They give it to you in single files either as text, xml, or other ways.
> Big files. Hundreds of megabytes. That makes it so that you can do just
> about anything with it you want. It is one of the seven wonders of the
> world.
>
> I do suggest that a plain text file of the entire documentation be made
> part of the documentation armamentarium.

 From <http://www.postgresql.org/docs/manuals/>:
"The DocBook SGML source for the manuals is available as part of the 
PostgreSQL source download available in the FTP area."

-- 
Lew

---

From: Peter Hunsberger <[email-masked]>
Date: Tue, 8 Jun 2010 09:23:13 -0500

On Tue, Jun 8, 2010 at 4:04 AM, John Gage <[email-masked]> wrote:
> Yet, the only one file edition of the Postgres documentation is in...pdf
> format.  Huh?
>

I suppose the next thing you'll be suggesting is that, because
Postgres is a database, the documentation should be stored as some
form of searchable table within the database itself....?

<runs and hides/>

-- 
Peter Hunsberger

---

From: Stephen Frost <[email-masked]>
Date: Tue, 8 Jun 2010 11:01:15 -0400

* John Gage ([email-masked]) wrote:
> But either I am a visitor from the Crab Nebula, or there is someone else 
> out there who would like to have a text file of the entire  
> documentation.

Soo..  there are quite a few man pages, and in-psql's help is also
pretty nice (\h <command> and \?).  That's certainly what I typically
use.  I admit that we don't include the full command description in the
\h (just the syntax), but that's still extremely useful.

Would a \h+ that gave you the text from the web-page be useful..?  That,
plus the various man pages, would cover an awful lot of what's in SGML..

	Thanks,

		Stephen

---

From: Justin Graf <[email-masked]>
Date: Tue, 08 Jun 2010 10:36:40 -0500

On 6/8/2010 9:23 AM, Peter Hunsberger wrote:
> On Tue, Jun 8, 2010 at 4:04 AM, John Gage<[email-masked]>  wrote:
>    

Its also available in chm  windows help file format.  Which i find allot 
more useful
http://www.postgresql.org/docs/manuals/
you could print chm to a text file.

also it not hard to dump a PDF document into a text file.

All legitimate Magwerks Corporation quotations are sent in a .PDF file attachment with a unique ID number generated by our proprietary quotation system. Quotations received via any other form of communication will not be honored.

CONFIDENTIALITY NOTICE: This e-mail, including attachments, may contain legally privileged, confidential or other information proprietary to Magwerks Corporation and is intended solely for the use of the individual to whom it addresses. If the reader of this e-mail is not the intended recipient or authorized agent, the reader is hereby notified that any unauthorized viewing, dissemination, distribution or copying of this e-mail is strictly prohibited. If you have received this e-mail in error, please notify the sender by replying to this message and destroy all occurrences of this e-mail immediately.
Thank you.

---

From: Chris Browne <[email-masked]>
Date: Tue, 08 Jun 2010 16:29:35 +0000

> also it not hard to dump a PDF document into a text file.

I wish I could find a converter that would generate one of the common
eBook formats (epub, mobi).

There do exist CHM readers on mobile platforms such as Android, but
they're much clumsier to work with than the rather more heavily used
eBook readers.

I have poked around for conversions; nothing particularly suitable has
emerged :-(.
-- 
select 'cbbrowne' || '@' || 'cbbrowne.com';
http://cbbrowne.com/info/internet.html
"MS  apparently now  has a  team dedicated  to tracking  problems with
Linux  and publicizing them.   I guess  eventually they'll  figure out
this back fires... ;)" -- William Burrow <[email-masked]>

---

From: John Gage <[email-masked]>
Date: Tue, 8 Jun 2010 18:42:05 +0200

Thank you all for your suggestions.  Thank you very much.

John

1) I suppose the next thing you'll be suggesting is that, because
Postgres is a database, the documentation should be stored as some
form of searchable table within the database itself....?

<runs and hides/>

------Well, that is exactly what I have done with the MeSH subject  
headings.  And it works like a charm.

2) Its also available in chm  windows help file format.  Which i find  
allot
more useful
http://www.postgresql.org/docs/manuals/
you could print chm to a text file.

------I'll have to boot over to XP, ugh.  Will do.

3)  also it not hard to dump a PDF document into a text file.

------I would print out what the dump looks like, but this is a family  
program

4) Would a \h+ that gave you the text from the web-page be useful..?   
That,
plus the various man pages, would cover an awful lot of what's in SGML..

-----I'm headed there.  It's just that given the incredibly good  
documentation and the fact that it's available in just about every  
format except a text file, I was sort of hoping for a policy change on  
the part of the powers that be.

---

From: Justin Graf <[email-masked]>
Date: Tue, 08 Jun 2010 13:21:49 -0500

***SNIP***
> ------I'll have to boot over to XP, ugh.  Will do.
There are linux chm readers

http://www.linux.com/news/software/applications/8209-chm-viewers-for-linux

and one for firefox
https://addons.mozilla.org/en-US/firefox/addon/3235/

---

From: John R Pierce <[email-masked]>
Date: Tue, 08 Jun 2010 11:30:01 -0700

> There are linux chm readers
...

Note that even Microsoft deprecated CHM back in 2003 after it was 
realized it was full of potential security exploits that couldn't 
readily be abated.

---

From: Josh Kupershmidt <[email-masked]>
Date: Tue, 8 Jun 2010 15:56:40 -0400

On Tue, Jun 8, 2010 at 5:04 AM, John Gage <[email-masked]> wrote:
> I do suggest that a plain text file of the entire documentation be made part
> of the documentation armamentarium.

Not that I see a whole lot of utility in this endeavor, but it's
possible to do a decent PDF to plain text conversion. I tried some of
the online tools that do this and found <http://www.pdftextonline.com>
to do the best job with the Postgres manual. I didn't bother trying
any client-side applications which do the same job.

Attached is a short snippet of a text export of the PDF manual.

Josh

---

From: John Gage <[email-masked]>
Date: Wed, 9 Jun 2010 07:28:54 +0200

1) On a list that howls with complaints when posts are in html, it is  
surprising that there is resistance to the idea of documentation in  
plain text.

2) Posters are correctly referred to the documentation as frequently  
as possible.  In fact, very frequently.  The frequency might decrease  
if the documentation were in plain text.  It is easier to search a  
single plain text file than any other source, except perhaps the  
database itself.

3) Postgres is getting pushed off the map at the low end by MySQL, now  
owned by Oracle.    If Postgres ceased to exist, Ellison would be  
thrilled.  I chose A2 Hosting (with whom I am very happy) for my  
website because they support Postgres.  I'm writing cgi scripts in  
perl.  I had to install the postgres driver for dbi.  It was not pre- 
installed.  There are about four buttons for MySQL on the cPanel and  
two farther over on the right for Postgres.

An anecdote.  I discovered the tsvector functionality a while back.  I  
have used it to create indices for my text files and several other  
tasks.  I recently was re-looking at my files and saw  
"tsvector::text".  I had forgotten that the double colon is one way to  
cast a type.  Double colon is not in the html index of the  
documentation.  I found it by searching my plain text version of the  
pdf file.  In my opinion, the html documentation is useful for reading  
it like a novel or referencing it in these lists.

On Jun 8, 2010, at 9:56 PM, Josh Kupershmidt wrote:

> Not that I see a whole lot of utility in this endeavor

---

From: Greg Smith <[email-masked]>
Date: Wed, 09 Jun 2010 02:24:53 -0400

John Gage wrote:
> Posters are correctly referred to the documentation as frequently as 
> possible.  In fact, very frequently.  The frequency might decrease if 
> the documentation were in plain text.  It is easier to search a single 
> plain text file than any other source, except perhaps the database 
> itself.

In reality searches are being done on the web, which combines the HTML 
version of the official documentation with blog posts, presentation 
materials, the wiki, and similar other resources.  This is why I don't 
actually care about a text version of the docs; I've just gotten used to 
using Google to search the PostgreSQL documentation.  The occasional 
time when I know I just want to search the manual instead, I can search 
the PDF version.  Neither of those are great solutions, but they're good 
enough that it's not worth fighting to build a text version over as I 
see it.  I'd use it if it were around, but there's little motivation for 
most of us to work on it.

> Postgres is getting pushed off the map at the low end by MySQL, now 
> owned by Oracle.

The dynamics are much more complicated than that.  Big MySQL sites are 
switching to NoSQL; medium sized MySQL sites are switching to PostgreSQL 
to get rid of scaling and reliability issues (I personally have been 
seeing a lot of this from Rails installs lately); small to medium size 
Oracle shops are switching to PostgreSQL to lower licensing costs.

The idea that plain-text documentation for the database would be a 
significant driver in any of these trends would be greatly exaggerating 
the significance of a technical detail important to a pretty small 
number of people.  On my personal list of "things that could be improved 
in the documentation", good plain text format is there, but there's a 
whole lot of things above it.

-- 
Greg Smith  2ndQuadrant US  Baltimore, MD
PostgreSQL Training, Services and Support
[email-masked]   www.2ndQuadrant.us

---

From: Brian Modra <[email-masked]>
Date: Wed, 9 Jun 2010 08:59:25 +0200

>> Not that I see a whole lot of utility in this endeavor

Personally I like to use html docs, and it would be good if the
documentation were downloadable from the postgresql website in other
formats, for convenience...

But, what I use is this, which works pretty well:

(e.g. to get the 8.1 dosc)

mkdir postgresql
cd postgresql
wget -r -nH -l 10 -k -np
http://www.postgresql.org/docs/8.1/interactive/index.html

... then after it all downloads:

open the file docs/8.1/interactive/index.html
in your web browser.

e.g.
links docs/8.1/interactive/index.html

HTML is "text", so you can search using grep e.g.
grep -r "ALTER TABLE .* ADD COLUMN" docs/8.1

>
>
> --
> Sent via pgsql-general mailing list ([email-masked])
> To make changes to your subscription:
> http://www.postgresql.org/mailpref/pgsql-general
>

-- 
Brian Modra   Land line: +27 23 5411 462
Mobile: +27 79 69 77 082
5 Jan Louw Str, Prince Albert, 6930
Postal: P.O. Box 2, Prince Albert 6930
South Africa
http://www.zwartberg.com/
Fax: +27865510467

---

From: Torsten Zühlsdorff <[email-masked]>
Date: Wed, 09 Jun 2010 09:44:10 +0200

> grep -r "ALTER TABLE .* ADD COLUMN" docs/8.1

Thats the way i do too. A huge pdf is often not very helpful. In my 
personal case i programm often in a train, using my laptop. Searching a 
PDF with more than 1.000 pages really hits my battery. With html-files i 
could preselect the items to search.
Also it's possible to import the html-files in a postgres-db and using 
fulltext-search. ;)

Greetings,
Torsten
-- 
http://www.dddbl.de - ein Datenbank-Layer, der die Arbeit mit 8 
verschiedenen Datenbanksystemen abstrahiert,
Queries von Applikationen trennt und automatisch die Query-Ergebnisse 
auswerten kann.

---

From: Dave Coventry <[email-masked]>
Date: Wed, 9 Jun 2010 10:24:17 +0200

My tupp'th:

Formatted text, whether PDF, HTML or (heaven forbid!) Word Documents,
is easier to read than unformatted plain text, and those of us without
the OP's very admirable proficiency in vi remain at the mercy of the
various readers and their associated search functions.

However, I sure that it's not too arduous a task to extract the text
in these documents and strip them of their formatting?

Or am I missing something?

---

From: Dimitri Fontaine <[email-masked]>
Date: Wed, 09 Jun 2010 11:38:56 +0200

> Or am I missing something?

Info documentation format. Text based, super user aware, easy to
browse and search, has an index.

You can even produce postgres.info today, it's just not optimised to be
very friendly, it's missing mainly convenient table support and
index. 

Regards,
-- 
dim

---

From: Alvaro Herrera <[email-masked]>
Date: Wed, 09 Jun 2010 12:46:04 -0400

Excerpts from John Gage's message of mié jun 09 01:28:54 -0400 2010:

> I recently was re-looking at my files and saw  

I just added an index entry for ::, thanks for pointing out that it was
missing.

If you notice other missing index entries, do not hesitate to point it
out in this mailing list or pgsql-docs.

-- 
Álvaro Herrera <[email-masked]>
The PostgreSQL Company - Command Prompt, Inc.
PostgreSQL Replication, Consulting, Custom Development, 24x7 support

---

From: Lew <[email-masked]>
Date: Wed, 09 Jun 2010 20:18:51 -0400

> formats, for convenience...

Good thing it is, then, albeit not in the most convenient format, i.e., 
DocBook.  But then, from there you can generate pretty much any format you 
want, right?

-- 
Lew

---

From: Peter Eisentraut <[email-masked]>
Date: Thu, 10 Jun 2010 09:50:14 +0300

> else out there who would like to have a text file of the entire  

As I said back then, doing this is straightforward, but we kind of need
more than one user who asks for it before we make it part of a regular
service, which comes with maintenance costs.

---

From: Alvaro Herrera <[email-masked]>
Date: Thu, 10 Jun 2010 11:16:37 -0400

> service, which comes with maintenance costs.

Hey, count me as another interested person in a single-file plain-text
doc output format.

---

From: Tom Lane <[email-masked]>
Date: Thu, 10 Jun 2010 11:24:00 -0400

> Hey, count me as another interested person in a single-file plain-text

Well, there are two separate things here:

* providing a Makefile target to build plain-text output.

* shipping prebuilt plain text docs in standard distributions.

I am for #1, not so much for #2, mainly on the grounds of size.  But
given #1 it would be possible for packagers to make their own choices
about whether to include plain-text docs.

			regards, tom lane

---

From: Leif Biberg Kristensen <[email-masked]>
Date: Thu, 10 Jun 2010 17:33:18 +0200

> about whether to include plain-text docs.

Wouldn't it suffice to make it downloadable, like the pdf doc?

regards,
-- 
Leif Biberg Kristensen
http://solumslekt.org/blog/

---

From: John Gage <[email-masked]>
Date: Thu, 10 Jun 2010 22:57:36 +0200

Like all visitors from the Crab Nebula (except our leaders who are  
genetically separate) I qualify as a novice when it comes to  
Postgres.  What is more, the people (humans, that is) who need the  
documentation the most are those who, well, need the documentation the  
most.

Hence, if this were to be made available, it would be great if it was  
novice speed.

Thanks everyone for even contemplating it.

John

---

From: Robert Gravsjö <[email-masked]>
Date: Fri, 11 Jun 2010 14:41:29 +0200

Leif Biberg Kristensen skrev 2010-06-10 17.33:
>> Alvaro Herrera<[email-masked]>  writes:

And/or make the HTML version downloadable side by side with the PDF.

There are good reasons for wanting access to the complete document when 
being offline. PDF is not such a bad format but it do have some 
limitations as have been previously mentioned.

As for building the docs I don't think everyone, not even all 
developers, has the tool chain installed (or even wants to).

Regards,
roppert

---

From: Bruce Momjian <[email-masked]>
Date: Fri, 11 Jun 2010 21:20:51 -0400 (EDT)

> And/or make the HTML version downloadable side by side with the PDF.

That might be easy to do.  We already build the HTML, and requiring
people to recursively use wget is not user-friendly.

-- 
  Bruce Momjian  <[email-masked]>        http://momjian.us
  EnterpriseDB                             http://enterprisedb.com

  + None of us is going to be here forever. +

---

From: John Gage <[email-masked]>
Date: Sat, 12 Jun 2010 11:18:00 +0200

A one file html version would be a godsend.

On Jun 12, 2010, at 3:20 AM, Bruce Momjian wrote:

> Robert GravsjÃ¶ wrote:

---

From: Peter Eisentraut <[email-masked]>
Date: Sat, 12 Jun 2010 12:56:14 +0300

> A one file html version would be a godsend.

I've committed a build target for that now.  Use 'make postgres.html' in
doc/src/sgml/.

---

From: Tom Lane <[email-masked]>
Date: Sat, 12 Jun 2010 09:10:29 -0400

> I've committed a build target for that now.  Use 'make postgres.html' in

Huh, is that actually worth anything?  How many browsers will open it
without crashing, or will navigate the page with decent performance
if they do manage to open it?

(Not that I object to providing this Make target.  But I thought the
discussion was about plain-text output.)

			regards, tom lane

---

From: John Gage <[email-masked]>
Date: Sat, 12 Jun 2010 16:34:17 +0200

It was.  But if the compromise is single file html, that is a vast  
improvement over the current system imho.

What I want is the thing that is maximally amenable to being searched  
conveniently using all the tools at our disposal especially regular  
expressons.  The point has been made that Google is the best system to  
search for Postgres documentation/knowledge/etc.  Frankly, I don't  
necessarily agree with that, particularly for the novice.  The  
documentation is where it is at, and it is the documentation that is  
referenced the most in these posts.

But there are no dichotomies here.  It is not either or.  It is a  
balance between what is easiest to produce and maintain and what is  
most productive to use.

And the background for my request is my respect for the extraordinary  
power and elegance of postgres.

John

---

From: Stephen Frost <[email-masked]>
Date: Sat, 12 Jun 2010 10:35:58 -0400

> if they do manage to open it?

If it works with links, that'd probably work well for the use case
described...

	Stephen

---

From: Peter Eisentraut <[email-masked]>
Date: Sat, 12 Jun 2010 18:42:24 +0300

> if they do manage to open it?

Text output is generated by going through HTML.  I haven't figured out
the best way to do the second step yet.  We use lynx for INSTALL and
HISTORY, but the results for this big file aren't very clean.

Browsers seem to handle the file OK, btw.

---

From: Bruce Momjian <[email-masked]>
Date: Sat, 12 Jun 2010 12:01:58 -0400 (EDT)

Peter Eisentraut wrote:
> On l?r, 2010-06-12 at 09:10 -0400, Tom Lane wrote:
> > > On l?r, 2010-06-12 at 11:18 +0200, John Gage wrote:

Well, I tried lynx and the output looked fine to me, so I applied the
attached patch to allow single-page text output.  You can see the HTML
and text file results here:

	http://momjian.us/expire/

The new rule name is postgres.txt.  The file size are:

	7,789,730  postgres.html
	5,155,672  postgres.txt

  + None of us is going to be here forever. +
Index: doc/src/sgml/Makefile
===================================================================
RCS file: /cvsroot/pgsql/doc/src/sgml/Makefile,v
retrieving revision 1.145
diff -c -c -r1.145 Makefile
*** doc/src/sgml/Makefile	12 Jun 2010 15:42:44 -0000	1.145
--- doc/src/sgml/Makefile	12 Jun 2010 15:57:31 -0000
***************
*** 108,113 ****
--- 108,117 ----
  postgres.html: postgres.sgml $(ALLSGML) stylesheet.dsl
  	$(JADE.html.call) -V nochunks -V rootchunk -V '(define %root-filename% #f)' -V '(define use-output-dir #f)' -i include-index $<
  
+ # single-page text
+ postgres.txt: postgres.html
+ 	$(LYNX) -force_html -dump -nolist -stdin $< > $@
+ 
  HTML.index: postgres.sgml $(ALMOSTALLSGML) stylesheet.dsl
  	@$(MKDIR_P) html
  	$(JADE.html.call) -V html-index $<

---

From: John Gage <[email-masked]>
Date: Sat, 12 Jun 2010 18:39:55 +0200

UFB!  This was definitely worth the visit from the Nebula.

Thanks very, very much.

Sensational.

Thanks again,

John Gage

---

From: Bruce Momjian <[email-masked]>
Date: Sat, 12 Jun 2010 12:56:21 -0400 (EDT)

> UFB!  This was definitely worth the visit from the Nebula.

We still have to decide how to make these accessible from our web site.

---

From: John Gage <[email-masked]>
Date: Sat, 12 Jun 2010 19:13:49 +0200

Ah, perhaps I have misunderstood. Double ah.

So that's the reason for the current html format: ease of browser  
access.

On the other hand, the pdf file, at 17MB, is the biggest one and, as  
far as I am concerned, the least useful.

I would just make the single file html and text files available for  
download beside the pdf.  See how many people take which format.

The only change I would make in the current html browser version is to  
put the "main page" link at the top as well as the bottom.

Thanks again,

John

---

From: Tim Landscheidt <[email-masked]>
Date: Sat, 12 Jun 2010 17:15:01 +0000

Bruce Momjian <[email-masked]> wrote:

> [...]
                                          ^^^^^^
> [...]

Isn't that unnecessary/wrong as the filename is supplied on
the command line?

Tim

---

From: Bruce Momjian <[email-masked]>
Date: Sat, 12 Jun 2010 13:17:57 -0400 (EDT)

> Isn't that unnecessary/wrong as the filename is supplied on

Ah, good catch. I wasn't able to test that because my lynx version
doesn't support -stdin.  Thanks, and updated.

---

From: Chris Browne <[email-masked]>
Date: Mon, 14 Jun 2010 15:33:37 -0400

[email-masked] (Tom Lane) writes:
>> On lÃ¶r, 2010-06-12 at 11:18 +0200, John Gage wrote:

I expect that having one HTML file would make it reasonably easy to
use lynx/links/w3m [some text-based HTML browser] to transform HTML
into plain text.

I should think that it would also enable using analagous tools to
transform HTML into eBook formats like ePub and mobi, which are the
frequently-preferable-formats on the emerging category of "electronic
book" appliances.

I have browsed the CHM form of the docs using a CHM reader on my
phone, and found that less than wonderful, which had a lot to do with
the reader not being particularly great.  A format that plays well
with a decent reader may turn out pretty happily.
-- 
select 'cbbrowne' || '@' || 'cbbrowne.com';
http://cbbrowne.com/info/internet.html
"MS  apparently now  has a  team dedicated  to tracking  problems with
Linux  and publicizing them.   I guess  eventually they'll  figure out
this back fires... ;)" -- William Burrow <[email-masked]>