Subject: Re: non-ASCII characters in SGML documentation (and
 elsewhere)
From: Peter Eisentraut <peter_e@gmx.net>
To: Alvaro Herrera <alvherre@commandprompt.com>
Cc: Tom Lane <tgl@sss.pgh.pa.us>, pgsql-docs <pgsql-docs@postgresql.org>
In-Reply-To: <1305893700-sup-7077@alvh.no-ip.org>
References: <1305841740.3952.32.camel@vanquo.pezone.net>
	<8242.1305892618@sss.pgh.pa.us>  <1305893700-sup-7077@alvh.no-ip.org>
Content-Type: text/plain; charset="UTF-8"
Date: Wed, 01 Jun 2011 22:28:22 +0300
Message-ID: <1306956502.2279.6.camel@vanquo.pezone.net>
Mime-Version: 1.0
Content-Transfer-Encoding: 8bit

On fre, 2011-05-20 at 08:16 -0400, Alvaro Herrera wrote:
> > >       * Should we allow/use non-ASCII characters in the release
> notes?
> > >       * What encoding should the HISTORY file have?
> > 
> > Ideally "sure, if entity-ified", but I don't know what to do about
> > HISTORY.
> 
> Can we recode that to plain ascii?  I think iconv has a //TRANSLIT
> flag or something like that.

To make this work on FreeBSD, where we build the releases, we need to
use the following command:

 "/usr/bin/perl" -p -e 's/<H(1|2)$/<H\1 align=center/g' HISTORY.html | LC_ALL=en_US.ISO8859-1 lynx -force_html -dump -nolist -stdin | iconv -f latin1 -t us-ascii//TRANSLIT > HISTORY

This also works on Linux/glibc, but FreeBSD is a bit stricter/more
limited.  Not sure about other platforms, but I'd guess if they don't
have the required locales, they'd be no worse off than now anyway.

The results are reasonable.  It actually depends on the platform
what //TRANSLIT does, e.g. on FreeBSD ö -> "o, on Linux ö -> o.