Date: Mon, 18 Mar 2019 22:19:23 +0100
From: "Peter J. Holzer" <hjp-pgsql@hjp.at>
To: pgsql-general@lists.postgresql.org
Subject: Re: Facing issue in using special characters
Message-ID: <20190318211923.eaobabpawdcjm5fh@hjp.at>
References: 
 <VI1PR0701MB271817E2E03434C39C4EDFFAAE440@VI1PR0701MB2718.eurprd07.prod.outlook.com>
 <c69d480d-89c4-f2c6-83bb-d0202785211b@gusw.net>
 <B446C5BC-7195-4BA0-80E6-A15D5CBDF365@uab.edu>
MIME-Version: 1.0
Content-Type: multipart/signed; micalg=pgp-sha256;
	protocol="application/pgp-signature"; boundary="3rk3wjlethzajamu"
Content-Disposition: inline
In-Reply-To: <B446C5BC-7195-4BA0-80E6-A15D5CBDF365@uab.edu>
User-Agent: NeoMutt/20170113 (1.7.2)
Precedence: bulk


--3rk3wjlethzajamu
Content-Type: text/plain; charset=utf-8
Content-Disposition: inline
Content-Transfer-Encoding: quoted-printable

On 2019-03-17 15:01:40 +0000, Warner, Gary, Jr wrote:
> Many of us have faced character encoding issues because we are not in con=
trol
> of our input sources and made the common assumption that UTF-8 covers
> everything.

UTF-8 covers "everything" in the sense that there is a round-trip from
each character in every commonly-used charset/encoding to Unicode and
back.

The actual code may of course be different. For example, the =E2=82=AC sign=
 is
0xA4 in iso-8859-15, but U+20AC in Unicode. So you need an
encoding/decoding step.

And "commonly-used" means just that. Unicode covers a lot of character
sets, but it can't cover every character set ever invented (I invented
my own character sets when I was sixteen. Nobody except me ever used
them and they have long succumbed to bit rot).

> In my lab, as an example, some of our social media posts have included Za=
wGyi
> Burmese character sets rather than Unicode Burmese.  (Because Myanmar dev=
eloped
> technology In a closed to the world environment, they made up their own
> non-standard character set which is very common still in Mobile phones.).

I'd be surprised if there was a character set which is "very common in
Mobile phones", even in a relatively poor country like Myanmar. Does
ZawGyi actually include characters which aren't in Unicode are are they
just encoded differently?

        hp

--=20
   _  | Peter J. Holzer    | we build much bigger, better disasters now
|_|_) |                    | because we have much more sophisticated
| |   | hjp@hjp.at         | management tools.
__/   | http://www.hjp.at/ | -- Ross Anderson <https://www.edge.org/>

--3rk3wjlethzajamu
Content-Type: application/pgp-signature; name="signature.asc"

-----BEGIN PGP SIGNATURE-----

iQIzBAABCAAdFiEETtJbRjyPwVTYGJ5k8g5IURL+KF0FAlyQC1YACgkQ8g5IURL+
KF2yXg/8D5SZrQgrFeeEcLDpwfqcCSfrJ4S29pblZZ2QMxKmKBHg2oxnJcS783uq
VUHaTxIBRXs0x2xqhefAj5i9qjXDX6T/XJp9xdmNGnh5j6cpybWA39znFG9ESGXr
+AmCpa705RvHpyWPZSH/cHDDPvHfGdXsgVkNdxltosrvgvJ3Mc6sSSCxc0UgV8ch
AIetVGudW3sxVfdP3w/ltjuKUIn9TV0jOo6FIOSwThv38B1Eu4E4pGleX50z9pm4
Nk3Clvsueygs81IbAms/oY5RFMIkFq5DfHKpfOhhKpbUaZjo87wSsRa8rTmlrq10
ym/hkzNCOuu7squVQ63Mu6pM09awpJQE6DWW0p/CcEj1t/KdCcXKfuRHm1SfrZbK
o9WD0upaO3NY7bhOcM6pVNuUlrqXxbh8S80ZLGtuiX8p4uQR84a37aSvHL7YGcF3
HCoI+5VVmKbhLQ3ADQLvhM/Yj/DT2VefiP+n692NzBbPZgfk74D1gfB5xDWoJIJK
7SjlRB+IhgC5tXv26d0HQiAN9blPt1yjB3oEBKBAb4YcBU60WQRxHGvB7gpaTRL3
Hwwk8BnKK7fQreuX7iZaFNMdWpv7GOakjQNAZ8cCj73tVt3xJSuPwdkljvCoZM3l
zIdalQ+Q9Pznz64/eYZthjT7s9jI3rUVu5Y9vL+ce98y8BehKnQ=
=7v46
-----END PGP SIGNATURE-----

--3rk3wjlethzajamu--