MIME-version: 1.0
Content-type: multipart/alternative;
 boundary="Boundary_(ID_SccWcICTpPGBBMQY4HLaUw)"
From: fortin.christian@videotron.ca
To: Craig Ringer <craig@2ndquadrant.com>
Cc: Andrew Dunstan <andrew@dunslane.net>, pgsql-translators@postgresql.org,
 PostgreSQL Hackers <pgsql-hackers@postgresql.org>
Message-id: <747095644b99f.563141a9@videotron.ca>
Date: Wed, 28 Oct 2015 21:44:09 -0400
Content-language: fr
Subject: Re : Re: Re : Re: UTF-32 support in PostgreSQL ?
In-reply-to: <7310b2824e3f9.563179b7@videotron.ca>
References: <7310b292c0558.562afaab@videotron.ca>
 <7310dc30c2b71.562afae8@videotron.ca> <7390a359c6c9c.562afb25@videotron.ca>
 <7310a455c4445.562ac2f1@videotron.ca> <562E9241.1080507@dunslane.net>
 <73f0e719d0e30.562e9c08@videotron.ca> <7440ac47d28b0.562e9c45@videotron.ca>
 <7430acc9d1847.562e9c82@videotron.ca> <7270ae0fd7bf8.562e9cbf@videotron.ca>
 <73a0e3fdd5f4c.562e9cfc@videotron.ca> <7310adbfd20ec.562e9d3a@videotron.ca>
 <7460e6e6d633b.562e9d77@videotron.ca> <74309713d4d45.562e6560@videotron.ca>
 <CAMsr+YEZ57vcqEMFLBxjDxmz5O+h69-QfPYVxjAz2RPT_mycbg@mail.gmail.com>
 <7300e9c549171.5631765d@videotron.ca> <7340e24249fdf.56317699@videotron.ca>
 <7430f0ac4afda.563176d7@videotron.ca> <746080524ffaa.56317715@videotron.ca>
 <7360a8134f954.56317752@videotron.ca> <72f0cf044df14.56317790@videotron.ca>
 <7330c5474d021.563177cd@videotron.ca> <7470b4ac4d9af.5631780b@videotron.ca>
 <7430cfc24cee9.56317848@videotron.ca> <72f0c9e34d9e1.56317885@videotron.ca>
 <7360cc8d4b969.563178c3@videotron.ca> <73708983480a6.56317900@videotron.ca>
 <7450c3c04a838.5631793d@videotron.ca> <7310b2824e3f9.563179b7@videotron.ca>
Precedence: bulk
Sender: pgsql-hackers-owner@postgresql.org

This is a multi-part message in MIME format.

--Boundary_(ID_SccWcICTpPGBBMQY4HLaUw)
Content-type: text/plain; charset=iso-8859-1
Content-transfer-encoding: quoted-printable
Content-disposition: inline

Now I received the authorization to give you an answer to the WHY questi=
on!
Because basicly=2C this project is classified TOP SECTRET=2E

Well=2C we know then we have no real avantage to use UTF-32 in comparais=
on to UTF-8=2E
But we need to establish a gateway between two huge networks=2E

One network is Internet=2C the other is =2E=2E=2E named it extra-Interne=
t=2E
Extra-Internet is older then the Internet that you already know=2E
It don=27t use IP protocol=2C but use a 32 bit per character encoding=2E=

This 32 bit encoding is not UTF-32=2C but supports 40 languages=2E Langu=
ages which are not include in UTF-32=2E
The language which have the less characters=2C use 100 characters=2E
The bigger alphabet have 10000 characters=2E
The most used language has 500 characters=2E

This extra-internet is as big as the actual Internet that you know=2E
This extra-Internet has not been built by USA=2C but by an other country=
=2E
Well=2C I try to convince peoples to use UTF-32=2E
I will need to ask to UNICODE to integrate the foreign 32 bits encoding =
in the future release of UTF-32=2E
And ask to the extra-internet authority=2C to integrate the UTF-32 in th=
ere standard 32 bits encoding=2E
I request to IETF to support UTF-32 int IPv6=2E I asked to w3=2Eorg to s=
upport UTF-32 in the future HTML format=2E
I plan to propose to the extra-Internet autority to upgrade to IPv6=2E
They actualy have problems with the availability of address=2C like we h=
ave with IPv4=2E The protocol they use is very basic=2C more basic than =
IPv4=2E And fail very often=2E

Well=2C hope it give answer to our question=2E

Le 26/10/15=2C Craig Ringer  =3Ccraig=402ndquadrant=2Ecom=3E a =E9crit =3A=

=3E On 27 October 2015 at 05=3A39=2C =3Cfortin=2Echristian=40videotron=2E=
ca=3E wrote=3A
=3E =

=3E =3E I mean for ALL=2C data stored=2C source code=2C and translation =
files=2E
=3E =3E For source code=2C I think then GCC must support UTF-32 before=2E=

=3E =

=3E Why=3F
=3E =

=3E UTF-32 is an incredibly inefficient way to store text that=27s
=3E predominantly or entirely within the 7-bit ASCII space=2E UTF-8 is a=

=3E much better way to handle it=2E
=3E =

=3E Anyway=2C while gcc supports sources encoded in utf-8 just fine=2C i=
t=27s
=3E more typical to represent chars using byte escapes so that people wi=
th
=3E misconfigured text editors don=27t mangle them=2E It does not suppor=
t
=3E utf-8 identifiers (variable names=2C function names=2C etc) containi=
ng
=3E characters outside the 7-bit ASCII space=2C but you can work around =
it
=3E with UCN if you need to=3B see the FAQ=3A
=3E =

=3E https=3A//gcc=2Egnu=2Eorg/wiki/FAQ=23What=5Fis=5Fthe=5Fstatus=5Fof=5F=
adding=5Fthe=5FUTF-8=5Fsupport=5Ffor=5Fidentifier=5Fnames=5Fin=5FGCC=2E3=
F
=3E =

=3E I don=27t think the PostgreSQL project is likely to accept patches u=
sing
=3E characters outside the 7-bit ascii space in the near future=2C as
=3E compiler and text editor support is unfortunately still too primitiv=
e=2E
=3E We support a variety of legacy platforms and toolchains=2C many of w=
hich
=3E won=27t cope at all=2E There isn=27t a pressing reason=2C since at t=
he user
=3E level the support for a wide variety of charsets (including all
=3E characters in the UTF-32 space) is already present=2E
=3E =

=3E I am aware this is a form of English-language privilege=2E Of course=

=3E it=27s easy for me as an English first-language speaker to say =22oh=
=2C we
=3E don=27t need support for your language in the code=22=2E It=27s also=
 practical
=3E though - code in a variety of languages=2C so that no one person can=

=3E read or understand all of it=2C is not maintainable in the long term=
=2E
=3E Especially when people join and leave the project=2E It=27s the same=

=3E reason the project is picky about introducing new programming
=3E languages=2C even though it might be nice to be able to write parts =
of
=3E the system in Python=2C parts in Haskell=2C etc=2E
=3E =

=3E So I don=27t think we need UTF-32 source code support=2C or even ful=
l
=3E UTF-8 source code support=2C because even if we had it we probably
=3E wouldn=27t use it=2E
=3E =

=3E =

=3E =3E I sent an e-mail to Oracle to see what they tink about this huge=
 idea=2E
=3E =

=3E I don=27t understand how this is a huge idea=2E The representation o=
f the
=3E characters doesn=27t matter=2C so long as the DB can represent the f=
ull
=3E character suite=2E Right=3F
=3E =

=3E =3E Well=2C I know it=27s not efficient space wise=2C but this in th=
e only way that we
=3E =3E can deployed worldwide=2E
=3E =

=3E UTF-8 is widely used worldwide and covers the full Unicode 32-bit co=
de space=2E
=3E =

=3E I wonder if you are misunderstanding UTF-8 vs UCS-2 vs UTF-16 vs UTF=
-32=2E
=3E =

=3E UTF-8 is an encoding that can represent the full 32-bit Unicode spac=
e
=3E using escape sequences=2E It is endianness-independent=2E One charac=
ter is
=3E a variable number of bytes=2C so lookups to find the n=27th characte=
r=2C
=3E substring operations=2C etc are a bit ugly=2E UTF-8 is the character=
 set
=3E used by most UNIX APIs=2E
=3E =

=3E UCS-2 is a legacy encoding that can represent the lower 16 bits of t=
he
=3E Unicode space=2E It cannot represent the full 32-bit Unicode space=2E=
 It
=3E has two different forms=2C little-endian and big-endian=2C so you ha=
ve to
=3E include a marker to say which is which=2C or be careful about handli=
ng
=3E it in your code=2E It=27s easy to do n=27th character lookups=2C sub=
strings=2C
=3E etc=2E
=3E =

=3E UTF-16 is like UCS-2=2C but adds UTF-8-like escape sequences to hand=
le
=3E the high 16 bits of the 32-bit Unicode space=2E It combines the wors=
t
=3E features of UTF-8 and UCS-2=2E UTF-16 is the character set used by
=3E Windows APIs and the ICU library=2E
=3E =

=3E UTF-32 (UCS-4) is much like UCS-2=2C but uses 4 bytes per character =
to
=3E represent the full Unicode character set=2E The downside is that it =
uses
=3E a full 4 bytes for every character=2C even when only one byte would =
be
=3E needed if you were using utf-8=2E It=27s easy to do substrings and n=
=27th
=3E character lookups=2E UCS-4 is horrible on CPU cache and memory=2E Fe=
w APIs
=3E use native UTF-32=2E
=3E =

=3E So we already support one of the best text encodings available=2E
=3E =

=3E We could add support for using UTF-16 and UTF-32 as the
=3E client=5Fencoding on the wire=2E But really=2C the client applicatio=
n can
=3E convert between the protocol=27s UTF-8 and whatever it wants to use
=3E internally=3B there=27s no benefit to using UTF-16 or UTF-32 on the =
wire=2C
=3E and it=27d be a lot slower=2E Especially without protocol compressio=
n=2E
=3E =

=3E So can you explain why you believe UTF-32 support is necessary=3F
=3E =

=3E =A0Craig Ringer http=3A//www=2E2ndQuadrant=2Ecom/
=3E =A0PostgreSQL Development=2C 24x7 Support=2C Training =26 Services
=3E =

=3E 

--Boundary_(ID_SccWcICTpPGBBMQY4HLaUw)
Content-type: text/html; charset=iso-8859-1
Content-transfer-encoding: quoted-printable
Content-disposition: inline

Now I received the authorization to give you an answer to the WHY questi=
on!=3Cbr /=3EBecause basicly=2C this project is classified TOP SECTRET=2E=
=3Cbr /=3E=3Cbr /=3EWell=2C we know then we have no real avantage to use=
 UTF-32 in comparaison to UTF-8=2E=3Cbr /=3EBut we need to establish a g=
ateway between two huge networks=2E=3Cbr /=3E=3Cbr /=3EOne network is In=
ternet=2C the other is =2E=2E=2E named it=A0 extra-Internet=2E=3Cbr /=3E=
Extra-Internet is older then the Internet that you already know=2E=3Cbr =
/=3EIt don=27t use IP protocol=2C but use a 32 bit per character encodin=
g=2E=3Cbr /=3EThis 32 bit encoding is not UTF-32=2C but supports 40 lang=
uages=2E Languages which are not include in UTF-32=2E=3Cbr /=3EThe langu=
age which have the less characters=2C use 100 characters=2E=3Cbr /=3EThe=
 bigger alphabet have 10000 characters=2E=3Cbr /=3EThe most used languag=
e has 500 characters=2E=3Cbr /=3E=3Cbr /=3EThis extra-internet is as big=
 as the actual Internet that you know=2E=3Cbr /=3EThis extra-Internet ha=
s not been built by USA=2C but by an other country=2E=3Cbr /=3EWell=2C I=
 try to convince peoples to use UTF-32=2E=3Cbr /=3EI will need to ask to=
 UNICODE to integrate the foreign 32 bits encoding in the future release=
 of UTF-32=2E=3Cbr /=3EAnd ask to the extra-internet authority=2C to int=
egrate the UTF-32 in there standard 32 bits encoding=2E=3Cbr /=3EI reque=
st to IETF to support UTF-32 int IPv6=2E I asked to w3=2Eorg to support =
UTF-32 in the future HTML format=2E=3Cbr /=3EI plan to propose to the ex=
tra-Internet autority to upgrade to IPv6=2E=3Cbr /=3EThey actualy have p=
roblems with the availability of address=2C like we have with IPv4=2E Th=
e protocol they use is very basic=2C more basic than IPv4=2E And fail ve=
ry often=2E=3Cbr /=3E=3Cbr /=3EWell=2C hope it give answer to our questi=
on=2E=3Cbr /=3E=3Cbr /=3E=3Cspan=3ELe 26/10/15=2C =3Cb class=3D=22name=22=
=3ECraig Ringer =3C/b=3E =26lt=3Bcraig=402ndquadrant=2Ecom=26gt=3B a =E9=
crit =3A=3C/span=3E=3Cblockquote cite=3D=22mid=3ACAMsr+YEZ57vcqEMFLBxjDx=
mz5O+h69-QfPYVxjAz2RPT=5Fmycbg=40mail=2Egmail=2Ecom=22 class=3D=22iwcQuo=
te=22 style=3D=22border-left=3A 1px solid =2300F=3B padding-left=3A 13px=
=3B margin-left=3A 0=3B=22 type=3D=22cite=22=3E=3Cdiv class=3D=22mimepar=
t text plain=22=3EOn 27 October 2015 at 05=3A39=2C=A0 =26lt=3Bfortin=2Ec=
hristian=40videotron=2Eca=26gt=3B wrote=3A=3Cbr /=3E=3Cbr /=3E=26gt=3B I=
 mean for ALL=2C data stored=2C source code=2C and translation files=2E=3C=
br /=3E=26gt=3B For source code=2C I think then GCC must support UTF-32 =
before=2E=3Cbr /=3E=3Cbr /=3EWhy=3F=3Cbr /=3E=3Cbr /=3EUTF-32 is an incr=
edibly inefficient way to store text that=27s=3Cbr /=3Epredominantly or =
entirely within the 7-bit ASCII space=2E UTF-8 is a=3Cbr /=3Emuch better=
 way to handle it=2E=3Cbr /=3E=3Cbr /=3EAnyway=2C while gcc supports sou=
rces encoded in utf-8 just fine=2C it=27s=3Cbr /=3Emore typical to repre=
sent chars using byte escapes so that people with=3Cbr /=3Emisconfigured=
 text editors don=27t mangle them=2E It does not support=3Cbr /=3Eutf-8 =
identifiers (variable names=2C function names=2C etc) containing=3Cbr /=3E=
characters outside the 7-bit ASCII space=2C but you can work around it=3C=
br /=3Ewith UCN if you need to=3B see the FAQ=3A=3Cbr /=3E=3Cbr /=3E=3Ca=
 href=3D=22https=3A//gcc=2Egnu=2Eorg/wiki/FAQ=23What=5Fis=5Fthe=5Fstatus=
=5Fof=5Fadding=5Fthe=5FUTF-8=5Fsupport=5Ffor=5Fidentifier=5Fnames=5Fin=5F=
GCC=2E3F=22 target=3D=22l=22=3Ehttps=3A//gcc=2Egnu=2Eorg/wiki/FAQ=23What=
=5Fis=5Fthe=5Fstatus=5Fof=5Fadding=5Fthe=5FUTF-8=5Fsupport=5Ffor=5Fident=
ifier=5Fnames=5Fin=5FGCC=2E3F=3C/a=3E=3Cbr /=3E=3Cbr /=3EI don=27t think=
 the PostgreSQL project is likely to accept patches using=3Cbr /=3Echara=
cters outside the 7-bit ascii space in the near future=2C as=3Cbr /=3Eco=
mpiler and text editor support is unfortunately still too primitive=2E=3C=
br /=3EWe support a variety of legacy platforms and toolchains=2C many o=
f which=3Cbr /=3Ewon=27t cope at all=2E There isn=27t a pressing reason=2C=
 since at the user=3Cbr /=3Elevel the support for a wide variety of char=
sets (including all=3Cbr /=3Echaracters in the UTF-32 space) is already =
present=2E=3Cbr /=3E=3Cbr /=3EI am aware this is a form of English-langu=
age privilege=2E Of course=3Cbr /=3Eit=27s easy for me as an English fir=
st-language speaker to say =26quot=3Boh=2C we=3Cbr /=3Edon=27t need supp=
ort for your language in the code=26quot=3B=2E It=27s also practical=3Cb=
r /=3Ethough - code in a variety of languages=2C so that no one person c=
an=3Cbr /=3Eread or understand all of it=2C is not maintainable in the l=
ong term=2E=3Cbr /=3EEspecially when people join and leave the project=2E=
 It=27s the same=3Cbr /=3Ereason the project is picky about introducing =
new programming=3Cbr /=3Elanguages=2C even though it might be nice to be=
 able to write parts of=3Cbr /=3Ethe system in Python=2C parts in Haskel=
l=2C etc=2E=3Cbr /=3E=3Cbr /=3ESo I don=27t think we need UTF-32 source =
code support=2C or even full=3Cbr /=3EUTF-8 source code support=2C becau=
se even if we had it we probably=3Cbr /=3Ewouldn=27t use it=2E=3Cbr /=3E=
=3Cbr /=3E=3Cbr /=3E=26gt=3B I sent an e-mail to Oracle to see what they=
 tink about this huge idea=2E=3Cbr /=3E=3Cbr /=3EI don=27t understand ho=
w this is a huge idea=2E The representation of the=3Cbr /=3Echaracters d=
oesn=27t matter=2C so long as the DB can represent the full=3Cbr /=3Echa=
racter suite=2E Right=3F=3Cbr /=3E=3Cbr /=3E=26gt=3B Well=2C I know it=27=
s not efficient space wise=2C but this in the only way that we=3Cbr /=3E=
=26gt=3B can deployed worldwide=2E=3Cbr /=3E=3Cbr /=3EUTF-8 is widely us=
ed worldwide and covers the full Unicode 32-bit code space=2E=3Cbr /=3E=3C=
br /=3EI wonder if you are misunderstanding UTF-8 vs UCS-2 vs UTF-16 vs =
UTF-32=2E=3Cbr /=3E=3Cbr /=3EUTF-8 is an encoding that can represent the=
 full 32-bit Unicode space=3Cbr /=3Eusing escape sequences=2E It is endi=
anness-independent=2E One character is=3Cbr /=3Ea variable number of byt=
es=2C so lookups to find the n=27th character=2C=3Cbr /=3Esubstring oper=
ations=2C etc are a bit ugly=2E UTF-8 is the character set=3Cbr /=3Eused=
 by most UNIX APIs=2E=3Cbr /=3E=3Cbr /=3EUCS-2 is a legacy encoding that=
 can represent the lower 16 bits of the=3Cbr /=3EUnicode space=2E It can=
not represent the full 32-bit Unicode space=2E It=3Cbr /=3Ehas two diffe=
rent forms=2C little-endian and big-endian=2C so you have to=3Cbr /=3Ein=
clude a marker to say which is which=2C or be careful about handling=3Cb=
r /=3Eit in your code=2E It=27s easy to do n=27th character lookups=2C s=
ubstrings=2C=3Cbr /=3Eetc=2E=3Cbr /=3E=3Cbr /=3EUTF-16 is like UCS-2=2C =
but adds UTF-8-like escape sequences to handle=3Cbr /=3Ethe high 16 bits=
 of the 32-bit Unicode space=2E It combines the worst=3Cbr /=3Efeatures =
of UTF-8 and UCS-2=2E UTF-16 is the character set used by=3Cbr /=3EWindo=
ws APIs and the ICU library=2E=3Cbr /=3E=3Cbr /=3EUTF-32 (UCS-4) is much=
 like UCS-2=2C but uses 4 bytes per character to=3Cbr /=3Erepresent the =
full Unicode character set=2E The downside is that it uses=3Cbr /=3Ea fu=
ll 4 bytes for every character=2C even when only one byte would be=3Cbr =
/=3Eneeded if you were using utf-8=2E It=27s easy to do substrings and n=
=27th=3Cbr /=3Echaracter lookups=2E UCS-4 is horrible on CPU cache and m=
emory=2E Few APIs=3Cbr /=3Euse native UTF-32=2E=3Cbr /=3E=3Cbr /=3ESo we=
 already support one of the best text encodings available=2E=3Cbr /=3E=3C=
br /=3EWe could add support for using UTF-16 and UTF-32 as the=3Cbr /=3E=
client=5Fencoding on the wire=2E But really=2C the client application ca=
n=3Cbr /=3Econvert between the protocol=27s UTF-8 and whatever it wants =
to use=3Cbr /=3Einternally=3B there=27s no benefit to using UTF-16 or UT=
F-32 on the wire=2C=3Cbr /=3Eand it=27d be a lot slower=2E Especially wi=
thout protocol compression=2E=3Cbr /=3E=3Cbr /=3ESo can you explain why =
you believe UTF-32 support is necessary=3F=3Cbr /=3E=3Cbr /=3E=A0Craig R=
inger=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0 =3Ca href=3D=
=22http=3A//www=2E2ndQuadrant=2Ecom/=22 target=3D=22l=22=3Ehttp=3A//www=2E=
2ndQuadrant=2Ecom/=3C/a=3E=3Cbr /=3E=A0PostgreSQL Development=2C 24x7 Su=
pport=2C Training =26amp=3B Services=3Cbr /=3E=3C/div=3E=3C/blockquote=3E

--Boundary_(ID_SccWcICTpPGBBMQY4HLaUw)--