Received: from malur.postgresql.org ([217.196.149.56])
 by arkaria.postgresql.org with esmtp (Exim 4.80) (envelope-from
 <pgsql-hackers-owner+M234067=pgsql-hackers-arkari@postgresql.org>)
 id 1VMpOr-0001Yj-H8
 for pgsql-hackers@arkaria.postgresql.org; Fri, 20 Sep 2013 01:20:49 +0000
Received: from localhost ([127.0.0.1] helo=postgresql.org)
 by malur.postgresql.org with smtp (Exim 4.80) (envelope-from
 <pgsql-hackers-owner+M234067=pgsql-hackers-arkari@postgresql.org>)
 id 1VMpOr-0007BR-0m
 for pgsql-hackers@arkaria.postgresql.org; Fri, 20 Sep 2013 01:20:49 +0000
Received: from magus.postgresql.org ([2a02:c0:301:0:ffff::29])
 by malur.postgresql.org with esmtp (Exim 4.80)
 (envelope-from <valgog@gmail.com>) id 1VMpOp-0007BI-Dq
 for pgsql-hackers@postgresql.org; Fri, 20 Sep 2013 01:20:47 +0000
Received: from mail-ie0-f170.google.com ([209.85.223.170])
 by magus.postgresql.org with esmtp (Exim 4.80)
 (envelope-from <valgog@gmail.com>)
 id 1VMpOh-0007jk-BO; Fri, 20 Sep 2013 01:20:46 +0000
Received: by mail-ie0-f170.google.com with SMTP id x13so17114399ief.1
 for <multiple recipients>; Thu, 19 Sep 2013 18:20:22 -0700 (PDT)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113;
 h=mime-version:in-reply-to:references:from:date:message-id:subject:to
 :cc:content-type;
 bh=dAiI07rMs5UvE7TYKyUpJWwRNw0lwejU2bs0cV8SNGw=;
 b=QysT9/PuTPoV+mGazXRYQWJkkmIM0NuCtt21CT99RcdYRwmTxLf9nI7HZ5Kda1Kp8V
 BAi3dJfbC8ygPwRpGOL3xpjhdQ+KuKC6U6JLCPbjTyUWh4inuwmIt5Q+72phZbHRncTx
 I5C1bYnTxLiTIDMFJN4kRvvKlAsrIfOQ/AeZurkjqho6U2TSGYe0jZ7G0MhXuwi+1Egj
 f0NmHkdqIH2m6LL2CbhYJbbfS28msPN/dIiM1nHyBALhnFJDQCisfa/v+juZFbN8hZB7
 9xAz7ipEPMx7MekcdZWX6QAkSeZs/6/9ASChblDAxNAfjJLWdtKOnv2uWNSJ7jvCOvuE
 TP+A==
X-Received: by 10.42.250.148 with SMTP id mo20mr2759305icb.34.1379640022678;
 Thu, 19 Sep 2013 18:20:22 -0700 (PDT)
MIME-Version: 1.0
Received: by 10.64.102.161 with HTTP; Thu, 19 Sep 2013 18:20:02 -0700 (PDT)
In-Reply-To: <37B76474BB3149FD841373E12E355851@maumau>
References: <A756FAD7EDC2E24F8CAB7E2F3B5375E918E2D5B8@FALEX03.au.fjanz.com>
 <522594E8.2050106@vmware.com>
 <A756FAD7EDC2E24F8CAB7E2F3B5375E918F332A6@FALEX03.au.fjanz.com>
 <904.1378304922@sss.pgh.pa.us> <B1A7485194DE4FDAB8FA781AFB570079@maumau>
 <CA+TgmobWv_96dSfSHv2ZJWSXAD=QTiBEKw-LT359sGArYe+AXQ@mail.gmail.com>
 <B00EACB87A2441069B4EE4DEA6D9282A@maumau>
 <CA+TgmoaT-h5C25LXd1DgfyiW2t9K92eVdVcJHC_1We0JpJFqgg@mail.gmail.com>
 <37B76474BB3149FD841373E12E355851@maumau>
From: Valentine Gogichashvili <valgog@gmail.com>
Date: Fri, 20 Sep 2013 03:20:02 +0200
Message-ID: 
 <CAP93muULVtyd-HQd=h2VOWzaPUrf2Z9efqXDJvmV0Xx3Auj16Q@mail.gmail.com>
Subject: Re: UTF8 national character data type support WIP patch and
 list of open issues.
To: MauMau <maumau307@gmail.com>, ishii@postgresql.org
Cc: Robert Haas <robertmhaas@gmail.com>, Tom Lane <tgl@sss.pgh.pa.us>,
 "Boguk, Maksym" <maksymb@fast.au.fujitsu.com>,
 Heikki Linnakangas <hlinnakangas@vmware.com>,
 PostgreSQL Hackers <pgsql-hackers@postgresql.org>
Content-Type: multipart/alternative; boundary=20cf300e4e517846af04e6c679b4
X-Pg-Spam-Score: -2.7 (--)
List-Archive: <http://archives.postgresql.org/pgsql-hackers>
List-Help: <mailto:majordomo@postgresql.org?body=help>
List-ID: <pgsql-hackers.postgresql.org>
List-Owner: <mailto:pgsql-hackers-owner@postgresql.org>
List-Post: <mailto:pgsql-hackers@postgresql.org>
List-Subscribe: <mailto:majordomo@postgresql.org?body=sub%20pgsql-hackers>
List-Unsubscribe: <mailto:majordomo@postgresql.org?body=unsub%20pgsql-hackers>
X-Mailing-List: pgsql-hackers
Precedence: bulk
Sender: pgsql-hackers-owner@postgresql.org

--20cf300e4e517846af04e6c679b4
Content-Type: text/plain; charset=UTF-8

Hi,


>  That may be what's important to you, but it's not what's important to
>> me.
>>
>
> National character types support may be important to some potential users
> of PostgreSQL and the popularity of PostgreSQL, not me.  That's why
> national character support is listed in the PostgreSQL TODO wiki.  We might
> be losing potential users just because their selection criteria includes
> national character support.
>
>
the whole NCHAR appeared as hack for the systems, that did not have it from
the beginning. It would not be needed, if all the text would be magically
stored in UNICODE or UTF from the beginning and idea of character would be
the same as an idea of a rune and not a byte.

PostgreSQL has a very powerful possibilities for storing any kind of
encoding. So maybe it makes sense to add the ENCODING as another column
property, the same way a COLLATION was added?

It would make it possible to have a database, that talks to the clients in
UTF8 and stores text and varchar data in the encoding that is the most
appropriate for the situation.

It will make it impossible (or complicated) to make the database have a
non-UTF8 default encoding (I wonder who should need that in this case), as
conversions will not be possible from the broader charsets into the default
database encoding.

One could define an additional DATABASE property like LC_ENCODING that
would work for the ENCODING property of a column like LC_COLLATE for
COLLATE property of a column.

Text operations should work automatically, as in memory all strings will be
converted to the database encoding.

This approach will also open a possibility to implement custom ENCODINGs
for the column data storage, like snappy compression or even BSON, gobs or
protbufs for much more compact type storage.

Regards,

-- Valentine Gogichashvili

--20cf300e4e517846af04e6c679b4
Content-Type: text/html; charset=UTF-8
Content-Transfer-Encoding: quoted-printable

<div dir=3D"ltr">Hi,=C2=A0<div><br><div class=3D"gmail_extra"><div class=3D=
"gmail_quote"><blockquote class=3D"gmail_quote" style=3D"margin:0px 0px 0px=
 0.8ex;border-left-width:1px;border-left-color:rgb(204,204,204);border-left=
-style:solid;padding-left:1ex">

<div class=3D"im"><br>
<blockquote class=3D"gmail_quote" style=3D"margin:0px 0px 0px 0.8ex;border-=
left-width:1px;border-left-color:rgb(204,204,204);border-left-style:solid;p=
adding-left:1ex">
That may be what&#39;s important to you, but it&#39;s not what&#39;s import=
ant to<br>
me.<br>
</blockquote>
<br></div>
National character types support may be important to some potential users o=
f PostgreSQL and the popularity of PostgreSQL, not me. =C2=A0That&#39;s why=
 national character support is listed in the PostgreSQL TODO wiki. =C2=A0We=
 might be losing potential users just because their selection criteria incl=
udes national character support.<div class=3D"im">

<br></div></blockquote><div><br></div><div>the whole NCHAR appeared as hack=
 for the systems, that did not have it from the beginning. It would not be =
needed, if all the text would be magically stored in UNICODE or UTF from th=
e beginning and idea of character would be the same as an idea of a rune an=
d not a byte.</div>

<div><br></div><div>PostgreSQL has a very powerful possibilities for storin=
g any kind of encoding. So maybe it makes sense to add the ENCODING as anot=
her column property, the same way a COLLATION was added?</div><div><br>

</div><div>It would make it possible to have a database, that talks to the =
clients in UTF8 and stores text and varchar data in the encoding that is th=
e most appropriate for the situation.</div><div><br></div><div>It will make=
 it impossible (or complicated) to make the database have a non-UTF8 defaul=
t encoding (I wonder who should need that in this case), as conversions wil=
l not be possible from the broader charsets into the default database encod=
ing.</div>

<div><br></div><div>One could define an additional DATABASE property like L=
C_ENCODING that would work for the ENCODING property of a column like LC_CO=
LLATE for COLLATE property of a column.</div><div><br></div><div>Text opera=
tions should work automatically, as in memory all strings will be converted=
 to the database encoding.</div>

<div><br></div><div><div>This approach will also open a possibility to impl=
ement custom ENCODINGs for the column data storage, like snappy compression=
 or even BSON, gobs or protbufs for much more compact type storage.</div>

</div><div><br></div><div>Regards,=C2=A0</div><div><br></div><div>-- Valent=
ine Gogichashvili</div></div></div></div></div>

--20cf300e4e517846af04e6c679b4--