From: Nico Williams <nico@cryptonector.com>
To: Chapman Flack <chap@anastigmatix.net>
Cc: Jeff Davis <pgsql@j-davis.com>
Cc: Robert Haas <robertmhaas@gmail.com>
Cc: pgsql-hackers@postgresql.org
Subject: Re: Pre-proposal: unicode normalized text
Date: Wed, 4 Oct 2023 17:15:47 -0500
Message-ID: <ZR3kE0didxGGfSif@ubby21> (raw)
In-Reply-To: <02d05bc98ca5b2d8ab38fec5fe5b7625@anastigmatix.net>
References: <f30b58657ceb71d5be032decf4058d454cc1df74.camel@j-davis.com>
<ZR2fnTPDwPb0IamC@ubby21>
<CA+TgmoY6vNbDoBugDT=3rdJ-diyLw-kQ9sxB2JXenv48xV_tVQ@mail.gmail.com>
<43864caecd65d02081d623d05ccc1683@anastigmatix.net>
<a415e27830b8c94cea1b1c4bd60d254f0f397866.camel@j-davis.com>
<02d05bc98ca5b2d8ab38fec5fe5b7625@anastigmatix.net>
On Wed, Oct 04, 2023 at 05:32:50PM -0400, Chapman Flack wrote:
> Well, for what reason does anybody run PG now with the encoding set
> to anything besides UTF-8? I don't really have my finger on that pulse.
Because they still have databases that didn't use UTF-8 10 or 20 years
ago that they haven't migrated to UTF-8?
It's harder to think of why one might _want_ to store text in any
encoding other than UTF-8 for _new_ databases.
Though too there's no reason that it should be impossible other than
lack of developer interest: as long as text is tagged with its encoding,
it should be possible to store text in any number of encodings.
> Could it be that it bloats common strings in their local script, and
> with enough of those to store, it could matter to use the local
> encoding that stores them more economically?
UTF-8 bloat is not likely worth the trouble. UTF-8 is only clearly
bloaty when compared to encodings with 1-byte code units, like
ISO-8859-*. For CJK UTF-8 is not much more bloaty than native
non-Unicode encodings like SHIFT_JIS.
UTF-8 is not much bloatier than UTF-16 in general either.
Bloat is not really a good reason to avoid Unicode or any specific TF.
> Also, while any Unicode transfer format can encode any Unicode code
> point, I'm unsure whether it's yet the case that {any Unicode code
> point} is a superset of every character repertoire associated with
> every non-Unicode encoding.
It's not always been the case that Unicode is a strict superset of all
currently-in-use human scripts. Making Unicode a strict superset of all
currently-in-use human scripts seems to be the Unicode Consortium's aim.
I think you're asking why not just use UTF-8 for everything, all the
time. It's a fair question. I don't have a reason to answer in the
negative (maybe someone else does). But that doesn't mean that one
couldn't want to store text in many encodings (e.g., for historical
reasons).
Nico
--
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Reply to all the recipients using the --to and --cc options:
reply via email
To: pgsql-hackers@postgresql.org
Cc: nico@cryptonector.com, chap@anastigmatix.net, pgsql@j-davis.com, robertmhaas@gmail.com
Subject: Re: Pre-proposal: unicode normalized text
In-Reply-To: <ZR3kE0didxGGfSif@ubby21>
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
This inbox is served by DDX for PostgreSQL; see mirroring instructions
for how to clone and mirror all data and code used for this inbox