Date: Thu, 2 Nov 2023 17:38:47 -0500
From: Nico Williams <nico@cryptonector.com>
To: Robert Haas <robertmhaas@gmail.com>
Cc: Isaac Morland <isaac.morland@gmail.com>, Jeff Davis <pgsql@j-davis.com>,
	Chapman Flack <chap@anastigmatix.net>, pgsql-hackers@postgresql.org
Subject: Re: Pre-proposal: unicode normalized text
Message-ID: <ZUQk99GFe787WC/z@ubby21>
References: 
 <CAMsGm5eyegAkb2Eq+8d2j+BxDnSZXH0AUciwi6rA60V_ft=1dw@mail.gmail.com>
 <94a0e631b787d56fec1b17e03fe045b5e14907c1.camel@j-davis.com>
 <CAMsGm5f8k4_C6VuerSbF2gXeVwD9kMZQk43kO85Oi4oGVL7EMA@mail.gmail.com>
 <CA+TgmobAxizsgjxvZdEQxjEs6RA3qu7JLti_LdXtaXODJoWzNw@mail.gmail.com>
 <ZR8LLrk9AJVxEFbX@ubby21>
 <CA+TgmoaF_KHkgcisLFaKzR_husGtAOnjEez9biH2QHyX-4dAyA@mail.gmail.com>
 <ZSBGJalw0ytQ63Sz@ubby21>
 <CA+TgmoYGrrOOahgxNO=W7_cvG-g--WVtuK6SYkWa+PEDdTOUqw@mail.gmail.com>
 <ZSBRKFJ+gy5iUXxa@ubby21>
 <CA+TgmoavAkU0kqVc7mBMsdKDsj7Bq9moPJQAkW6mSw3GgpGriw@mail.gmail.com>
MIME-Version: 1.0
Content-Type: text/plain; charset=us-ascii
Content-Disposition: inline
In-Reply-To: 
 <CA+TgmoavAkU0kqVc7mBMsdKDsj7Bq9moPJQAkW6mSw3GgpGriw@mail.gmail.com>
Archived-At: <https://www.postgresql.org/message-id/ZUQk99GFe787WC/z%40ubby21>
Precedence: bulk

On Fri, Oct 06, 2023 at 02:37:06PM -0400, Robert Haas wrote:
> > Sure, because TEXT in PG doesn't have codeset+encoding as part of it --
> > it's whatever the database's encoding is.  Collation can and should be a
> > porperty of a column, since for Unicode it wouldn't be reasonable to
> > make that part of the type.  But codeset+encoding should really be a
> > property of the type if PG were to support more than one.  IMO.
> 
> No, what I mean is, you can't just be like "oh, the varlena will be
> different in memory than on disk" as if that were no big deal.

It would have to be the same in memory as on disk, indeed, but you might
need new types in C as well for that.

> I agree that, as an alternative to encoding being a column property,
> it could instead be completely a type property, meaning that if you
> want to store, say, LATIN1 text in your UTF-8 database, you first
> create a latint1text data type and then use it, rather than, as in the
> model I proposed, creating a text column and then applying a setting
> like ENCODING latin1 to it. I think that there might be some problems

Yes, that was the idea.

> with that model, but it could also have some benefits. [...]

Mainly, I think, whether you want PG to do automatic codeset conversions
(ugly and problematic) or not, like for when using text functions.

Automatic codeset conversions are problematic because a) it can be lossy
(so what to do when it is?) and b) automatic type conversions can be
surprising.

Ultimately the client would have to do its own codeset conversions, if
it wants them, or treat text in codesets other than its local one as
blobs and leave it for a higher app layer to deal with.

I wouldn't want to propose automatic codeset conversions.  If you'd want
that then you might as well declare it has to all be UTF-8 and say no to
any other codesets.

> But, even if we were all convinced that this kind of feature was good
> to add, I think it would almost certainly be wrong to invent new
> varlena features along the way.

Yes.

Nico
--