From: Nico Williams <nico@cryptonector.com>
To: Jeff Davis <pgsql@j-davis.com>
Cc: pgsql-hackers@postgresql.org
Subject: Re: Pre-proposal: unicode normalized text
Date: Tue, 3 Oct 2023 18:01:16 -0500
Message-ID: <ZRydPDKCwiWnnk6X@ubby21> (raw)
In-Reply-To: <a41cfc7c3fcd00ed5d1008cc6fa810340f35fd47.camel@j-davis.com>
References: <f30b58657ceb71d5be032decf4058d454cc1df74.camel@j-davis.com>
<ZRsnnGfQ701AA091@ubby21>
<23edd434b490f74a82b564b8027b6a52059f8de7.camel@j-davis.com>
<ZRx2VcsWomTBcE+L@ubby21>
<a41cfc7c3fcd00ed5d1008cc6fa810340f35fd47.camel@j-davis.com>
On Tue, Oct 03, 2023 at 03:34:44PM -0700, Jeff Davis wrote:
> On Tue, 2023-10-03 at 15:15 -0500, Nico Williams wrote:
> > Ugh, My client is not displying 'a' correctly
>
> Ugh. Is that an argument in favor of normalization or against?
Heheh, well, it's an argument in favor of more software getting this
right (darn it).
It's also an argument for building a time machine so HFS+ can just
always have used NFC. But the existence of UTF-16 is proof that time
machines don't exist (or that only bad actors have them).
> I've also noticed that some fonts render the same character a bit
> differently depending on the constituent code points. For instance, if
> the accent is its own code point, it seems to be more prominent than if
> a single code point represents both the base character and the accent.
> That seems to be a violation, but I can understand why that might be
> useful.
Yes, that happens. Did you know that the ASCII character set was
designed with overstrike in mind for typing of accented Latin
characters? Unicode combining sequences are kinda like that, but more
complex.
Yes, the idea really was that you could write a<BS>' (or '<BS>a) to get รก.
That's how people did it with typewriters anyways.
> > Almost every Latin input mode out there produces precomposed
> > characters and so they effectively produce NFC.
>
> The problem is not the normal case, the problem will be things like
> obscure input methods, some kind of software that's being too clever,
> or some kind of malicious user trying to confuse the database.
_HFS+ enters the chat_
> > That means that indices
> > need to normalize strings, but tables need to store unnormalized
> > strings.
>
> That's an interesting idea. Would the equality operator normalize
> first, or are you saying that the index would need to recheck the
> results?
You can optimize this to avoid having to normalize first. Most strings
are not equal, and they tend to differ early. And most strings will
likely be ASCII-mostly or in the same form anyways. So you can just
walk a cursor down each string looking at two bytes, and if they are
both ASCII then you move each cursor forward by one byte, and if then
are not both ASCII then you take a slow path where you normalize one
grapheme cluster at each cursor (if necessary) and compare that. (ZFS
does this.)
You can also assume ASCII-mostly, load as many bits of each string
(padding as needed) as will fit in SIMD registers, compare and check
that they're all ASCII, and if not then jump to the slow path.
You can also normalize one grapheme cluster at a time when hashing
(e.g., for hash indices), thus avoiding a large allocation if the string
is large.
Nico
--
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Reply to all the recipients using the --to and --cc options:
reply via email
To: pgsql-hackers@postgresql.org
Cc: nico@cryptonector.com, pgsql@j-davis.com
Subject: Re: Pre-proposal: unicode normalized text
In-Reply-To: <ZRydPDKCwiWnnk6X@ubby21>
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
This inbox is served by DDX for PostgreSQL; see mirroring instructions
for how to clone and mirror all data and code used for this inbox