Subject: Re: Unicode normalization SQL functions
To: John Naylor <john.naylor@2ndquadrant.com>
References: <6d55838c-9353-486f-a508-77ee07d59667@manitou-mail.org>
 <620b155b-a89c-d8e0-3cbb-315c3ca538f4@2ndquadrant.com>
 <7052cc8f-0164-72a8-d9a4-fd32066c938e@2ndquadrant.com>
 <CACPNZCsiE6qN=yc-LBNLU5b9Hy06yagOXwsWGTJTvpXcrMHNTw@mail.gmail.com>
Cc: pgsql-hackers <pgsql-hackers@postgresql.org>,
 Daniel Verite <daniel@manitou-mail.org>, Andreas Karlsson <andreas@proxel.se>
From: Peter Eisentraut <peter.eisentraut@2ndquadrant.com>
Organization: 2ndQuadrant
Message-ID: <2d3e7d66-0f89-a663-280b-45d5b88b196b@2ndquadrant.com>
Date: Thu, 2 Apr 2020 09:51:19 +0200
User-Agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10.14; rv:52.0)
 Gecko/20100101 Thunderbird/52.9.1
MIME-Version: 1.0
In-Reply-To: 
 <CACPNZCsiE6qN=yc-LBNLU5b9Hy06yagOXwsWGTJTvpXcrMHNTw@mail.gmail.com>
Content-Type: text/plain; charset=utf-8; format=flowed
Content-Language: en-US
Content-Transfer-Encoding: 7bit
Precedence: bulk

On 2020-03-26 18:41, John Naylor wrote:
> We don't have a trie implementation in Postgres, but we do have a
> perfect hash implementation. Doing that would bring the tables back to
> 64 bits per entry, but would likely be noticeably faster than binary
> search. Since v4 has left out the biggest tables entirely, I think
> this might be worth a look for the smaller tables remaining.
> 
> In the attached v5, when building the hash tables, we sort the code
> points by NO/MAYBE, and store the index of the beginning of the NO
> block:

This is a valuable idea, but I fear it's a bit late now in this cycle. 
I have questions about some details.  For example, you mention that you 
had to fiddle with the hash seed.  How does that affect other users of 
PerfectHash?  What happens when we update Unicode data and the hash 
doesn't work anymore?  These discussions might derail this patch at this 
hour, so I have committed the previous patch.  We can consider your 
patch as a follow-up patch, either now or in the future.

 > Also, if we go with v4, I noticed the following test is present twice:
 >
 > +SELECT "normalize"('abc', 'def');  -- run-time error

I think this is correct.  The other test is for "is_normalized".

-- 
Peter Eisentraut              http://www.2ndQuadrant.com/
PostgreSQL Development, 24x7 Support, Remote DBA, Training & Services