Subject: Re: Unicode normalization SQL functions
From: Peter Eisentraut <peter.eisentraut@2ndquadrant.com>
To: pgsql-hackers <pgsql-hackers@postgresql.org>
References: <6d55838c-9353-486f-a508-77ee07d59667@manitou-mail.org>
 <620b155b-a89c-d8e0-3cbb-315c3ca538f4@2ndquadrant.com>
 <7052cc8f-0164-72a8-d9a4-fd32066c938e@2ndquadrant.com>
Cc: Daniel Verite <daniel@manitou-mail.org>,
 Andreas Karlsson <andreas@proxel.se>
Organization: 2ndQuadrant
Message-ID: <b93f79b0-7ea6-4807-e993-b0b3bbe8337a@2ndquadrant.com>
Date: Thu, 2 Apr 2020 09:45:09 +0200
User-Agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10.14; rv:52.0)
 Gecko/20100101 Thunderbird/52.9.1
MIME-Version: 1.0
In-Reply-To: <7052cc8f-0164-72a8-d9a4-fd32066c938e@2ndquadrant.com>
Content-Type: text/plain; charset=iso-8859-15; format=flowed
Content-Language: en-US
Content-Transfer-Encoding: 7bit
Precedence: bulk

On 2020-03-26 08:25, Peter Eisentraut wrote:
> On 2020-03-24 10:20, Peter Eisentraut wrote:
>> Now I have some concerns about the size of the new table in
>> unicode_normprops_table.h, and the resulting binary size.  At the very
>> least, we should probably make that #ifndef FRONTEND or something like
>> that so libpq isn't bloated by it unnecessarily.  Perhaps there is a
>> better format for that table?  Any ideas?
> 
> I have figured this out.  New patch is attached.
> 
> First, I have added #ifndef FRONTEND, as mentioned above, so libpq isn't
> bloated.  Second, I have changed the lookup structure to a bitfield, so
> each entry is only 32 bits instead of 64.  Third, I have dropped the
> quickcheck tables for the NFD and NFKD forms.  Those are by far the
> biggest tables, and you still get okay performance if you do the
> normalization check the long way, since we don't need the recomposition
> step on those cases, which is by far the slowest part.  The main use
> case of all of this, I expect, is to check for NFC normalization, so
> it's okay if the other variants are not optimized to the same extent.

I have committed this version.

-- 
Peter Eisentraut              http://www.2ndQuadrant.com/
PostgreSQL Development, 24x7 Support, Remote DBA, Training & Services