Subject: Re: Unicode normalization SQL functions
To: Andreas Karlsson <andreas@proxel.se>,
 Daniel Verite <daniel@manitou-mail.org>
References: <623fa07e-348f-4273-afa4-7110ad43ca57@manitou-mail.org>
 <43f13518-010a-8319-8013-f319522ea719@2ndquadrant.com>
 <26150b35-240f-941c-e5a7-24f2d489b316@proxel.se>
Cc: pgsql-hackers <pgsql-hackers@postgresql.org>
From: Peter Eisentraut <peter.eisentraut@2ndquadrant.com>
Organization: 2ndQuadrant
Message-ID: <bba8933e-44ef-761d-55fb-076f2b6b5650@2ndquadrant.com>
Date: Mon, 17 Feb 2020 16:56:01 +0100
User-Agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10.14; rv:52.0)
 Gecko/20100101 Thunderbird/52.9.1
MIME-Version: 1.0
In-Reply-To: <26150b35-240f-941c-e5a7-24f2d489b316@proxel.se>
Content-Type: text/plain; charset=iso-8859-15; format=flowed
Content-Language: en-US
Content-Transfer-Encoding: 7bit
Precedence: bulk

On 2020-02-13 01:23, Andreas Karlsson wrote:
> A potential optimization would be to merge utf8_to_unicode() and
> pg_utf_mblen() into one function in unicode_normalize_func() since
> utf8_to_unicode() already knows length of the character. Probably not
> worth it though.

This would also require untangling the entire encoding API.

> It feels a bit wasteful to measure output_size in
> unicode_is_normalized() since unicode_normalize() actually already knows
> the length of the buffer, it just does not return it.

Sure, but really most string APIs work like that.  They surely know the 
string length internally, but afterwards you often have to call strlen() 
again.

> A potential optimization for the normalized case would be to abort the
> quick check on the first maybe and normalize from that point on only. If
> I can find the time I might try this out and benchmark it.

Are you sure this would always be valid?  The fact that this wasn't 
mentioned in UTR #15 makes me suspicious.

> Nitpick: "split/\s*;\s*/, $line" in generate-unicode_normprops_table.pl
> should be "split /\s*;\s*/, $line".

done

> What about using else if in the code below for clarity?
> 
> +		if (check == UNICODE_NORM_QC_NO)
> +			return UNICODE_NORM_QC_NO;
> +		if (check == UNICODE_NORM_QC_MAYBE)
> +			result = UNICODE_NORM_QC_MAYBE;

done

> Remove extra space in the line below.
> 
> +	else if (quickcheck == UNICODE_NORM_QC_NO )

I didn't find this in my local copy.

-- 
Peter Eisentraut              http://www.2ndQuadrant.com/
PostgreSQL Development, 24x7 Support, Remote DBA, Training & Services