From: Peter Eisentraut <peter.eisentraut@2ndquadrant.com>
To: Andreas Karlsson <andreas@proxel.se>
To: Daniel Verite <daniel@manitou-mail.org>
Cc: pgsql-hackers <pgsql-hackers@postgresql.org>
Subject: Re: Unicode normalization SQL functions
Date: Mon, 17 Feb 2020 16:56:01 +0100
Message-ID: <bba8933e-44ef-761d-55fb-076f2b6b5650@2ndquadrant.com> (raw)
In-Reply-To: <26150b35-240f-941c-e5a7-24f2d489b316@proxel.se>
References: <623fa07e-348f-4273-afa4-7110ad43ca57@manitou-mail.org>
<43f13518-010a-8319-8013-f319522ea719@2ndquadrant.com>
<26150b35-240f-941c-e5a7-24f2d489b316@proxel.se>
On 2020-02-13 01:23, Andreas Karlsson wrote:
> A potential optimization would be to merge utf8_to_unicode() and
> pg_utf_mblen() into one function in unicode_normalize_func() since
> utf8_to_unicode() already knows length of the character. Probably not
> worth it though.
This would also require untangling the entire encoding API.
> It feels a bit wasteful to measure output_size in
> unicode_is_normalized() since unicode_normalize() actually already knows
> the length of the buffer, it just does not return it.
Sure, but really most string APIs work like that. They surely know the
string length internally, but afterwards you often have to call strlen()
again.
> A potential optimization for the normalized case would be to abort the
> quick check on the first maybe and normalize from that point on only. If
> I can find the time I might try this out and benchmark it.
Are you sure this would always be valid? The fact that this wasn't
mentioned in UTR #15 makes me suspicious.
> Nitpick: "split/\s*;\s*/, $line" in generate-unicode_normprops_table.pl
> should be "split /\s*;\s*/, $line".
done
> What about using else if in the code below for clarity?
>
> + if (check == UNICODE_NORM_QC_NO)
> + return UNICODE_NORM_QC_NO;
> + if (check == UNICODE_NORM_QC_MAYBE)
> + result = UNICODE_NORM_QC_MAYBE;
done
> Remove extra space in the line below.
>
> + else if (quickcheck == UNICODE_NORM_QC_NO )
I didn't find this in my local copy.
--
Peter Eisentraut http://www.2ndQuadrant.com/
PostgreSQL Development, 24x7 Support, Remote DBA, Training & Services
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Reply to all the recipients using the --to and --cc options:
reply via email
To: pgsql-hackers@postgresql.org
Cc: peter.eisentraut@2ndquadrant.com, andreas@proxel.se, daniel@manitou-mail.org
Subject: Re: Unicode normalization SQL functions
In-Reply-To: <bba8933e-44ef-761d-55fb-076f2b6b5650@2ndquadrant.com>
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
This inbox is served by DDX for PostgreSQL; see mirroring instructions
for how to clone and mirror all data and code used for this inbox