From: Jacob Champion <pchampion@vmware.com>
To: john.naylor@enterprisedb.com <john.naylor@enterprisedb.com>
Cc: pgsql-hackers@postgresql.org <pgsql-hackers@postgresql.org>
Cc: pavel.stehule@gmail.com <pavel.stehule@gmail.com>
Cc: laurenz.albe@cybertec.at <laurenz.albe@cybertec.at>
Cc: peter.eisentraut@enterprisedb.com <peter.eisentraut@enterprisedb.com>
Cc: michael@paquier.xyz <michael@paquier.xyz>
Cc: horikyota.ntt@gmail.com <horikyota.ntt@gmail.com>
Subject: Re: badly calculated width of emoji in psql
Date: Tue, 24 Aug 2021 17:50:50 +0000
Message-ID: <b51c5abaf17a46bf82027cd833cfc91ab6d45aad.camel@vmware.com> (raw)
In-Reply-To: <CAFBsxsGOCpzV7c-f3a8ADsA1n4uZ=8puCctQp+x7W0vgkv=w+g@mail.gmail.com>
References: <9e3c847108bac6041f50e86d08d8835f5cf7cd78.camel@vmware.com>
<YPUtzT0J8cZByTkD@paquier.xyz>
<f6a06ecdf378b50ed871e363aa9ac7e706101ab6.camel@cybertec.at>
<c2a556a0158b642d6c597508d2a47059091e4d56.camel@vmware.com>
<2f7ca10c7847f69267685beca7b5aa92dd876259.camel@vmware.com>
<CAFj8pRCuaHgxa8M_FxmhGDg46qWp4yF-Xz_Xkj5NZT-F9fkQ1w@mail.gmail.com>
<CAFBsxsEW0oAD2gdWQ9OQ8dS52iz476kWK2pqV1xmGuVjaygZTA@mail.gmail.com>
<bedc833c3f60bbcd72ce613fc6a677be609d19dd.camel@vmware.com>
<CAFBsxsGZTwzDnTs=TVM38CCTPP3Y0D3=h+UiWt8M83D5THHf9A@mail.gmail.com>
<CAFBsxsFQ=scF=hgvrUWwDBzf1ZSkdmq1XRN4NJPMtniqNfpBCQ@mail.gmail.com>
<YRnRFIe5ycU3GYaX@paquier.xyz>
<CAFBsxsG39ZQNKDQpE3Yu6rytemsCk8naJ9ib=WyiSUia24_3qw@mail.gmail.com>
<CAFBsxsEX8+K4fvvUEJYTqbhpj9ENenWwYm9SrMXZ1P2cPG8P+Q@mail.gmail.com>
<39db2e88a680c5c0c113eb1977db90fa19f97155.camel@vmware.com>
<CAFBsxsGOCpzV7c-f3a8ADsA1n4uZ=8puCctQp+x7W0vgkv=w+g@mail.gmail.com>
On Fri, 2021-08-20 at 13:05 -0400, John Naylor wrote:
> On Thu, Aug 19, 2021 at 8:05 PM Jacob Champion <pchampion@vmware.com> wrote:
> > I guess it just depends on what the end result looks/performs like.
> > We'd save seven hops or so in the worst case?
>
> Something like that. Attached is what I had in mind (using real
> patches to see what the CF bot thinks):
>
> 0001 is a simple renaming
> 0002 puts the char width inside the mbinterval so we can put arbitrary values there
0002 introduces a mixed declarations/statements warning for
ucs_wcwidth(). Other than that, LGTM overall.
> --- a/src/common/wchar.c
> +++ b/src/common/wchar.c
> @@ -583,9 +583,9 @@ pg_utf_mblen(const unsigned char *s)
>
> struct mbinterval
> {
> - unsigned short first;
> - unsigned short last;
> - signed short width;
> + unsigned int first;
> + unsigned int last:21;
> + signed int width:4;
> };
Oh, right -- my patch moved mbinterval from short to int, but should I
have used uint32 instead? It would only matter in theory for the
`first` member now that the bitfields are there.
> I think the adjustments to 0003 result in a cleaner and more
> extensible design, but a case could be made otherwise. The former
> combining table script is a bit more complex than the sum of its
> former self and Jacob's proposed new script, but just slightly.
The microbenchmark says it's also more performant, so +1 to your
version.
Does there need to be any sanity check for overlapping ranges between
the combining and fullwidth sets? The Unicode data on a dev's machine
would have to be broken somehow for that to happen, but it could
potentially go undetected for a while if it did.
> Also, I checked the behavior of this comment that I proposed to remove upthread:
>
> - * - Other format characters (general category code Cf in the Unicode
> - * database) and ZERO WIDTH SPACE (U+200B) have a column width of 0.
>
> We don't handle the latter in our current setup:
>
> SELECT U&'foo\200Bbar';
> +----------+
> | ?column? |
> +----------+
> | foobar |
> +----------+
> (1 row)
>
> Not sure if we should do anything about this. It was an explicit
> exception years ago in our vendored manual table, but is not labeled
> as such in the official Unicode files.
I'm wary of changing too many things at once, but it does seem like we
should be giving that codepoint a width of 0.
On Tue, 2021-08-24 at 12:05 -0400, John Naylor wrote:
> I plan to commit my proposed v2 this week unless I hear reservations
> about my adjustments, or bikeshedding. I'm thinking of squashing 0001
> and 0002.
+1
Thanks!
--Jacob
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Reply to all the recipients from your mail client; the link fills in
the headers below: reply via email
Some clients, webmail especially, drop In-Reply-To and References from
a link, and the reply then starts a new thread: use the mbox below.
To: pgsql-hackers@postgresql.org
Cc: pchampion@vmware.com, john.naylor@enterprisedb.com, pavel.stehule@gmail.com, laurenz.albe@cybertec.at, peter.eisentraut@enterprisedb.com, michael@paquier.xyz, horikyota.ntt@gmail.com
Subject: Re: badly calculated width of emoji in psql
In-Reply-To: <b51c5abaf17a46bf82027cd833cfc91ab6d45aad.camel@vmware.com>
References: <9e3c847108bac6041f50e86d08d8835f5cf7cd78.camel@vmware.com> <YPUtzT0J8cZByTkD@paquier.xyz> <f6a06ecdf378b50ed871e363aa9ac7e706101ab6.camel@cybertec.at> <c2a556a0158b642d6c597508d2a47059091e4d56.camel@vmware.com> <2f7ca10c7847f69267685beca7b5aa92dd876259.camel@vmware.com> <CAFj8pRCuaHgxa8M_FxmhGDg46qWp4yF-Xz_Xkj5NZT-F9fkQ1w@mail.gmail.com> <CAFBsxsEW0oAD2gdWQ9OQ8dS52iz476kWK2pqV1xmGuVjaygZTA@mail.gmail.com> <bedc833c3f60bbcd72ce613fc6a677be609d19dd.camel@vmware.com> <CAFBsxsGZTwzDnTs=TVM38CCTPP3Y0D3=h+UiWt8M83D5THHf9A@mail.gmail.com> <CAFBsxsFQ=scF=hgvrUWwDBzf1ZSkdmq1XRN4NJPMtniqNfpBCQ@mail.gmail.com> <YRnRFIe5ycU3GYaX@paquier.xyz> <CAFBsxsG39ZQNKDQpE3Yu6rytemsCk8naJ9ib=WyiSUia24_3qw@mail.gmail.com> <CAFBsxsEX8+K4fvvUEJYTqbhpj9ENenWwYm9SrMXZ1P2cPG8P+Q@mail.gmail.com> <39db2e88a680c5c0c113eb1977db90fa19f97155.camel@vmware.com> <CAFBsxsGOCpzV7c-f3a8ADsA1n4uZ=8puCctQp+x7W0vgkv=w+g@mail.gmail.com> <b51c5abaf17a46bf82027cd833cfc91ab6d45aad.camel@vmware.com>
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
This inbox is served by DDX for PostgreSQL; see mirroring instructions
for how to clone and mirror all data and code used for this inbox