pg.ddx.io  pgsql-hackers@postgresql.org mailing list archive  
help / color / mirror / Atom feed
From: Jacob Champion <pchampion@vmware.com>
To: john.naylor@enterprisedb.com <john.naylor@enterprisedb.com>
Cc: pgsql-hackers@postgresql.org <pgsql-hackers@postgresql.org>
Cc: pavel.stehule@gmail.com <pavel.stehule@gmail.com>
Cc: laurenz.albe@cybertec.at <laurenz.albe@cybertec.at>
Cc: peter.eisentraut@enterprisedb.com <peter.eisentraut@enterprisedb.com>
Cc: michael@paquier.xyz <michael@paquier.xyz>
Cc: horikyota.ntt@gmail.com <horikyota.ntt@gmail.com>
Subject: Re: badly calculated width of emoji in psql
Date: Tue, 24 Aug 2021 17:50:50 +0000
Message-ID: <b51c5abaf17a46bf82027cd833cfc91ab6d45aad.camel@vmware.com> (raw)
In-Reply-To: <CAFBsxsGOCpzV7c-f3a8ADsA1n4uZ=8puCctQp+x7W0vgkv=w+g@mail.gmail.com>
References: <9e3c847108bac6041f50e86d08d8835f5cf7cd78.camel@vmware.com>
	<YPUtzT0J8cZByTkD@paquier.xyz>
	<f6a06ecdf378b50ed871e363aa9ac7e706101ab6.camel@cybertec.at>
	<c2a556a0158b642d6c597508d2a47059091e4d56.camel@vmware.com>
	<2f7ca10c7847f69267685beca7b5aa92dd876259.camel@vmware.com>
	<CAFj8pRCuaHgxa8M_FxmhGDg46qWp4yF-Xz_Xkj5NZT-F9fkQ1w@mail.gmail.com>
	<CAFBsxsEW0oAD2gdWQ9OQ8dS52iz476kWK2pqV1xmGuVjaygZTA@mail.gmail.com>
	<bedc833c3f60bbcd72ce613fc6a677be609d19dd.camel@vmware.com>
	<CAFBsxsGZTwzDnTs=TVM38CCTPP3Y0D3=h+UiWt8M83D5THHf9A@mail.gmail.com>
	<CAFBsxsFQ=scF=hgvrUWwDBzf1ZSkdmq1XRN4NJPMtniqNfpBCQ@mail.gmail.com>
	<YRnRFIe5ycU3GYaX@paquier.xyz>
	<CAFBsxsG39ZQNKDQpE3Yu6rytemsCk8naJ9ib=WyiSUia24_3qw@mail.gmail.com>
	<CAFBsxsEX8+K4fvvUEJYTqbhpj9ENenWwYm9SrMXZ1P2cPG8P+Q@mail.gmail.com>
	<39db2e88a680c5c0c113eb1977db90fa19f97155.camel@vmware.com>
	<CAFBsxsGOCpzV7c-f3a8ADsA1n4uZ=8puCctQp+x7W0vgkv=w+g@mail.gmail.com>

On Fri, 2021-08-20 at 13:05 -0400, John Naylor wrote:
> On Thu, Aug 19, 2021 at 8:05 PM Jacob Champion <pchampion@vmware.com> wrote:
> > I guess it just depends on what the end result looks/performs like.
> > We'd save seven hops or so in the worst case?
> 
> Something like that. Attached is what I had in mind (using real
> patches to see what the CF bot thinks):
> 
> 0001 is a simple renaming
> 0002 puts the char width inside the mbinterval so we can put arbitrary values there

0002 introduces a mixed declarations/statements warning for
ucs_wcwidth(). Other than that, LGTM overall.

> --- a/src/common/wchar.c
> +++ b/src/common/wchar.c
> @@ -583,9 +583,9 @@ pg_utf_mblen(const unsigned char *s)
> 
>  struct mbinterval
>  {
> -   unsigned short first;
> -   unsigned short last;
> -   signed short width;
> +   unsigned int first;
> +   unsigned int last:21;
> +   signed int  width:4;
>  };

Oh, right -- my patch moved mbinterval from short to int, but should I
have used uint32 instead? It would only matter in theory for the
`first` member now that the bitfields are there.

> I think the adjustments to 0003 result in a cleaner and more
> extensible design, but a case could be made otherwise. The former
> combining table script is a bit more complex than the sum of its
> former self and Jacob's proposed new script, but just slightly.

The microbenchmark says it's also more performant, so +1 to your
version.

Does there need to be any sanity check for overlapping ranges between
the combining and fullwidth sets? The Unicode data on a dev's machine
would have to be broken somehow for that to happen, but it could
potentially go undetected for a while if it did.

> Also, I checked the behavior of this comment that I proposed to remove upthread:
> 
> - *  - Other format characters (general category code Cf in the Unicode
> - * database) and ZERO WIDTH SPACE (U+200B) have a column width of 0.
> 
> We don't handle the latter in our current setup:
> 
> SELECT U&'foo\200Bbar';
> +----------+
> | ?column? |
> +----------+
> | foobar  |
> +----------+
> (1 row)
> 
> Not sure if we should do anything about this. It was an explicit
> exception years ago in our vendored manual table, but is not labeled
> as such in the official Unicode files.

I'm wary of changing too many things at once, but it does seem like we
should be giving that codepoint a width of 0.

On Tue, 2021-08-24 at 12:05 -0400, John Naylor wrote:
> I plan to commit my proposed v2 this week unless I hear reservations
> about my adjustments, or bikeshedding. I'm thinking of squashing 0001
> and 0002.

+1

Thanks!
--Jacob


view thread (37+ messages)  latest in thread

Message-ID: <b51c5abaf17a46bf82027cd833cfc91ab6d45aad.camel@vmware.com>
Permalink:  ../b51c5abaf17a46bf82027cd833cfc91ab6d45aad.camel@vmware.com/
Also on:    postgresql.org/message-id/b51c5abaf17a46bf82027cd833cfc91ab6d45aad.camel@vmware.com

 · 

reply

Reply instructions:

You may reply publicly to this message via plain-text email
using any one of the following methods:

* Reply to all the recipients from your mail client; the link fills in
  the headers below: reply via email
  Some clients, webmail especially, drop In-Reply-To and References from
  a link, and the reply then starts a new thread: use the mbox below.

  To: pgsql-hackers@postgresql.org
  Cc: pchampion@vmware.com, john.naylor@enterprisedb.com, pavel.stehule@gmail.com, laurenz.albe@cybertec.at, peter.eisentraut@enterprisedb.com, michael@paquier.xyz, horikyota.ntt@gmail.com
  Subject: Re: badly calculated width of emoji in psql
  In-Reply-To: <b51c5abaf17a46bf82027cd833cfc91ab6d45aad.camel@vmware.com>
  References: <9e3c847108bac6041f50e86d08d8835f5cf7cd78.camel@vmware.com> <YPUtzT0J8cZByTkD@paquier.xyz> <f6a06ecdf378b50ed871e363aa9ac7e706101ab6.camel@cybertec.at> <c2a556a0158b642d6c597508d2a47059091e4d56.camel@vmware.com> <2f7ca10c7847f69267685beca7b5aa92dd876259.camel@vmware.com> <CAFj8pRCuaHgxa8M_FxmhGDg46qWp4yF-Xz_Xkj5NZT-F9fkQ1w@mail.gmail.com> <CAFBsxsEW0oAD2gdWQ9OQ8dS52iz476kWK2pqV1xmGuVjaygZTA@mail.gmail.com> <bedc833c3f60bbcd72ce613fc6a677be609d19dd.camel@vmware.com> <CAFBsxsGZTwzDnTs=TVM38CCTPP3Y0D3=h+UiWt8M83D5THHf9A@mail.gmail.com> <CAFBsxsFQ=scF=hgvrUWwDBzf1ZSkdmq1XRN4NJPMtniqNfpBCQ@mail.gmail.com> <YRnRFIe5ycU3GYaX@paquier.xyz> <CAFBsxsG39ZQNKDQpE3Yu6rytemsCk8naJ9ib=WyiSUia24_3qw@mail.gmail.com> <CAFBsxsEX8+K4fvvUEJYTqbhpj9ENenWwYm9SrMXZ1P2cPG8P+Q@mail.gmail.com> <39db2e88a680c5c0c113eb1977db90fa19f97155.camel@vmware.com> <CAFBsxsGOCpzV7c-f3a8ADsA1n4uZ=8puCctQp+x7W0vgkv=w+g@mail.gmail.com> <b51c5abaf17a46bf82027cd833cfc91ab6d45aad.camel@vmware.com>

* Save the following mbox file, import it into your mail client,
  and reply-to-all from there: mbox

This inbox is served by DDX for PostgreSQL; see mirroring instructions
for how to clone and mirror all data and code used for this inbox