pg.ddx.io  pgsql-bugs@postgresql.org mailing list archive  
help / color / mirror / Atom feed
From: Jeff Davis <pgsql@j-davis.com>
To: Richard Guo <guofenglinux@gmail.com>
To: PostgreSQL mailing lists <pgsql-bugs@lists.postgresql.org>
Subject: Re: Assert failure with ICU support
Date: Wed, 19 Apr 2023 13:31:17 -0700
Message-ID: <ece303a3068026c6d6f10cab2a7f3f0836f4adb7.camel@j-davis.com> (raw)
In-Reply-To: <CAMbWs49Q6UoKGeT8pBkMtJGJd+16CBFZaaWUk9Du+2ERE5g_YA@mail.gmail.com>
References: <CAMbWs49Q6UoKGeT8pBkMtJGJd+16CBFZaaWUk9Du+2ERE5g_YA@mail.gmail.com>

On Wed, 2023-04-19 at 18:30 +0800, Richard Guo wrote:
> I happened to run into an assert failure by chance with ICU support.
> Here is the query:
> 
>     SELECT '1' SIMILAR TO '\൧';
> 
> The failure happens in lexescape(),
> 
>         default:
>             assert(iscalpha(c));
>             FAILW(REG_EESCAPE); /* unknown alphabetic escape */
>             break;
> 
> Without ICU support, the same query just gives an error.
> 
> # SELECT '1' SIMILAR TO '\൧';
> ERROR:  invalid regular expression: invalid escape \ sequence
> 
> FWIW, I googled a bit and '൧' seems to be number 1 in Malayalam.

Thank you for the report and analysis! The problem exists all the way
back if you do:

  SELECT '1' COLLATE "en-US-x-icu" SIMILAR TO '\൧';

The root cause (which you allude to) is that the code makes the
assumption that digits only include 0-9, but u_isdigit('൧') == true,
which violates that assumption.

For Linux[1] specifically, it seems that the assumption should hold for
iswdigit(). But looking here[2], it seems that the behavior of
iswdigit() depends on the locale and I don't think it's correct to make
that assumption.

I did some experimentation on ICU and I found (pseudocode -- the real
code needs to create a UChar32 from an encoded string first):

  char name: MALAYALAM DIGIT ONE
  u_isalnum('൧'): true
  u_isalpha('൧'): false
  u_isdigit('൧'): true
  u_charType('൧') == U_DECIMAL_DIGIT_NUMBER: true
  u_hasBinaryProperty('൧', UCHAR_POSIX_XDIGIT): true
  u_hasBinaryProperty('൧', UCHAR_POSIX_ALNUM): true

The docs[3] for ICU say:

  "There are also functions that provide easy migration from C/POSIX
functions like isblank(). Their use is generally discouraged because
the C/POSIX standards do not define their semantics beyond the ASCII
range, which means that different implementations exhibit very
different behavior. Instead, Unicode properties should be used
directly."

We should probably just check that it's plain ASCII.

Unfortunately I would not be surprised if there are more bugs similar
to this one.

Regards,
	Jeff Davis

[1] https://man7.org/linux/man-pages/man3/iswdigit.3.html
[2]
https://pubs.opengroup.org/onlinepubs/9699919799/functions/iswdigit.html
[3] 
https://unicode-org.github.io/icu-docs/apidoc/dev/icu4c/uchar_8h.html#details





view thread (8+ messages)  latest in thread

Message-ID: <ece303a3068026c6d6f10cab2a7f3f0836f4adb7.camel@j-davis.com>
Permalink:  ../ece303a3068026c6d6f10cab2a7f3f0836f4adb7.camel@j-davis.com/
Also on:    postgresql.org/message-id/ece303a3068026c6d6f10cab2a7f3f0836f4adb7.camel@j-davis.com

 · 

reply

Reply instructions:

You may reply publicly to this message via plain-text email
using any one of the following methods:

* Reply to all the recipients from your mail client; the link fills in
  the headers below: reply via email
  Some clients, webmail especially, drop In-Reply-To and References from
  a link, and the reply then starts a new thread: use the mbox below.

  To: pgsql-bugs@postgresql.org
  Cc: pgsql@j-davis.com, guofenglinux@gmail.com
  Subject: Re: Assert failure with ICU support
  In-Reply-To: <ece303a3068026c6d6f10cab2a7f3f0836f4adb7.camel@j-davis.com>
  References: <CAMbWs49Q6UoKGeT8pBkMtJGJd+16CBFZaaWUk9Du+2ERE5g_YA@mail.gmail.com> <ece303a3068026c6d6f10cab2a7f3f0836f4adb7.camel@j-davis.com>

* Save the following mbox file, import it into your mail client,
  and reply-to-all from there: mbox

This inbox is served by DDX for PostgreSQL; see mirroring instructions
for how to clone and mirror all data and code used for this inbox