pg.ddx.io pgsql-bugs@postgresql.org mailing list archive
help / color / mirror / Atom feedBUG #15476: Problem on show_trgm with 4 byte UTF-8 characters
3+ messages / 2 participants
[nested] [flat]
Commits from this thread:
2018-11-03 981dc2baa8 bf4a9562e8 73dbaed93b 6e6092989f 0ae902e39e 33c697e9d4 f7ba6e951a Make ts_locale.c's character-type functions cope with UTF-16.
* BUG #15476: Problem on show_trgm with 4 byte UTF-8 characters
@ 2018-11-01 02:39 PG Bug reporting form <noreply@postgresql.org>
0 siblings, 1 reply; 3+ messages in thread
From: PG Bug reporting form @ 2018-11-01 02:39 UTC (permalink / raw)
To: pgsql-bugs@lists.postgresql.org; +Cc: h8mastre@gmail.com
The following bug has been logged on the website:
Bug reference: 15476
Logged by: Kenji Uno
Email address: h8mastre@gmail.com
PostgreSQL version: 9.6.2
Operating system: Windows Server 2012 Japanese
Description:
# Problem on show_trgm with 4 byte UTF-8 characters
On Encoding=UTF-8 database, try:
SELECT show_trgm('123');
→ OK
SELECT show_trgm('日本語');
→ probably OK.
SELECT show_trgm('🔍');
→ ERROR!
ERROR: invalid multibyte character for locale
HINT: The server's LC_CTYPE locale is probably incompatible with the
database encoding.
SQL state: 22021
I have reviewed some of your source code. And I have found a suspect
point.
Please check: t_isdigit, t_isspace, t_isalpha, and t_isprint.
https://github.com/postgres/postgres/blob/322548a8abe225f2cfd6a48e07b99e2711d28ef7/src/backend/tsear...
char2wchar 4th parameter should take number of input bytes. However they
pass character count.
int clen = pg_mblen(ptr);
...
char2wchar(character, 2, ptr, clen, mylocale);
I'm afraid, but could you look into about this?
^ permalink raw reply [nested|flat] 3+ messages in thread
* Re: BUG #15476: Problem on show_trgm with 4 byte UTF-8 characters
@ 2018-11-03 04:44 Tom Lane <tgl@sss.pgh.pa.us>
parent: PG Bug reporting form <noreply@postgresql.org>
0 siblings, 1 reply; 3+ messages in thread
From: Tom Lane @ 2018-11-03 04:44 UTC (permalink / raw)
To: h8mastre@gmail.com; +Cc: pgsql-bugs@lists.postgresql.org
=?utf-8?q?PG_Bug_reporting_form?= <noreply@postgresql.org> writes:
> On Encoding=UTF-8 database, try:
> SELECT show_trgm('123');
> → OK
> SELECT show_trgm('日本語');
> → probably OK.
> SELECT show_trgm('🔍');
> ERROR: invalid multibyte character for locale
> HINT: The server's LC_CTYPE locale is probably incompatible with the
> database encoding.
> SQL state: 22021
I failed to reproduce this on a Linux machine. It looks to me like the
problem is that Windows' MultiByteToWideChar doesn't think that UTF8
character is valid.
> Please check: t_isdigit, t_isspace, t_isalpha, and t_isprint.
> https://github.com/postgres/postgres/blob/322548a8abe225f2cfd6a48e07b99e2711d28ef7/src/backend/tsear...
> char2wchar 4th parameter should take number of input bytes. However they
> pass character count.
> int clen = pg_mblen(ptr);
> ...
> char2wchar(character, 2, ptr, clen, mylocale);
Huh? pg_mblen returns the number of bytes in a multibyte character,
so this looks fine to me.
regards, tom lane
^ permalink raw reply [nested|flat] 3+ messages in thread
* Re: BUG #15476: Problem on show_trgm with 4 byte UTF-8 characters
@ 2018-11-03 17:03 Tom Lane <tgl@sss.pgh.pa.us>
parent: Tom Lane <tgl@sss.pgh.pa.us>
0 siblings, 0 replies; 3+ messages in thread
From: Tom Lane @ 2018-11-03 17:03 UTC (permalink / raw)
To: kenji uno <h8mastre@gmail.com>; +Cc: pgsql-bugs@lists.postgresql.org
kenji uno <h8mastre@gmail.com> writes:
>> I failed to reproduce this on a Linux machine. It looks to me like the
>> problem is that Windows' MultiByteToWideChar doesn't think that UTF8
>> character is valid.
> I'm just wondering why my issue occurs only on Windows.
> But I knew why: char2wchar's tolen requires +1 output buffer size, due to
> null-termination.
Oooh ... the problem, effectively, is that the ts_locale.c functions are
expecting to get back UTF32 but what they'll actually get on Windows is
UTF16. So if the given character is outside the BMP range, char2wchar
needs to produce a surrogate pair, which there's not room for given that
the output buffer can only hold 1 wchar_t plus trailing null.
Then the other problem is that the Windows-Unicode code path in char2wchar
just fails for an undersized output buffer, which you would not expect
from its documentation. And it fails with a misleading error message,
too.
I'll see what I can do about this --- thanks for the report!
regards, tom lane
^ permalink raw reply [nested|flat] 3+ messages in thread
end of thread, other threads:[~2018-11-03 17:03 UTC | newest]
Thread overview: 3+ messages (download: mbox mbox.gz follow: Atom feed)
-- links below jump to the message on this page --
2018-11-01 02:39 BUG #15476: Problem on show_trgm with 4 byte UTF-8 characters PG Bug reporting form <noreply@postgresql.org>
2018-11-03 04:44 ` Tom Lane <tgl@sss.pgh.pa.us>
2018-11-03 17:03 ` Tom Lane <tgl@sss.pgh.pa.us>
This inbox is served by DDX for PostgreSQL; see mirroring instructions
for how to clone and mirror all data and code used for this inbox