From: Tom Lane <tgl@sss.pgh.pa.us>
To: kenji uno <h8mastre@gmail.com>
Cc: pgsql-bugs@lists.postgresql.org
Subject: Re: BUG #15476: Problem on show_trgm with 4 byte UTF-8 characters
Date: Sat, 03 Nov 2018 13:03:20 -0400
Message-ID: <3873.1541264600@sss.pgh.pa.us> (raw)
In-Reply-To: <CAKK3KjuY5_CYAW-TNcoec5AG9xvf-5PN4-DKghQ4GaHRPiFewQ@mail.gmail.com>
References: <15476-4314f480acf0f114@postgresql.org>
<2101.1541220270@sss.pgh.pa.us>
<CAKK3KjuY5_CYAW-TNcoec5AG9xvf-5PN4-DKghQ4GaHRPiFewQ@mail.gmail.com>
kenji uno <h8mastre@gmail.com> writes:
>> I failed to reproduce this on a Linux machine. It looks to me like the
>> problem is that Windows' MultiByteToWideChar doesn't think that UTF8
>> character is valid.
> I'm just wondering why my issue occurs only on Windows.
> But I knew why: char2wchar's tolen requires +1 output buffer size, due to
> null-termination.
Oooh ... the problem, effectively, is that the ts_locale.c functions are
expecting to get back UTF32 but what they'll actually get on Windows is
UTF16. So if the given character is outside the BMP range, char2wchar
needs to produce a surrogate pair, which there's not room for given that
the output buffer can only hold 1 wchar_t plus trailing null.
Then the other problem is that the Windows-Unicode code path in char2wchar
just fails for an undersized output buffer, which you would not expect
from its documentation. And it fails with a misleading error message,
too.
I'll see what I can do about this --- thanks for the report!
regards, tom lane
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Reply to all the recipients using the --to and --cc options:
reply via email
To: pgsql-bugs@postgresql.org
Cc: tgl@sss.pgh.pa.us, h8mastre@gmail.com, pgsql-bugs@lists.postgresql.org
Subject: Re: BUG #15476: Problem on show_trgm with 4 byte UTF-8 characters
In-Reply-To: <3873.1541264600@sss.pgh.pa.us>
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
This inbox is served by DDX for PostgreSQL; see mirroring instructions
for how to clone and mirror all data and code used for this inbox