Message-ID: <de1e72d9-0a89-6d96-1995-db44f5476956@gmail.com>
Date: Tue, 16 May 2023 19:00:00 +0300
MIME-Version: 1.0
User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:102.0) Gecko/20100101
 Thunderbird/102.4.2
Subject: Re: Order changes in PG16 since ICU introduction
Content-Language: en-US
To: Jeff Davis <pgsql@j-davis.com>, Tom Lane <tgl@sss.pgh.pa.us>
Cc: Daniel Verite <daniel@manitou-mail.org>,
 Andrew Gierth <andrew@tao11.riddles.org.uk>,
 Peter Eisentraut <peter.eisentraut@enterprisedb.com>,
 Sandro Santilli <strk@kbt.io>, Regina Obe <lr@pcorp.us>,
 pgsql-hackers@lists.postgresql.org
References: <543e4ce6-8205-4aea-b54b-12b9ef747bf6@manitou-mail.org>
 <654a49f7ff7461bcf47be4181430678d45f93858.camel@j-davis.com>
 <37520ec1ae9591f83132f82dbd625f3fc2d69c16.camel@j-davis.com>
 <483826.1683582475@sss.pgh.pa.us>
 <aa6cac622417eda631a8a53287905fda1a963723.camel@j-davis.com>
 <2098874d-c111-41e4-9063-30bcf135226b@gmail.com>
 <91c0cdd2a402d59f8d6eaac40862d6f68e0b937c.camel@j-davis.com>
From: Alexander Lakhin <exclusion@gmail.com>
In-Reply-To: <91c0cdd2a402d59f8d6eaac40862d6f68e0b937c.camel@j-davis.com>
Content-Type: text/plain; charset=UTF-8; format=flowed
Content-Transfer-Encoding: 7bit
Archived-At: <https://www.postgresql.org/message-id/de1e72d9-0a89-6d96-1995-db44f5476956%40gmail.com>
Precedence: bulk

Hi Jeff,

16.05.2023 00:03, Jeff Davis wrote:
> On Sat, 2023-05-13 at 13:00 +0300, Alexander Lakhin wrote:
>> On the current master (after 455f948b0, and before f7faa9976, of
>> course)
>> I get an ASAN-detected failure with the following query:
>> CREATE COLLATION col (provider = icu, locale = '123456789012');
>>
> Thank you for the report!
>
> ICU source specifically says:
>
>    /**
>     * Useful constant for the maximum size of the language
>       part of a locale ID.
>     * (including the terminating NULL).
>     * @stable ICU 2.0
>     */
>    #define ULOC_LANG_CAPACITY 12
>
> So the fact that it returning success without nul-terminating the
> result is an ICU bug in my opinion, and I reported it here:
>
> https://unicode-org.atlassian.net/browse/ICU-22394
>
> Unfortunately that means we need to be a bit more paranoid and always
> check for that warning, even if we have a preallocated buffer of the
> "correct" size. It also means that both U_STRING_NOT_TERMINATED_WARNING
> and U_BUFFER_OVERFLOW_ERROR will be user-facing errors (potentially
> scary), unless we check for those errors each time and report specific
> errors for them.
>
> Another approach is to always check the length and loop using dynamic
> allocation so that we never overflow (and forget about any static
> buffers). That seems like overkill given that the problem case is
> obviously invalid input; I think as long as we catch it and throw an
> ERROR it's fine. But I can do this if others think it's worthwhile.
>
> Patch attached. It just checks for the U_STRING_NOT_TERMINATED_WARNING
> in a few places and turns it into an ERROR. It also cleans up the loop
> for uloc_getLanguageTag() to check for U_STRING_NOT_TERMINATED_WARNING
> rather than (U_SUCCESS(status) && len >= buflen).

I'm not sure about the proposed change in icu_from_uchar(). It seems that
len_result + 1 bytes should always be enough for the result string terminated
with NUL. If that's not true (we want to protect from some ICU bug here),
then the change should be backpatched?

Best regards,
Alexander