MIME-Version: 1.0
Subject: Re: Built-in CTYPE provider
From: "Daniel Verite" <daniel@manitou-mail.org>
To: "Jeff Davis" <pgsql@j-davis.com>
Cc: Robert Haas <robertmhaas@gmail.com>,
 Jeremy Schneider <schneider@ardentperf.com>,
 pgsql-hackers@postgresql.org
In-Reply-To: <12e4f6a78403b33c303c20e44976f891d879be09.camel@j-davis.com>
Date: Mon, 15 Jan 2024 15:30:16 +0100
Message-Id: <ae044bfb-682b-449f-ad4a-c46e4332ee48@manitou-mail.org>
Content-type: multipart/mixed; boundary="----------=_1705329019-17423-0"
Archived-At: <https://www.postgresql.org/message-id/ae044bfb-682b-449f-ad4a-c46e4332ee48%40manitou-mail.org>
Precedence: bulk

This is a multi-part message in MIME format...

------------=_1705329019-17423-0
Content-Type: text/plain; charset="iso-8859-15"
Content-Disposition: inline
Content-Transfer-Encoding: quoted-printable

	Jeff Davis wrote:

> New version attached.

[v16]

Concerning the target category_test, it produces failures with
versions of ICU with Unicode < 15. The first one I see with Ubuntu
22.04 (ICU 70.1) is:

category_test: Postgres Unicode version:	15.1
category_test: ICU Unicode version:		14.0
category_test: FAILURE for codepoint 0x000c04
category_test: Postgres property=20=20=20=20=20=20=20
alphabetic/lowercase/uppercase/white_space/hex_digit/join_control:
1/0/0/0/0/0
category_test: ICU	property=20=20=20=20=20=20=20
alphabetic/lowercase/uppercase/white_space/hex_digit/join_control:
0/0/0/0/0/0

U+0C04 is a codepoint added in Unicode 11.
https://en.wikipedia.org/wiki/Telugu_(Unicode_block)

In Unicode.txt:
0C04;TELUGU SIGN COMBINING ANUSVARA ABOVE;Mn;0;NSM;;;;;N;;;;;

In Unicode 15, it has been assigned Other_Alphabetic in PropList.txt
$ grep 0C04 PropList.txt=20
0C04	      ; Other_Alphabetic # Mn	    TELUGU SIGN COMBINING ANUSVARA
ABOVE

But in Unicode 14 it was not there.
As a result its binary property UCHAR_ALPHABETIC has changed from
false to true in ICU 72 vs previous versions.

As I understand, the stability policy says that such things happen.
=46rom https://www.unicode.org/policies/stability_policy.html

   Once a character is encoded, its properties may still be changed,
   but not in such a way as to change the fundamental identity of the
   character.

   The Consortium will endeavor to keep the values of the other
   properties as stable as possible, but some circumstances may arise
   that require changing them. Particularly in the situation where
   the Unicode Standard first encodes less well-documented characters
   and scripts, the exact character properties and behavior initially
   may not be well known.

   As more experience is gathered in implementing the characters,
   adjustments in the properties may become necessary. Examples of
   such properties include, but are not limited to, the following:

    - General_Category
    - Case mappings
    - Bidirectional properties
    [...]

I've commented the exit(1) in category_test to collect all errors, and
built it with versions of ICU from 74 down to 60 (that is Unicode 10.0).
Results are attached. As expected, the older the ICU version, the more
differences are found against Unicode 15.1.

I find these results interesting because they tell us what contents
can break regexp-based check constraints on upgrades.

But about category_test as a pass-or-fail kind of test, it can only be
used when the Unicode version in ICU is the same as in Postgres.


Best regards,
--=20
Daniel V=E9rit=E9
https://postgresql.verite.pro/
Twitter: @DanielVerite

------------=_1705329019-17423-0
Content-Type: application/octet-stream;
 name="results-category-tests-multiple-icu-versions.tar.bz2"
Content-Disposition: attachment;
 filename="results-category-tests-multiple-icu-versions.tar.bz2"
Content-Transfer-Encoding: base64

QlpoOTFBWSZTWe6OcLoBRPN/pvwxEYBAA//wKyfaAP//3+AIAAACAgAACGAP
v3QeBAPIh8aQIlAAAAkVIkUUKipJSqoqOaYmAjTAjCMAAAATCMAqqVDaj1Ht
P3qqoABoAAAAAABpJ6aUBoNA9RoNA0AAABpoYKqqe/1VSYRgTQwAAEyNMJpg
TEYClSUmGU1P0wlPap6ntCZQP1IPUDTRozRlAKUkFMmTSajZITUMgwZQDaQN
AZP5L8qsdGP4VfVVrerctEZxSD3+zn4NqmhxERDMERIAA1jFGpTFrcBiyDDA
sAxKIyroh8VZA1V99RX1lTBiYmBiYrKsrD+BQn/oCYoT8CkwGEjAR7VcqeEw
sLDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDAwMH9SSSSSSSSSSSSSSSSSSSSS
SSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSMTAxMEkkkkYHgYK0f
7mjDTTGMYw0YaMYsNGGjDDDDFjRotGmLRYWzDZsaaaNGGGGGGGGGGGGGGGGG
GB9aaWmNFjAVMAmquCpumDByY2UJ/qrPyP63+h0Nz3Pc+D6HwaTctxo6H7j8
j953PuP1nJycnJyeDc3Oo7HY2OHc0dDoeC8HJo0vIxaY06vDq5TGMYxjHxSS
SSSSSSSSSSSSYxjiscQatWra02Hg8HJycng8Gk3LcNHQ7nqdj0PU9Tk5OTk5
PBubnY7Gjd3NHkeR4LwXJo0eRppp4eHV5vDw8PDw8MSSSSSSSSSSSSSSTDTG
MatWra0xjY2NjGMY5LEkkkkkkkkkkkkkkkm9vb243HAbW1tbmjgOeaY6qm6r
oqyq6iJdiAsCUuNYqwMUQXSrFWgqbCLpKmhLnJxE96pa1yQhazal6Fbqt1WF
VhVYoT5IC7vtd31nVj37uhkuZws6vyQFsVqVqVzVcNXSrJWCnjV0Vf3Vqcxo
aWttYDLh1atmzh4ePNmzZttwUpHAlhhYgYUsMWJYsFhWBhSYVVMMKYwDAJgo
MJWCVMUmJisMWGLFMYYsRhWCwpMWAsMJgkhgBYsJWAQ7FCehQn9BQmihPoKE
4AEyATrKE6QE7ShOkoT4FCYKE4gCcRQnXVncyrWyVeAE8EwCfI+ZfM+92foc
HBcHDhw0008nwe59O/YoUeFUWIqqxYUUWhoKv2q7pTyRBwKsAsCqsFe2htVg
2FMFcXR7D9JU7puy6ZNen0J5b8cZyRaYTYNsE+w2S0A6hOowmlKsCxS8IC/N
29VCe0MUJnmdD3MHu93u4Ny4OHDhppp8nd768yV6ATFgEwlIwSvZ8mK/YZ7C
ugWFVN2EtnDRubvDbdw3NOW433dbar3q33R9XV+xyC8mCbLBhlO+Ox7Fo9j2
ez2OTkuTk5cuTTTS3KnR169fw16Li9MeU2PJw20y9jMfNkctmzf5HmepsYPD
w8HBuXBw4cGmml6DzOO/6x3mDdg3waYN06nq2HJ2cjY+hhh4e3PG1XZt226G
ybHh4eDg4Tg4cODTTThOvo9Ts6evYzbh247sq46nNtkrjZnlZtm+252O7Hds
aNmNmmmnJF2dEdOqrqaxab1fJseZ6LVXHfW7gwdeM3ddx3d3dsNmzZpppyTh
3/wdtnu27u7nyG/l42G2verbxvxnOvMdG/TPI+pYYtbtg4FYPIwYaB0eT2PV
6vVsbNmzTTTo7nVjMdvNq1tmvxa3TNu+yrGu+OzK+0jR3d3dsbNmzTTTuV0d
9C8gwW4XQ6MbDTB5tmhmEbmGHB83keR5vNy3Tc3bt262bNnD63GwudvU3dQ6
l+Clb6sKqxVg7yi7RSwBMATBKwQqPnfugL1KkXuQFs+1AWVYqRf5oC3QF/wB
bFQxUyoVYvwQFpVo/JUi/egLYp+oqfQqYPwVMJpATeAnoATVAkelXzPof1H5
nL7nBwcHDhw000/aUJ2HSL6QnAAnxSonrKE3lBMyrrKE1lCddX9KncVJNyhP
dX4q2ERP7l8z5nzPkf2Ho93JycnJycuWMcivrLkwBPYoTsbnU6HR0e5cHBwc
OHBZMnMl4gJyk6C2FSTfVuFV7BJD/asJDZVrKCc9XIPorZU8ik+xRiUYCPco
T0KbVUL9ACfYiJtV8D4Pg93u+HBwcHDhw000/E5NivhB/NV7qshe2kSwu4oT
QAmkoT5K7uru000000+FbFPgoJ+4Anmq/nMYYVoqfQBNC/Or4VothET4VdKq
FoC4KE8yhO9UsAJkAmCp5lNAJ+eELvVyEkaq+wBP0gXQBP7T7z6mHd9bZubm
mG7ds0009VfexjHcxpjGNGNMYxI+8YsFiTyYxh8IC+dXSrnVOdRKsoqE2lCa
4QvSr6xRVpUtgE9wE0AnGU76sKtIoqzq9KuoBM6phAXQUJiVyLhKe1X2K3ii
raAmgBORU5ShONWlVuq7H2lTkoTQCfcUJ4KE7yhM5QTBQTjpRMyt57K6L4V+
omimgE5BJ8laVP2lOqSTgmyoU7FCaATKV6yhOYi+K+RtVx0iXmAm6qhbDnAT
Oi+ROIBNgE/pQF8oqE7QhehQT0KnuEJ4LFisTulCeoCaShOIoJtqoWhWwoJ9
ChP2gJhQm4FzV5IC0oKehQmFSTIQv4/8+DxeR5PNpKE9Kp1FBMgE7FJO8VCf
UUJqELeVwVJOCVtKdititRQngAncTyKE61YrAkj3q7QhdKuCr86uJAW2rOUJ
uKEwUE2KE9KqSPNWwCYIieALrV3gSOIQv4ql2ATlcK6RImwF1X/1XyATVXms
1W9XbVlVyFCZlbChO1c5QT+VW5T5iUmwCfigL1ATIoTlqhNEVCeo9kD/xdyR
ThQkO6OcLoA=

------------=_1705329019-17423-0--