public inbox for [email protected]
help / color / mirror / Atom feedFrom: Tobias Bussmann <[email protected]>
To: pgsql-hackers <[email protected]>
Cc: Peter Eisentraut <[email protected]>
Cc: Thomas Munro <[email protected]>
Subject: Re: Collation version tracking for macOS
Date: Fri, 10 Jun 2022 02:48:33 +0200
Message-ID: <[email protected]> (raw)
In-Reply-To: <CA+hUKGLAAd+ftrUxJUV=Grgmf7gk0oVd_msG3oty2ufv-HVx+g@mail.gmail.com>
References: <[email protected]>
<CA+hUKGKjkL=APm5nd3vGOtMAD_07nyvrbAaGaDduVD-EKM5Drw@mail.gmail.com>
<[email protected]>
<CA+hUKGLAAd+ftrUxJUV=Grgmf7gk0oVd_msG3oty2ufv-HVx+g@mail.gmail.com>
Thanks for picking this up!
> How can I see evidence of this? I'm comparing Debian, FreeBSD and
> macOS 12.4 and when I run "LC_COLLATE=en_US.UTF-8 sort
> /usr/share/dict/words" I get upper and lower case mixed together on
> the other OSes, but on the Mac the upper case comes first, which is my
> usual smoke test for "am I looking at binary sort order?"
Perhaps I can shed some light on this matter:
Apple's libc collations have always been a bit special in that concern, even for the non-UTF8 ones. Rooted in ancient FreeBSD they "try to keep collating table backward compatible with ASCII" thus upper and lower cases characters are separated (There are exceptions like 'cs_CZ.ISO8859-2'). The latest public sources I can find are in adv_cmds-119 [1] which belongs to OSX 10.5 [2] - these correspond to the ones used in FreeBSD till v10 [3], whereby the timestamps rather point its origin around FreeBSD 5. Further, there are only very few locales actually present on macOS (36 - none of it supporting Unicode) and these have not changed for a very long time (I verified that from OS X 10.6.8 till macOS 12.4 [4], exception is a 'de_DE-A.ISO8859-1' present only in macOS 10.15).
What they do instead is symlinking [5] missing collations to similar ones even across encodings, often resulting in la_LN.US-ASCII ('la_LN' seem to stand for a Latin meta language) being used which is exactly byte order [6]. These symlinks have not changed [7] from OS X 10.6.8 till macOS 10.15.7. But in macOS 11 many of these symlinks changed their target. So did the popular 'en_US.UTF-8' from 'la_LN.US-ASCII' to 'la_LN.ISO8859-1' or 'de_DE.UTF-8' from 'la_LN.US-ASCII' to 'de_DE.ISO8859-1'. In effect, about half of the UTF-8 collations change from no collation to partial/broken collation support. macOS 12 again shows no changes - tests for macOS 13 are outstanding.
# tl:dr;
With your smoke test "sort /usr/share/dict/words" on a modern macOS you won't see a difference between "C" and "en_US.UTF-8" but with "( echo '5£'; echo '£5' ) | LC_COLLATE=en_US.UTF-8 sort" you can produce a difference against "( echo '5£'; echo '£5' ) | LC_COLLATE=C sort". Or test with "diff -q <(LC_COLLATE=C sort /usr/share/dict/words) <(LC_COLLATE=es_ES.UTF-8 sort /usr/share/dict/words)"
The upside is that we don't have to cope with the new characters added in every version of Unicode (although I have not examined LC_CTYPE yet).
best regards
Tobias
[1]: https://github.com/apple-oss-distributions/adv_cmds/tree/adv_cmds-119/usr-share-locale.tproj/colldef
[2]: https://opensource.apple.com/releases/
[3]: https://github.com/freebsd/freebsd-src/tree/stable/10/share/colldef
[4]: find /usr/share/locale/*/LC_COLLATE -type f -exec md5 {} \;
[5]: https://github.com/apple-oss-distributions/adv_cmds/blob/adv_cmds-119/usr-share-locale.tproj/colldef...
[6]: https://github.com/apple-oss-distributions/adv_cmds/blob/adv_cmds-119/usr-share-locale.tproj/colldef...
[7]: find /usr/share/locale/*/LC_COLLATE -type l -exec stat -f "%N%SY" {} \;
view thread (139+ messages) latest in thread
reply
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Reply to all the recipients using the --to and --cc options:
reply via email
To: [email protected]
Cc: [email protected], [email protected], [email protected]
Subject: Re: Collation version tracking for macOS
In-Reply-To: <[email protected]>
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
This inbox is served by agora; see mirroring instructions
for how to clone and mirror all data and code used for this inbox