From: Thomas Munro <tmunro@postgresql.org>
To: pgsql-committers@lists.postgresql.org
Subject: pgsql: Add simple codepoint redirections to unaccent.rules.
Date: Fri, 05 Jul 2024 03:26:58 +0000
Message-ID: <E1sPZb6-000Mjd-Mp@gemulon.postgresql.org> (raw)
Add simple codepoint redirections to unaccent.rules.
Previously we searched for code points where the Unicode data file
listed an equivalent combining character sequence that added accents.
Some codepoints redirect to a single other codepoint, instead of doing
any combining. We can follow those references recursively to get the
answer.
Per bug report #18362, which reported missing Ancient Greek characters.
Specifically, precomposed characters with oxia (from the polytonic
accent system used for old Greek) just point to precomposed characters
with tonos (from the monotonic accent system for modern Greek), and we
have to follow the extra hop to find out that they are composed with
an acute accent.
Besides those, the new rule also:
* pulls in a lot of 'Mathematical Alphanumeric Symbols', which are
copies of the Latin and Greek alphabets and numbers rendered
in different typefaces, and
* corrects a single mathematical letter that previously came from the
CLDR transliteration file, but the new rule extracts from the main
Unicode database file, where clearly the latter is right and the
former is a wrong (reported to CLDR).
Reported-by: Cees van Zeeland <cees.van.zeeland@freedom.nl>
Reviewed-by: Robert Haas <robertmhaas@gmail.com>
Reviewed-by: Peter Eisentraut <peter@eisentraut.org>
Reviewed-by: Michael Paquier <michael@paquier.xyz>
Discussion: https://postgr.es/m/18362-be6d0cfe122b6354%40postgresql.org
Branch
------
master
Details
-------
https://git.postgresql.org/pg/commitdiff/18501841bcb4e693b9f1e9da2b2fb524c78940d8
Modified Files
--------------
contrib/unaccent/expected/unaccent.out | 2 +-
contrib/unaccent/generate_unaccent_rules.py | 19 +-
contrib/unaccent/unaccent.rules | 1013 ++++++++++++++++++++++++++-
3 files changed, 1025 insertions(+), 9 deletions(-)
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Reply to all the recipients using the --to and --cc options:
reply via email
To: pgsql-committers@postgresql.org
Cc: tmunro@postgresql.org, pgsql-committers@lists.postgresql.org
Subject: Re: pgsql: Add simple codepoint redirections to unaccent.rules.
In-Reply-To: <E1sPZb6-000Mjd-Mp@gemulon.postgresql.org>
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
This inbox is served by DDX for PostgreSQL; see mirroring instructions
for how to clone and mirror all data and code used for this inbox