Feedback-ID: i0fe9450f:Fastmail
Date: Mon, 26 Feb 2024 08:19:53 +0900
From: Michael Paquier <michael@paquier.xyz>
To: Thomas Munro <thomas.munro@gmail.com>
Cc: cees.van.zeeland@freedom.nl, pgsql-bugs@lists.postgresql.org
Subject: Re: BUG #18362: unaccent rules and Old Greek text
Message-ID: <ZdvLGeJ1BsXRkrdQ@paquier.xyz>
References: <18362-be6d0cfe122b6354@postgresql.org>
 <CA+hUKGLpoxTRryM1MxaHMpC1c5cxoKDm0iXfogQR9KGVUG=w9w@mail.gmail.com>
MIME-Version: 1.0
Content-Type: multipart/signed; micalg=pgp-sha512;
	protocol="application/pgp-signature"; boundary="Cn15ME/WG5QQW91d"
Content-Disposition: inline
In-Reply-To: 
 <CA+hUKGLpoxTRryM1MxaHMpC1c5cxoKDm0iXfogQR9KGVUG=w9w@mail.gmail.com>
Archived-At: 
 <https://www.postgresql.org/message-id/ZdvLGeJ1BsXRkrdQ%40paquier.xyz>
Precedence: bulk


--Cn15ME/WG5QQW91d
Content-Type: text/plain; charset=utf-8
Content-Disposition: inline
Content-Transfer-Encoding: quoted-printable

On Sun, Feb 25, 2024 at 04:21:36PM +1300, Thomas Munro wrote:
> On Sun, Feb 25, 2024 at 11:14=E2=80=AFAM PG Bug reporting form
> <noreply@postgresql.org> wrote:
>> So, there are reasons to keep the current unaccent.rules as it is, but...
>> there are other reasons to add a few lines to it, f.e. after line 955 and
>> insert five greek vowels with Oxia
>> Please add:
>> =E1=BD=B1       =CE=B1
>> =E1=BD=B3       =CE=B5
>> =E1=BD=B5       =CE=B7
>> =E1=BD=B7       =CE=B9
>> =E1=BD=B9       =CE=BF
>> =E1=BD=BB       =CF=85
>> =E1=BD=BD       =CF=89

Correct me if I'm wrong of course, but reading a bit on the matter at
[1], letters with Tonos or Oxia are actually equivalent since 1986,
and we only include character with Tonos in our unaccent.rules.

> We don't exactly maintain this list manually, we extract it from
> Unicode source data.  Can you see what needs to be adjusted in here to
> achieve that goal?

See commits like e3dd7c06e627 or 59f47fb98dab for some references.
Unfortunately, we've been using as policy to not backpatch any changes
to the in-core rules file, and you can plug in your own file.  Saying
that, these additions sound like a natural addition seen from here.

> Perhaps a new range or something like that?

It seems to me that it is a bit more complicated than that, because
Unicode.data decomposes the characters with Oxia as characters with
Tonos, and then characters with Tonos are decomposed with the "base"
alphabet characters + Tonos.  We do a recursive lookup at the unicode
table in get_plain_letter() and is_letter_with_marks(), so it seems to
me that we're not missing much, and I suspect that there should be no
need for a new custom range of characters..

Cees, perhaps you would like to get a shot at that?

[1]: https://en.wikipedia.org/wiki/Greek_diacritics#Unicode
--
Michael

--Cn15ME/WG5QQW91d
Content-Type: application/pgp-signature; name="signature.asc"

-----BEGIN PGP SIGNATURE-----

iQIzBAABCgAdFiEEG72nH6vTowiyblFKnvQgOdbyQH0FAmXbyxkACgkQnvQgOdby
QH3h8g/9FMPl46R6CplA8sfCmUjsiQEocpKAq1xaKKR4ae81idqIFysHdmgl9Q0+
gHvRQaFXTMNzY2Qh8Y0USdi7WRe7BVq2wmZWwYCTiOlWEjmP+Ll+bl9jK8GtyeJ7
jw2qjAePK3pJe9fMsJqxOUwIztFG+WwiFjk0UC9vWAZdHJVzAtNW64xXfmJUkEh3
Faw06UBY4UtDXZ7QG4N8KG67eDB0oVArmCSI254mgxGHRFK23xj2Hg69xRnsGpgJ
PNI6Xrs6exj7v6GLwj4nvkSIdJ4eL5S7C6kHIxTkupmMygV9KFRlI93IIiQ9HLph
gsSjlG1ERL1yVB7hjQiLb59xDMJL/maR7I6cx9j46kfeY94bDsf8a+0PBHGs7UjG
eA4I1NL0s4G6LdoT94lrVyoTLO5J0XrM6PcLE4bEEfTjI3sR6tWc4NijfB0HoaOW
rlyK00EvAinPENHlXNOgprcG2898jPlB5LMX7mUKzZ72SW5eOKWwwq4/ykLi4IAK
fwFrcqQ9VHPVGf9IhvVEucV7bs0OS4o9t5fYBDNPgis35sCN7ClNA8bp/99tBr/+
nxd59QIEIcGVvd/jrxR2YM4F6us3qoOM1Wl9PNUwAggsdSLkwplxUpMq7vfr2j/j
Y/NM0Tc4wN9JMtj5n04xoS+LjUD6JIREuEdEHuNGDC4d7Wa1Yp4=
=rMXb
-----END PGP SIGNATURE-----

--Cn15ME/WG5QQW91d--