pg.ddx.io pgsql-bugs@postgresql.org mailing list archive
help / color / mirror / Atom feedBUG #18057: unaccent removes intentional spaces
10+ messages / 3 participants
[nested] [flat]
Commits from this thread:
2023-09-20 59f47fb98d unaccent: Add support for quoted translated characters
* BUG #18057: unaccent removes intentional spaces
@ 2023-08-15 19:54 PG Bug reporting form <noreply@postgresql.org>
2023-08-16 00:00 ` Re: BUG #18057: unaccent removes intentional spaces Michael Paquier <michael@paquier.xyz>
0 siblings, 1 reply; 10+ messages in thread
From: PG Bug reporting form @ 2023-08-15 19:54 UTC (permalink / raw)
To: pgsql-bugs@lists.postgresql.org; +Cc: martin@schlossarek.me
The following bug has been logged on the website:
Bug reference: 18057
Logged by: Martin Schlossarek
Email address: martin@schlossarek.me
PostgreSQL version: 15.1
Operating system: Fedora 38
Description:
I discovered that the unaccent extension also removes intentional spaces
that are explicitly specified in the accent.rules. As far as I see it
correctly, all fraction characters are affected, for example:
```sql
# select unaccent('1½');
--- expected output: 1 1/2
--- actual output: 11/2
```
Affected characters:
```bash
$ curl -s
"https://git.postgresql.org/gitweb/?p=postgresql.git;a=blob_plain;f=contrib/unaccent/unaccent.rules;h...;
| grep -E " "
¼ 1/4
½ 1/2
¾ 3/4
⅐ 1/7
⅑ 1/9
⅒ 1/10
⅓ 1/3
⅔ 2/3
⅕ 1/5
⅖ 2/5
⅗ 3/5
⅘ 4/5
⅙ 1/6
⅚ 5/6
⅛ 1/8
⅜ 3/8
⅝ 5/8
⅞ 7/8
⅟ 1/
↉ 0/3
```
^ permalink raw reply [nested|flat] 10+ messages in thread
* Re: BUG #18057: unaccent removes intentional spaces
2023-08-15 19:54 BUG #18057: unaccent removes intentional spaces PG Bug reporting form <noreply@postgresql.org>
@ 2023-08-16 00:00 ` Michael Paquier <michael@paquier.xyz>
2023-08-18 06:18 ` Re: BUG #18057: unaccent removes intentional spaces Michael Paquier <michael@paquier.xyz>
0 siblings, 1 reply; 10+ messages in thread
From: Michael Paquier @ 2023-08-16 00:00 UTC (permalink / raw)
To: martin@schlossarek.me; pgsql-bugs@lists.postgresql.org
On Tue, Aug 15, 2023 at 07:54:57PM +0000, PG Bug reporting form wrote:
> I discovered that the unaccent extension also removes intentional spaces
> that are explicitly specified in the accent.rules. As far as I see it
> correctly, all fraction characters are affected, for example:
>
> ```sql
> # select unaccent('1½');
> --- expected output: 1 1/2
> --- actual output: 11/2
> ```
Agreed that this looks incorrect as-is. This goes as far as 9a206d0
when these has been introduced, and it looks like the culprit is
around initTrie() where the entries are loaded. See around t_isspace,
for example.
--
Michael
Attachments:
signature.asc (application/pgp-signature, 832B) download
^ permalink raw reply [nested|flat] 10+ messages in thread
* Re: BUG #18057: unaccent removes intentional spaces
2023-08-15 19:54 BUG #18057: unaccent removes intentional spaces PG Bug reporting form <noreply@postgresql.org>
2023-08-16 00:00 ` Re: BUG #18057: unaccent removes intentional spaces Michael Paquier <michael@paquier.xyz>
@ 2023-08-18 06:18 ` Michael Paquier <michael@paquier.xyz>
2023-08-19 16:30 ` Re: BUG #18057: unaccent removes intentional spaces Tom Lane <tgl@sss.pgh.pa.us>
0 siblings, 1 reply; 10+ messages in thread
From: Michael Paquier @ 2023-08-18 06:18 UTC (permalink / raw)
To: martin@schlossarek.me; pgsql-bugs@lists.postgresql.org
On Wed, Aug 16, 2023 at 09:00:43AM +0900, Michael Paquier wrote:
> Agreed that this looks incorrect as-is. This goes as far as 9a206d0
> when these has been introduced, and it looks like the culprit is
> around initTrie() where the entries are loaded. See around t_isspace,
> for example.
I was looking at the code, and my first impression was right. All
leading and trailing whitespaces between the two characters listed in
the rule file are discarded. The thing is that we clearly document
the parsing rules for the sake of any custom files one can feed to the
extension:
https://www.postgresql.org/docs/devel/unaccent.html
I am not sure what we can do here. Doing nothing is certainly an
option, but I am wondering if we could put in place an extra rule
where whitespaces can be part of the translated character if it uses
double quotes, for example. Thoughts?
--
Michael
Attachments:
signature.asc (application/pgp-signature, 832B) download
^ permalink raw reply [nested|flat] 10+ messages in thread
* Re: BUG #18057: unaccent removes intentional spaces
2023-08-15 19:54 BUG #18057: unaccent removes intentional spaces PG Bug reporting form <noreply@postgresql.org>
2023-08-16 00:00 ` Re: BUG #18057: unaccent removes intentional spaces Michael Paquier <michael@paquier.xyz>
2023-08-18 06:18 ` Re: BUG #18057: unaccent removes intentional spaces Michael Paquier <michael@paquier.xyz>
@ 2023-08-19 16:30 ` Tom Lane <tgl@sss.pgh.pa.us>
2023-08-19 23:57 ` Re: BUG #18057: unaccent removes intentional spaces Michael Paquier <michael@paquier.xyz>
0 siblings, 1 reply; 10+ messages in thread
From: Tom Lane @ 2023-08-19 16:30 UTC (permalink / raw)
To: Michael Paquier <michael@paquier.xyz>; +Cc: martin@schlossarek.me; pgsql-bugs@lists.postgresql.org
Michael Paquier <michael@paquier.xyz> writes:
> I was looking at the code, and my first impression was right. All
> leading and trailing whitespaces between the two characters listed in
> the rule file are discarded. The thing is that we clearly document
> the parsing rules for the sake of any custom files one can feed to the
> extension:
> https://www.postgresql.org/docs/devel/unaccent.html
> I am not sure what we can do here. Doing nothing is certainly an
> option, but I am wondering if we could put in place an extra rule
> where whitespaces can be part of the translated character if it uses
> double quotes, for example. Thoughts?
Yeah, we could extend the parsing rules that way. It would break
any rules files that currently use double quote as a plain character,
but it seems unlikely that there are any. In any case, as long as
this wasn't back-patched I think such a change would be acceptable.
regards, tom lane
^ permalink raw reply [nested|flat] 10+ messages in thread
* Re: BUG #18057: unaccent removes intentional spaces
2023-08-15 19:54 BUG #18057: unaccent removes intentional spaces PG Bug reporting form <noreply@postgresql.org>
2023-08-16 00:00 ` Re: BUG #18057: unaccent removes intentional spaces Michael Paquier <michael@paquier.xyz>
2023-08-18 06:18 ` Re: BUG #18057: unaccent removes intentional spaces Michael Paquier <michael@paquier.xyz>
2023-08-19 16:30 ` Re: BUG #18057: unaccent removes intentional spaces Tom Lane <tgl@sss.pgh.pa.us>
@ 2023-08-19 23:57 ` Michael Paquier <michael@paquier.xyz>
2023-08-20 00:08 ` Re: BUG #18057: unaccent removes intentional spaces Tom Lane <tgl@sss.pgh.pa.us>
0 siblings, 1 reply; 10+ messages in thread
From: Michael Paquier @ 2023-08-19 23:57 UTC (permalink / raw)
To: Tom Lane <tgl@sss.pgh.pa.us>; +Cc: martin@schlossarek.me; pgsql-bugs@lists.postgresql.org
On Sat, Aug 19, 2023 at 12:30:04PM -0400, Tom Lane wrote:
> Yeah, we could extend the parsing rules that way. It would break
> any rules files that currently use double quote as a plain character,
> but it seems unlikely that there are any. In any case, as long as
> this wasn't back-patched I think such a change would be acceptable.
Okay, thanks. Note that we do use double-quotes as a translated
character in a few cases, but as long as these are only handled as a
single character we could be OK. Or actually, wouldn't it be better
to always force escaping even for double quotes listed as single
characters? Based on what unaccent.rules has, that's not necessary,
but it could simplify the python code generating the file or the C
parsing. For example, the existing '”' would become listed as "\"" in
our unaccent.rules.
--
Michael
Attachments:
signature.asc (application/pgp-signature, 832B) download
^ permalink raw reply [nested|flat] 10+ messages in thread
* Re: BUG #18057: unaccent removes intentional spaces
2023-08-15 19:54 BUG #18057: unaccent removes intentional spaces PG Bug reporting form <noreply@postgresql.org>
2023-08-16 00:00 ` Re: BUG #18057: unaccent removes intentional spaces Michael Paquier <michael@paquier.xyz>
2023-08-18 06:18 ` Re: BUG #18057: unaccent removes intentional spaces Michael Paquier <michael@paquier.xyz>
2023-08-19 16:30 ` Re: BUG #18057: unaccent removes intentional spaces Tom Lane <tgl@sss.pgh.pa.us>
2023-08-19 23:57 ` Re: BUG #18057: unaccent removes intentional spaces Michael Paquier <michael@paquier.xyz>
@ 2023-08-20 00:08 ` Tom Lane <tgl@sss.pgh.pa.us>
2023-08-20 00:20 ` Re: BUG #18057: unaccent removes intentional spaces Michael Paquier <michael@paquier.xyz>
0 siblings, 1 reply; 10+ messages in thread
From: Tom Lane @ 2023-08-20 00:08 UTC (permalink / raw)
To: Michael Paquier <michael@paquier.xyz>; +Cc: martin@schlossarek.me; pgsql-bugs@lists.postgresql.org
Michael Paquier <michael@paquier.xyz> writes:
> On Sat, Aug 19, 2023 at 12:30:04PM -0400, Tom Lane wrote:
>> Yeah, we could extend the parsing rules that way. It would break
>> any rules files that currently use double quote as a plain character,
>> but it seems unlikely that there are any. In any case, as long as
>> this wasn't back-patched I think such a change would be acceptable.
> Okay, thanks. Note that we do use double-quotes as a translated
> character in a few cases, but as long as these are only handled as a
> single character we could be OK.
Oh, shame on me for not actually looking into unaccent.rules :-(
That does make it more complicated, although I think it's still the
case that an incompatible change could be OK.
> Or actually, wouldn't it be better
> to always force escaping even for double quotes listed as single
> characters?
I was envisioning following SQL identifier rules, that is you double
the double quote. Then the existing " entries would have to become
"""", which isn't much fun. But what you're suggesting would make
both " and \ into magic characters, doubling the chance of problems
with existing rules files.
We could perhaps allow standalone " to still be considered valid,
but that adds ambiguity that I think we'd eventually regret.
For example, what is the interpretation of
" "
Is that a (rather useless) translation of " to itself, or is it
a command to delete tab characters? So I think a clean break
would be better.
regards, tom lane
^ permalink raw reply [nested|flat] 10+ messages in thread
* Re: BUG #18057: unaccent removes intentional spaces
2023-08-15 19:54 BUG #18057: unaccent removes intentional spaces PG Bug reporting form <noreply@postgresql.org>
2023-08-16 00:00 ` Re: BUG #18057: unaccent removes intentional spaces Michael Paquier <michael@paquier.xyz>
2023-08-18 06:18 ` Re: BUG #18057: unaccent removes intentional spaces Michael Paquier <michael@paquier.xyz>
2023-08-19 16:30 ` Re: BUG #18057: unaccent removes intentional spaces Tom Lane <tgl@sss.pgh.pa.us>
2023-08-19 23:57 ` Re: BUG #18057: unaccent removes intentional spaces Michael Paquier <michael@paquier.xyz>
2023-08-20 00:08 ` Re: BUG #18057: unaccent removes intentional spaces Tom Lane <tgl@sss.pgh.pa.us>
@ 2023-08-20 00:20 ` Michael Paquier <michael@paquier.xyz>
2023-08-21 07:14 ` Re: BUG #18057: unaccent removes intentional spaces Michael Paquier <michael@paquier.xyz>
0 siblings, 1 reply; 10+ messages in thread
From: Michael Paquier @ 2023-08-20 00:20 UTC (permalink / raw)
To: Tom Lane <tgl@sss.pgh.pa.us>; +Cc: martin@schlossarek.me; pgsql-bugs@lists.postgresql.org
On Sat, Aug 19, 2023 at 08:08:26PM -0400, Tom Lane wrote:
> I was envisioning following SQL identifier rules, that is you double
> the double quote. Then the existing " entries would have to become
> """", which isn't much fun. But what you're suggesting would make
> both " and \ into magic characters, doubling the chance of problems
> with existing rules files.
Apologies for the confusion. I was thinking to also escape \ in
quoted strings. Your suggestion to use a second double-quote for the
escaping is fine by me. """" feels a bit ugly-ish in the rules file,
for sure, but that does not look like a huge issue to me as long as
the python script generates consistent contents ;)
--
Michael
Attachments:
signature.asc (application/pgp-signature, 832B) download
^ permalink raw reply [nested|flat] 10+ messages in thread
* Re: BUG #18057: unaccent removes intentional spaces
2023-08-15 19:54 BUG #18057: unaccent removes intentional spaces PG Bug reporting form <noreply@postgresql.org>
2023-08-16 00:00 ` Re: BUG #18057: unaccent removes intentional spaces Michael Paquier <michael@paquier.xyz>
2023-08-18 06:18 ` Re: BUG #18057: unaccent removes intentional spaces Michael Paquier <michael@paquier.xyz>
2023-08-19 16:30 ` Re: BUG #18057: unaccent removes intentional spaces Tom Lane <tgl@sss.pgh.pa.us>
2023-08-19 23:57 ` Re: BUG #18057: unaccent removes intentional spaces Michael Paquier <michael@paquier.xyz>
2023-08-20 00:08 ` Re: BUG #18057: unaccent removes intentional spaces Tom Lane <tgl@sss.pgh.pa.us>
2023-08-20 00:20 ` Re: BUG #18057: unaccent removes intentional spaces Michael Paquier <michael@paquier.xyz>
@ 2023-08-21 07:14 ` Michael Paquier <michael@paquier.xyz>
2023-09-19 07:26 ` Re: BUG #18057: unaccent removes intentional spaces Michael Paquier <michael@paquier.xyz>
0 siblings, 1 reply; 10+ messages in thread
From: Michael Paquier @ 2023-08-21 07:14 UTC (permalink / raw)
To: Tom Lane <tgl@sss.pgh.pa.us>; +Cc: martin@schlossarek.me; pgsql-bugs@lists.postgresql.org
On Sun, Aug 20, 2023 at 09:20:48AM +0900, Michael Paquier wrote:
> Apologies for the confusion. I was thinking to also escape \ in
> quoted strings. Your suggestion to use a second double-quote for the
> escaping is fine by me. """" feels a bit ugly-ish in the rules file,
> for sure, but that does not look like a huge issue to me as long as
> the python script generates consistent contents ;)
Please find attached a patch to achieve that. This includes tweaks
for the python script to update unaccent.rules, docs about the rules
for the quotes and tests.
The patch also includes a custom rule file that I have used to stress
more the parsing logic, but I intend to remove it in the final version
of the patch (and it fails with meson). It would be possible to have
a TAP test that sneaks a custom rule file in the installation tree,
but that's not worth the extra cycles IMO.
What do you think?
--
Michael
Attachments:
0001-unaccent-Add-support-for-quoted-translated-character.patch (text/x-diff, 11.5K) download
From 604737aff7fb805ec91e283f0c8b9a257f9039bb Mon Sep 17 00:00:00 2001
From: Michael Paquier <michael@paquier.xyz>
Date: Mon, 21 Aug 2023 16:06:42 +0900
Subject: [PATCH] unaccent: Add support for quoted translated characters
---
doc/src/sgml/unaccent.sgml | 16 +++++
contrib/unaccent/Makefile | 2 +-
contrib/unaccent/custom_unaccent.rules | 6 ++
contrib/unaccent/expected/unaccent.out | 74 ++++++++++++++++++++
contrib/unaccent/generate_unaccent_rules.py | 4 ++
contrib/unaccent/sql/unaccent.sql | 15 +++++
contrib/unaccent/unaccent--1.1.sql | 5 ++
contrib/unaccent/unaccent.c | 75 ++++++++++++++++++---
contrib/unaccent/unaccent.rules | 56 +++++++--------
9 files changed, 216 insertions(+), 37 deletions(-)
create mode 100644 contrib/unaccent/custom_unaccent.rules
diff --git a/doc/src/sgml/unaccent.sgml b/doc/src/sgml/unaccent.sgml
index f3ddc64bbc..94100ed260 100644
--- a/doc/src/sgml/unaccent.sgml
+++ b/doc/src/sgml/unaccent.sgml
@@ -84,6 +84,22 @@
</para>
</listitem>
+ <listitem>
+ <para>
+ Some characters, like numeric symbols, may require whitespaces in their
+ translation rule. It is possible to use double quotes around the translated
+ characters in this case. A double quote needs to be escaped with a second
+ double quote when including one in the translated character. For example:
+<programlisting>
+¼ " 1/4"
+½ " 1/2"
+¾ " 3/4"
+“ """"
+” """"
+</programlisting>
+ </para>
+ </listitem>
+
<listitem>
<para>
As with other <productname>PostgreSQL</productname> text search configuration files,
diff --git a/contrib/unaccent/Makefile b/contrib/unaccent/Makefile
index 652a3e774c..27c7b2ca6e 100644
--- a/contrib/unaccent/Makefile
+++ b/contrib/unaccent/Makefile
@@ -7,7 +7,7 @@ OBJS = \
EXTENSION = unaccent
DATA = unaccent--1.1.sql unaccent--1.0--1.1.sql
-DATA_TSEARCH = unaccent.rules
+DATA_TSEARCH = unaccent.rules custom_unaccent.rules
PGFILEDESC = "unaccent - text search dictionary that removes accents"
REGRESS = unaccent
diff --git a/contrib/unaccent/custom_unaccent.rules b/contrib/unaccent/custom_unaccent.rules
new file mode 100644
index 0000000000..d8791dd902
--- /dev/null
+++ b/contrib/unaccent/custom_unaccent.rules
@@ -0,0 +1,6 @@
+¼ " ""1/4"" "
+½ """1/2"" "
+¾ " ""3/4"""
+ʺ " a """" "
+“ " t """
+” """ b "
diff --git a/contrib/unaccent/expected/unaccent.out b/contrib/unaccent/expected/unaccent.out
index f080707c4a..64402010ce 100644
--- a/contrib/unaccent/expected/unaccent.out
+++ b/contrib/unaccent/expected/unaccent.out
@@ -51,6 +51,18 @@ SELECT unaccent('℗'); -- sound recording copyright
(P)
(1 row)
+SELECT unaccent('1½'); -- math expression with whitespace
+ unaccent
+----------
+ 1 1/2
+(1 row)
+
+SELECT unaccent('〝'); -- quote
+ unaccent
+----------
+ "
+(1 row)
+
SELECT unaccent('unaccent', 'foobar');
unaccent
----------
@@ -93,6 +105,56 @@ SELECT unaccent('unaccent', '℗');
(P)
(1 row)
+SELECT unaccent('unaccent', '1½');
+ unaccent
+----------
+ 1 1/2
+(1 row)
+
+SELECT unaccent('unaccent', '〝');
+ unaccent
+----------
+ "
+(1 row)
+
+-- XXX: Remove later.
+-- Just for the sake of checking the parsing logic.
+SELECT unaccent('custom_unaccent', '1¼');
+ unaccent
+------------
+ 1 "1/4"
+(1 row)
+
+SELECT unaccent('custom_unaccent', '1½');
+ unaccent
+------------
+ 1"1/2"
+(1 row)
+
+SELECT unaccent('custom_unaccent', '1¾');
+ unaccent
+----------
+ 1 "3/4"
+(1 row)
+
+SELECT unaccent('custom_unaccent', 'ʺ');
+ unaccent
+----------
+ a ""
+(1 row)
+
+SELECT unaccent('custom_unaccent', '“');
+ unaccent
+----------
+ t "
+(1 row)
+
+SELECT unaccent('custom_unaccent', '”');
+ unaccent
+----------
+ " b
+(1 row)
+
SELECT ts_lexize('unaccent', 'foobar');
ts_lexize
-----------
@@ -135,6 +197,18 @@ SELECT ts_lexize('unaccent', '℗');
{(P)}
(1 row)
+SELECT ts_lexize('unaccent', '1½');
+ ts_lexize
+-----------
+ {"1 1/2"}
+(1 row)
+
+SELECT ts_lexize('unaccent', '〝');
+ ts_lexize
+-----------
+ {"\""}
+(1 row)
+
-- Controversial case. Black-Letter Capital H (U+210C) is translated by
-- Latin-ASCII.xml as 'x', but it should be 'H'.
SELECT unaccent('ℌ');
diff --git a/contrib/unaccent/generate_unaccent_rules.py b/contrib/unaccent/generate_unaccent_rules.py
index b4b4c38beb..cffb7db7ce 100644
--- a/contrib/unaccent/generate_unaccent_rules.py
+++ b/contrib/unaccent/generate_unaccent_rules.py
@@ -58,6 +58,10 @@ COMBINING_MARK_RANGES = ((0x0300, 0x0362), # Mn: Accents, IPA
def print_record(codepoint, letter):
if letter:
+ # If the letter has whitespace or double quotes, escape double
+ # quotes and apply more quotes around it.
+ if (' ' in letter) or ('"' in letter):
+ letter = '"' + letter.replace('"', '""') + '"'
output = chr(codepoint) + "\t" + letter
else:
output = chr(codepoint)
diff --git a/contrib/unaccent/sql/unaccent.sql b/contrib/unaccent/sql/unaccent.sql
index 663646c1ac..9ef2fc5010 100644
--- a/contrib/unaccent/sql/unaccent.sql
+++ b/contrib/unaccent/sql/unaccent.sql
@@ -20,6 +20,8 @@ SELECT unaccent('˃˖˗˜');
SELECT unaccent('À'); -- Remove combining diacritical 0x0300
SELECT unaccent('℃℉'); -- degree signs
SELECT unaccent('℗'); -- sound recording copyright
+SELECT unaccent('1½'); -- math expression with whitespace
+SELECT unaccent('〝'); -- quote
SELECT unaccent('unaccent', 'foobar');
SELECT unaccent('unaccent', 'ёлка');
@@ -28,6 +30,17 @@ SELECT unaccent('unaccent', '˃˖˗˜');
SELECT unaccent('unaccent', 'À');
SELECT unaccent('unaccent', '℃℉');
SELECT unaccent('unaccent', '℗');
+SELECT unaccent('unaccent', '1½');
+SELECT unaccent('unaccent', '〝');
+
+-- XXX: Remove later.
+-- Just for the sake of checking the parsing logic.
+SELECT unaccent('custom_unaccent', '1¼');
+SELECT unaccent('custom_unaccent', '1½');
+SELECT unaccent('custom_unaccent', '1¾');
+SELECT unaccent('custom_unaccent', 'ʺ');
+SELECT unaccent('custom_unaccent', '“');
+SELECT unaccent('custom_unaccent', '”');
SELECT ts_lexize('unaccent', 'foobar');
SELECT ts_lexize('unaccent', 'ёлка');
@@ -36,6 +49,8 @@ SELECT ts_lexize('unaccent', '˃˖˗˜');
SELECT ts_lexize('unaccent', 'À');
SELECT ts_lexize('unaccent', '℃℉');
SELECT ts_lexize('unaccent', '℗');
+SELECT ts_lexize('unaccent', '1½');
+SELECT ts_lexize('unaccent', '〝');
-- Controversial case. Black-Letter Capital H (U+210C) is translated by
-- Latin-ASCII.xml as 'x', but it should be 'H'.
diff --git a/contrib/unaccent/unaccent--1.1.sql b/contrib/unaccent/unaccent--1.1.sql
index ecc8651780..a821e1c37d 100644
--- a/contrib/unaccent/unaccent--1.1.sql
+++ b/contrib/unaccent/unaccent--1.1.sql
@@ -32,3 +32,8 @@ CREATE TEXT SEARCH DICTIONARY unaccent (
TEMPLATE = unaccent,
RULES = 'unaccent'
);
+
+CREATE TEXT SEARCH DICTIONARY custom_unaccent (
+ TEMPLATE = unaccent,
+ RULES = 'custom_unaccent'
+);
diff --git a/contrib/unaccent/unaccent.c b/contrib/unaccent/unaccent.c
index 64c879e547..75fb4032b2 100644
--- a/contrib/unaccent/unaccent.c
+++ b/contrib/unaccent/unaccent.c
@@ -127,14 +127,16 @@ initTrie(const char *filename)
* src and trg are sequences of one or more non-whitespace
* characters, separated by whitespace. Whitespace at start
* or end of line is ignored. If trg is omitted, an empty
- * string is used as the replacement.
+ * string is used as the replacement. trg can be optionally
+ * quoted, in which case whitespaces are included in it.
*
* We use a simple state machine, with states
* 0 initial (before src)
* 1 in src
* 2 in whitespace after src
- * 3 in trg
- * 4 in whitespace after trg
+ * 3 in trg (non-quoted)
+ * 4 in trg (quoted)
+ * 5 in whitespace after trg
* -1 syntax error detected
*----------
*/
@@ -142,9 +144,12 @@ initTrie(const char *filename)
char *ptr;
char *src = NULL;
char *trg = NULL;
+ char *trgstore = NULL;
int ptrlen;
int srclen = 0;
int trglen = 0;
+ int trgstorelen = 0;
+ bool trgquoted = false;
state = 0;
for (ptr = line; *ptr; ptr += ptrlen)
@@ -156,8 +161,10 @@ initTrie(const char *filename)
if (state == 1)
state = 2;
else if (state == 3)
- state = 4;
- continue;
+ state = 5;
+ /* whitespaces are OK in quoted area */
+ if (state != 4)
+ continue;
}
switch (state)
{
@@ -173,14 +180,41 @@ initTrie(const char *filename)
break;
case 2:
/* start of trg */
+ if (*ptr == '"')
+ {
+ trgquoted = true;
+ state = 4;
+ }
+ else
+ state = 3;
+
trg = ptr;
trglen = ptrlen;
- state = 3;
break;
case 3:
- /* continue trg */
+ /* continue non-quoted trg */
trglen += ptrlen;
break;
+ case 4:
+ /* continue quoted trg */
+ trglen += ptrlen;
+
+ /*
+ * If this is a quote, consider it as the end of
+ * trg except if the follow-up character is itself
+ * a quote.
+ */
+ if (*ptr == '"')
+ {
+ if (*(ptr + 1) == '"')
+ {
+ ptr++;
+ trglen += 1;
+ }
+ else
+ state = 5;
+ }
+ break;
default:
/* bogus line format */
state = -1;
@@ -195,10 +229,35 @@ initTrie(const char *filename)
trglen = 0;
}
+ /* If still in a quoted area, fallback to an error */
+ if (state == 4)
+ state = -1;
+
+ /* If trg was quoted, remove its quotes and unescape it */
+ if (trgquoted)
+ {
+ /* Ignore first and end quotes */
+ trgstore = palloc0(sizeof(char *) * trglen - 2);
+ trgstorelen = 0;
+ for (int i = 1; i < trglen - 1; i++)
+ {
+ trgstore[trgstorelen] = trg[i];
+ trgstorelen++;
+ /* skip second double quotes */
+ if (trg[i] == '"' && trg[i + 1] == '"')
+ i++;
+ }
+ }
+ else
+ {
+ trgstore = trg;
+ trgstorelen = trglen;
+ }
+
if (state > 0)
rootTrie = placeChar(rootTrie,
(unsigned char *) src, srclen,
- trg, trglen);
+ trgstore, trgstorelen);
else if (state < 0)
ereport(WARNING,
(errcode(ERRCODE_CONFIG_FILE_ERROR),
diff --git a/contrib/unaccent/unaccent.rules b/contrib/unaccent/unaccent.rules
index 3030166ed6..ca6caa51f5 100644
--- a/contrib/unaccent/unaccent.rules
+++ b/contrib/unaccent/unaccent.rules
@@ -5,9 +5,9 @@
® (R)
± +/-
» >>
-¼ 1/4
-½ 1/2
-¾ 3/4
+¼ " 1/4"
+½ " 1/2"
+¾ " 3/4"
¿ ?
À A
Á A
@@ -403,7 +403,7 @@
ʪ ls
ʫ lz
ʹ '
-ʺ "
+ʺ """"
ʻ '
ʼ '
ʽ '
@@ -1058,15 +1058,15 @@
’ '
‚ ,
‛ '
-“ "
-” "
+“ """"
+” """"
„ ,,
-‟ "
+‟ """"
․ .
‥ ..
… ...
′ '
-″ "
+″ """"
‹ <
› >
‼ !!
@@ -1134,22 +1134,22 @@
ⅇ e
ⅈ i
ⅉ j
-⅐ 1/7
-⅑ 1/9
-⅒ 1/10
-⅓ 1/3
-⅔ 2/3
-⅕ 1/5
-⅖ 2/5
-⅗ 3/5
-⅘ 4/5
-⅙ 1/6
-⅚ 5/6
-⅛ 1/8
-⅜ 3/8
-⅝ 5/8
-⅞ 7/8
-⅟ 1/
+⅐ " 1/7"
+⅑ " 1/9"
+⅒ " 1/10"
+⅓ " 1/3"
+⅔ " 2/3"
+⅕ " 1/5"
+⅖ " 2/5"
+⅗ " 3/5"
+⅘ " 4/5"
+⅙ " 1/6"
+⅚ " 5/6"
+⅛ " 1/8"
+⅜ " 3/8"
+⅝ " 5/8"
+⅞ " 7/8"
+⅟ " 1/"
Ⅰ I
Ⅱ II
Ⅲ III
@@ -1182,7 +1182,7 @@
ⅽ c
ⅾ d
ⅿ m
-↉ 0/3
+↉ " 0/3"
− -
∕ /
∖ \
@@ -1296,8 +1296,8 @@
〙 ]
〚 [
〛 ]
-〝 "
-〞 "
+〝 """"
+〞 """"
㍱ hPa
㍲ da
㍳ AU
@@ -1512,7 +1512,7 @@
﹪ %
﹫ @
！ !
-＂ "
+＂ """"
＃ #
＄ $
％ %
--
2.40.1
* Re: BUG #18057: unaccent removes intentional spaces
2023-08-15 19:54 BUG #18057: unaccent removes intentional spaces PG Bug reporting form <noreply@postgresql.org>
2023-08-16 00:00 ` Re: BUG #18057: unaccent removes intentional spaces Michael Paquier <michael@paquier.xyz>
2023-08-18 06:18 ` Re: BUG #18057: unaccent removes intentional spaces Michael Paquier <michael@paquier.xyz>
2023-08-19 16:30 ` Re: BUG #18057: unaccent removes intentional spaces Tom Lane <tgl@sss.pgh.pa.us>
2023-08-19 23:57 ` Re: BUG #18057: unaccent removes intentional spaces Michael Paquier <michael@paquier.xyz>
2023-08-20 00:08 ` Re: BUG #18057: unaccent removes intentional spaces Tom Lane <tgl@sss.pgh.pa.us>
2023-08-20 00:20 ` Re: BUG #18057: unaccent removes intentional spaces Michael Paquier <michael@paquier.xyz>
2023-08-21 07:14 ` Re: BUG #18057: unaccent removes intentional spaces Michael Paquier <michael@paquier.xyz>
@ 2023-09-19 07:26 ` Michael Paquier <michael@paquier.xyz>
2023-09-20 23:24 ` Re: BUG #18057: unaccent removes intentional spaces Michael Paquier <michael@paquier.xyz>
0 siblings, 1 reply; 10+ messages in thread
From: Michael Paquier @ 2023-09-19 07:26 UTC (permalink / raw)
To: Tom Lane <tgl@sss.pgh.pa.us>; +Cc: martin@schlossarek.me; pgsql-bugs@lists.postgresql.org
On Mon, Aug 21, 2023 at 04:14:20PM +0900, Michael Paquier wrote:
> The patch also includes a custom rule file that I have used to stress
> more the parsing logic, but I intend to remove it in the final version
> of the patch (and it fails with meson). It would be possible to have
> a TAP test that sneaks a custom rule file in the installation tree,
> but that's not worth the extra cycles IMO.
So, it's been a couple of weeks here.
I have given this patch a second look and something that stood out is
that we'd still report "more than two strings in unaccent rule" as
warning when loading a rule where the target string is an unfinished
quoted area, which was kind of confusing.
The patch has been updated to improve that by making the error state a
bit smarter, and I have removed the previous tests with the custom
file and its fancy rules. Any comments and/or objections?
--
Michael
Attachments:
v2-0001-unaccent-Add-support-for-quoted-translated-charac.patch (text/x-diff, 9.6K) download
From 18e3cf3eb1b3af0fb2f2f805e17eb7d0540edc91 Mon Sep 17 00:00:00 2001
From: Michael Paquier <michael@paquier.xyz>
Date: Tue, 19 Sep 2023 16:18:21 +0900
Subject: [PATCH v2] unaccent: Add support for quoted translated characters
---
doc/src/sgml/unaccent.sgml | 16 ++++
contrib/unaccent/expected/unaccent.out | 36 +++++++++
contrib/unaccent/generate_unaccent_rules.py | 4 +
contrib/unaccent/sql/unaccent.sql | 6 ++
contrib/unaccent/unaccent.c | 84 ++++++++++++++++++---
contrib/unaccent/unaccent.rules | 56 +++++++-------
6 files changed, 164 insertions(+), 38 deletions(-)
diff --git a/doc/src/sgml/unaccent.sgml b/doc/src/sgml/unaccent.sgml
index f3ddc64bbc..94100ed260 100644
--- a/doc/src/sgml/unaccent.sgml
+++ b/doc/src/sgml/unaccent.sgml
@@ -84,6 +84,22 @@
</para>
</listitem>
+ <listitem>
+ <para>
+ Some characters, like numeric symbols, may require whitespaces in their
+ translation rule. It is possible to use double quotes around the translated
+ characters in this case. A double quote needs to be escaped with a second
+ double quote when including one in the translated character. For example:
+<programlisting>
+¼ " 1/4"
+½ " 1/2"
+¾ " 3/4"
+“ """"
+” """"
+</programlisting>
+ </para>
+ </listitem>
+
<listitem>
<para>
As with other <productname>PostgreSQL</productname> text search configuration files,
diff --git a/contrib/unaccent/expected/unaccent.out b/contrib/unaccent/expected/unaccent.out
index f080707c4a..d03374c799 100644
--- a/contrib/unaccent/expected/unaccent.out
+++ b/contrib/unaccent/expected/unaccent.out
@@ -51,6 +51,18 @@ SELECT unaccent('℗'); -- sound recording copyright
(P)
(1 row)
+SELECT unaccent('1½'); -- math expression with whitespace
+ unaccent
+----------
+ 1 1/2
+(1 row)
+
+SELECT unaccent('〝'); -- quote
+ unaccent
+----------
+ "
+(1 row)
+
SELECT unaccent('unaccent', 'foobar');
unaccent
----------
@@ -93,6 +105,18 @@ SELECT unaccent('unaccent', '℗');
(P)
(1 row)
+SELECT unaccent('unaccent', '1½');
+ unaccent
+----------
+ 1 1/2
+(1 row)
+
+SELECT unaccent('unaccent', '〝');
+ unaccent
+----------
+ "
+(1 row)
+
SELECT ts_lexize('unaccent', 'foobar');
ts_lexize
-----------
@@ -135,6 +159,18 @@ SELECT ts_lexize('unaccent', '℗');
{(P)}
(1 row)
+SELECT ts_lexize('unaccent', '1½');
+ ts_lexize
+-----------
+ {"1 1/2"}
+(1 row)
+
+SELECT ts_lexize('unaccent', '〝');
+ ts_lexize
+-----------
+ {"\""}
+(1 row)
+
-- Controversial case. Black-Letter Capital H (U+210C) is translated by
-- Latin-ASCII.xml as 'x', but it should be 'H'.
SELECT unaccent('ℌ');
diff --git a/contrib/unaccent/generate_unaccent_rules.py b/contrib/unaccent/generate_unaccent_rules.py
index b4b4c38beb..cffb7db7ce 100644
--- a/contrib/unaccent/generate_unaccent_rules.py
+++ b/contrib/unaccent/generate_unaccent_rules.py
@@ -58,6 +58,10 @@ COMBINING_MARK_RANGES = ((0x0300, 0x0362), # Mn: Accents, IPA
def print_record(codepoint, letter):
if letter:
+ # If the letter has whitespace or double quotes, escape double
+ # quotes and apply more quotes around it.
+ if (' ' in letter) or ('"' in letter):
+ letter = '"' + letter.replace('"', '""') + '"'
output = chr(codepoint) + "\t" + letter
else:
output = chr(codepoint)
diff --git a/contrib/unaccent/sql/unaccent.sql b/contrib/unaccent/sql/unaccent.sql
index 663646c1ac..70c7f1c0a0 100644
--- a/contrib/unaccent/sql/unaccent.sql
+++ b/contrib/unaccent/sql/unaccent.sql
@@ -20,6 +20,8 @@ SELECT unaccent('˃˖˗˜');
SELECT unaccent('À'); -- Remove combining diacritical 0x0300
SELECT unaccent('℃℉'); -- degree signs
SELECT unaccent('℗'); -- sound recording copyright
+SELECT unaccent('1½'); -- math expression with whitespace
+SELECT unaccent('〝'); -- quote
SELECT unaccent('unaccent', 'foobar');
SELECT unaccent('unaccent', 'ёлка');
@@ -28,6 +30,8 @@ SELECT unaccent('unaccent', '˃˖˗˜');
SELECT unaccent('unaccent', 'À');
SELECT unaccent('unaccent', '℃℉');
SELECT unaccent('unaccent', '℗');
+SELECT unaccent('unaccent', '1½');
+SELECT unaccent('unaccent', '〝');
SELECT ts_lexize('unaccent', 'foobar');
SELECT ts_lexize('unaccent', 'ёлка');
@@ -36,6 +40,8 @@ SELECT ts_lexize('unaccent', '˃˖˗˜');
SELECT ts_lexize('unaccent', 'À');
SELECT ts_lexize('unaccent', '℃℉');
SELECT ts_lexize('unaccent', '℗');
+SELECT ts_lexize('unaccent', '1½');
+SELECT ts_lexize('unaccent', '〝');
-- Controversial case. Black-Letter Capital H (U+210C) is translated by
-- Latin-ASCII.xml as 'x', but it should be 'H'.
diff --git a/contrib/unaccent/unaccent.c b/contrib/unaccent/unaccent.c
index 64c879e547..dd705c9cd6 100644
--- a/contrib/unaccent/unaccent.c
+++ b/contrib/unaccent/unaccent.c
@@ -127,24 +127,30 @@ initTrie(const char *filename)
* src and trg are sequences of one or more non-whitespace
* characters, separated by whitespace. Whitespace at start
* or end of line is ignored. If trg is omitted, an empty
- * string is used as the replacement.
+ * string is used as the replacement. trg can be optionally
+ * quoted, in which case whitespaces are included in it.
*
* We use a simple state machine, with states
* 0 initial (before src)
* 1 in src
* 2 in whitespace after src
- * 3 in trg
- * 4 in whitespace after trg
- * -1 syntax error detected
+ * 3 in trg (non-quoted)
+ * 4 in trg (quoted)
+ * 5 in whitespace after trg
+ * -1 syntax error detected (two strings)
+ * -2 syntax error detected (unfinished quoted string)
*----------
*/
int state;
char *ptr;
char *src = NULL;
char *trg = NULL;
+ char *trgstore = NULL;
int ptrlen;
int srclen = 0;
int trglen = 0;
+ int trgstorelen = 0;
+ bool trgquoted = false;
state = 0;
for (ptr = line; *ptr; ptr += ptrlen)
@@ -156,8 +162,10 @@ initTrie(const char *filename)
if (state == 1)
state = 2;
else if (state == 3)
- state = 4;
- continue;
+ state = 5;
+ /* whitespaces are OK in quoted area */
+ if (state != 4)
+ continue;
}
switch (state)
{
@@ -173,14 +181,41 @@ initTrie(const char *filename)
break;
case 2:
/* start of trg */
+ if (*ptr == '"')
+ {
+ trgquoted = true;
+ state = 4;
+ }
+ else
+ state = 3;
+
trg = ptr;
trglen = ptrlen;
- state = 3;
break;
case 3:
- /* continue trg */
+ /* continue non-quoted trg */
trglen += ptrlen;
break;
+ case 4:
+ /* continue quoted trg */
+ trglen += ptrlen;
+
+ /*
+ * If this is a quote, consider it as the end of
+ * trg except if the follow-up character is itself
+ * a quote.
+ */
+ if (*ptr == '"')
+ {
+ if (*(ptr + 1) == '"')
+ {
+ ptr++;
+ trglen += 1;
+ }
+ else
+ state = 5;
+ }
+ break;
default:
/* bogus line format */
state = -1;
@@ -195,14 +230,43 @@ initTrie(const char *filename)
trglen = 0;
}
+ /* If still in a quoted area, fallback to an error */
+ if (state == 4)
+ state = -2;
+
+ /* If trg was quoted, remove its quotes and unescape it */
+ if (trgquoted && state > 0)
+ {
+ /* Ignore first and end quotes */
+ trgstore = palloc0(sizeof(char *) * trglen - 2);
+ trgstorelen = 0;
+ for (int i = 1; i < trglen - 1; i++)
+ {
+ trgstore[trgstorelen] = trg[i];
+ trgstorelen++;
+ /* skip second double quotes */
+ if (trg[i] == '"' && trg[i + 1] == '"')
+ i++;
+ }
+ }
+ else
+ {
+ trgstore = trg;
+ trgstorelen = trglen;
+ }
+
if (state > 0)
rootTrie = placeChar(rootTrie,
(unsigned char *) src, srclen,
- trg, trglen);
- else if (state < 0)
+ trgstore, trgstorelen);
+ else if (state == -1)
ereport(WARNING,
(errcode(ERRCODE_CONFIG_FILE_ERROR),
errmsg("invalid syntax: more than two strings in unaccent rule")));
+ else if (state == -2)
+ ereport(WARNING,
+ (errcode(ERRCODE_CONFIG_FILE_ERROR),
+ errmsg("invalid syntax: unfinished quoted string in unaccent rule")));
pfree(line);
}
diff --git a/contrib/unaccent/unaccent.rules b/contrib/unaccent/unaccent.rules
index 3030166ed6..ca6caa51f5 100644
--- a/contrib/unaccent/unaccent.rules
+++ b/contrib/unaccent/unaccent.rules
@@ -5,9 +5,9 @@
® (R)
± +/-
» >>
-¼ 1/4
-½ 1/2
-¾ 3/4
+¼ " 1/4"
+½ " 1/2"
+¾ " 3/4"
¿ ?
À A
Á A
@@ -403,7 +403,7 @@
ʪ ls
ʫ lz
ʹ '
-ʺ "
+ʺ """"
ʻ '
ʼ '
ʽ '
@@ -1058,15 +1058,15 @@
’ '
‚ ,
‛ '
-“ "
-” "
+“ """"
+” """"
„ ,,
-‟ "
+‟ """"
․ .
‥ ..
… ...
′ '
-″ "
+″ """"
‹ <
› >
‼ !!
@@ -1134,22 +1134,22 @@
ⅇ e
ⅈ i
ⅉ j
-⅐ 1/7
-⅑ 1/9
-⅒ 1/10
-⅓ 1/3
-⅔ 2/3
-⅕ 1/5
-⅖ 2/5
-⅗ 3/5
-⅘ 4/5
-⅙ 1/6
-⅚ 5/6
-⅛ 1/8
-⅜ 3/8
-⅝ 5/8
-⅞ 7/8
-⅟ 1/
+⅐ " 1/7"
+⅑ " 1/9"
+⅒ " 1/10"
+⅓ " 1/3"
+⅔ " 2/3"
+⅕ " 1/5"
+⅖ " 2/5"
+⅗ " 3/5"
+⅘ " 4/5"
+⅙ " 1/6"
+⅚ " 5/6"
+⅛ " 1/8"
+⅜ " 3/8"
+⅝ " 5/8"
+⅞ " 7/8"
+⅟ " 1/"
Ⅰ I
Ⅱ II
Ⅲ III
@@ -1182,7 +1182,7 @@
ⅽ c
ⅾ d
ⅿ m
-↉ 0/3
+↉ " 0/3"
− -
∕ /
∖ \
@@ -1296,8 +1296,8 @@
〙 ]
〚 [
〛 ]
-〝 "
-〞 "
+〝 """"
+〞 """"
㍱ hPa
㍲ da
㍳ AU
@@ -1512,7 +1512,7 @@
﹪ %
﹫ @
！ !
-＂ "
+＂ """"
＃ #
＄ $
％ %
--
2.40.1
* Re: BUG #18057: unaccent removes intentional spaces
2023-08-15 19:54 BUG #18057: unaccent removes intentional spaces PG Bug reporting form <noreply@postgresql.org>
2023-08-16 00:00 ` Re: BUG #18057: unaccent removes intentional spaces Michael Paquier <michael@paquier.xyz>
2023-08-18 06:18 ` Re: BUG #18057: unaccent removes intentional spaces Michael Paquier <michael@paquier.xyz>
2023-08-19 16:30 ` Re: BUG #18057: unaccent removes intentional spaces Tom Lane <tgl@sss.pgh.pa.us>
2023-08-19 23:57 ` Re: BUG #18057: unaccent removes intentional spaces Michael Paquier <michael@paquier.xyz>
2023-08-20 00:08 ` Re: BUG #18057: unaccent removes intentional spaces Tom Lane <tgl@sss.pgh.pa.us>
2023-08-20 00:20 ` Re: BUG #18057: unaccent removes intentional spaces Michael Paquier <michael@paquier.xyz>
2023-08-21 07:14 ` Re: BUG #18057: unaccent removes intentional spaces Michael Paquier <michael@paquier.xyz>
2023-09-19 07:26 ` Re: BUG #18057: unaccent removes intentional spaces Michael Paquier <michael@paquier.xyz>
@ 2023-09-20 23:24 ` Michael Paquier <michael@paquier.xyz>
0 siblings, 0 replies; 10+ messages in thread
From: Michael Paquier @ 2023-09-20 23:24 UTC (permalink / raw)
To: Tom Lane <tgl@sss.pgh.pa.us>; +Cc: martin@schlossarek.me; pgsql-bugs@lists.postgresql.org
On Tue, Sep 19, 2023 at 04:26:43PM +0900, Michael Paquier wrote:
> The patch has been updated to improve that by making the error state a
> bit smarter, and I have removed the previous tests with the custom
> file and its fancy rules. Any comments and/or objections?
And applied as of 59f47fb98dab.
--
Michael
Attachments:
signature.asc (application/pgp-signature, 832B) download
^ permalink raw reply [nested|flat] 10+ messages in thread
end of thread, other threads:[~2023-09-20 23:24 UTC | newest]
Thread overview: 10+ messages (download: mbox mbox.gz follow: Atom feed)
-- links below jump to the message on this page --
2023-08-15 19:54 BUG #18057: unaccent removes intentional spaces PG Bug reporting form <noreply@postgresql.org>
2023-08-16 00:00 ` Michael Paquier <michael@paquier.xyz>
2023-08-18 06:18 ` Michael Paquier <michael@paquier.xyz>
2023-08-19 16:30 ` Tom Lane <tgl@sss.pgh.pa.us>
2023-08-19 23:57 ` Michael Paquier <michael@paquier.xyz>
2023-08-20 00:08 ` Tom Lane <tgl@sss.pgh.pa.us>
2023-08-20 00:20 ` Michael Paquier <michael@paquier.xyz>
2023-08-21 07:14 ` Michael Paquier <michael@paquier.xyz>
2023-09-19 07:26 ` Michael Paquier <michael@paquier.xyz>
2023-09-20 23:24 ` Michael Paquier <michael@paquier.xyz>
This inbox is served by DDX for PostgreSQL; see mirroring instructions
for how to clone and mirror all data and code used for this inbox