From: Artur Zakirov <zaartur@gmail.com>
To: Kyotaro Horiguchi <horikyota.ntt@gmail.com>
Cc: tomas.vondra@2ndquadrant.com
Cc: matti.linnanvuori@portalify.com
Cc: pgsql-bugs@lists.postgresql.org
Cc: noreply@postgresql.org
Subject: Re: BUG #16337: Finnish Ispell dictionary cannot be created
Date: Tue, 14 Apr 2020 12:44:44 +0900
Message-ID: <7ce82fff-5368-47b8-671e-31ea340b0cde@gmail.com> (raw)
In-Reply-To: <20200413.173610.1847967467851370073.horikyota.ntt@gmail.com>
References: <cae19e93-692f-d646-6ec7-3edc0ec27edb@gmail.com>
<20200403085509.bwrvm7hfh6h4vgrl@development>
<CAKNkYnxeHJJDkw3_s908oMgiv4pn0ODkqGXUxME0FvMDxhu0=g@mail.gmail.com>
<20200413.173610.1847967467851370073.horikyota.ntt@gmail.com>
Hello Horiguchi-san,
On 4/13/2020 5:36 PM, Kyotaro Horiguchi wrote:
> Looking man 5 ispell, "Any character with special meaning to parser
> can be changed to an uniterpreted token by backslashing it". It
> depends on how we sholud be strict on that, but I think it is safer
> that we think that any character prefixed by a backslash is an word
> character. (I don't understand how '-' can be in a word by the
> definition in the .affix file, though.)
>
> Since a escaped character is intended to be a part of a word, there's
> no point in identifying minus-sign ad-hockerly, I think.
Thank you to pay attention to the patch.
I don't mind if the patch will work in more broad cases. But I tested
ispell utility with other characters other than '-' before. It seems
that it ignores such affixes or doesn't work properly. But in general
maybe it is better to stick closer with the man page description.
I attached new version of the patch. It fixes only PAE_INFIND and
PAE_INREPL cases. I think we shouldn't allow to escape all cases and it
is safer to have some exceptions:
- In PAE_WAIT_MASK we shouldn't escape comment string which starts with '#'
- PAE_INMASK case is handled by regcomp.c separately and maybe it is
better to leave the string as-is
- PAE_WAIT_FIND can start only with '-'
- I don't think that there is a sense in escaping PAE_WAIT_REPL
And in PAE_INFIND and PAE_INREPL I think we shouldn't allow to escape
',' and '#'.
The condition:
if (t_iseq(str, '\\') && !isescaped &&
(state == PAE_INFIND || state == PAE_INREPL))
maybe is not great, but I cannot come up with a better solution.
--
Artur
Attachments:tsearch_escape_hyphen_v2.patch (text/plain, 3.4K) downloaddiff --git a/src/backend/tsearch/dicts/ispell_sample.affix b/src/backend/tsearch/dicts/ispell_sample.affixindex f29004ff1d..93b8d5435c 100644--- a/src/backend/tsearch/dicts/ispell_sample.affix+++ b/src/backend/tsearch/dicts/ispell_sample.affix@@ -5,6 +5,9 @@ prefixes
flag *B:
. > RE # As in enter > reenter
+flag E:+ . > IN\- # As in law > in-law+
flag U:
. > UN # As in natural > unnatural
diff --git a/src/backend/tsearch/dicts/ispell_sample.dict b/src/backend/tsearch/dicts/ispell_sample.dictindex 44df1967a6..558c7154c1 100644--- a/src/backend/tsearch/dicts/ispell_sample.dict+++ b/src/backend/tsearch/dicts/ispell_sample.dict@@ -5,4 +5,5 @@ foot/ZS
football/Z
ball/SZ\
klubber/Z
+law/E
sky/A
diff --git a/src/backend/tsearch/spell.c b/src/backend/tsearch/spell.cindex 8aab96d3b0..ccb46339e2 100644--- a/src/backend/tsearch/spell.c+++ b/src/backend/tsearch/spell.c@@ -909,11 +909,25 @@ parse_affentry(char *str, char *mask, char *find, char *repl)
char *pmask = mask,
*pfind = find,
*prepl = repl;
+ bool isescaped = false;
*mask = *find = *repl = '\0';
while (*str)
{
+ if (t_iseq(str, '\\') && !isescaped &&+ (state == PAE_INFIND || state == PAE_INREPL))+ {+ /*+ * Next character should be escaped. Currently it is applyed only to+ * PAE_INFIND and PAE_INREPL. We expect that <find> can start only+ * with '-' and <replace> can start only with an alphabet character.+ */+ isescaped = true;+ str += pg_mblen(str);+ continue;+ }+
if (state == PAE_WAIT_MASK)
{
if (t_iseq(str, '#'))
@@ -962,7 +976,7 @@ parse_affentry(char *str, char *mask, char *find, char *repl)
*pfind = '\0';
state = PAE_WAIT_REPL;
}
- else if (t_isalpha(str))+ else if (t_isalpha(str) || isescaped)
{
COPYCHAR(pfind, str);
pfind += pg_mblen(str);
@@ -996,7 +1010,7 @@ parse_affentry(char *str, char *mask, char *find, char *repl)
*prepl = '\0';
break;
}
- else if (t_isalpha(str))+ else if (t_isalpha(str) || isescaped)
{
COPYCHAR(prepl, str);
prepl += pg_mblen(str);
@@ -1009,6 +1023,7 @@ parse_affentry(char *str, char *mask, char *find, char *repl)
else
elog(ERROR, "unrecognized state in parse_affentry: %d", state);
+ isescaped = false;
str += pg_mblen(str);
}
diff --git a/src/test/regress/expected/tsdicts.out b/src/test/regress/expected/tsdicts.outindex c804293142..5a0353f552 100644--- a/src/test/regress/expected/tsdicts.out+++ b/src/test/regress/expected/tsdicts.out@@ -71,6 +71,19 @@ SELECT ts_lexize('ispell', 'unbook');
{book}
(1 row)
+-- Test for hyphen escaping+SELECT ts_lexize('ispell', 'in-law');+ ts_lexize +-----------+ {law}+(1 row)++SELECT ts_lexize('ispell', 'law');+ ts_lexize +-----------+ {law}+(1 row)+
SELECT ts_lexize('ispell', 'footklubber');
ts_lexize
----------------
diff --git a/src/test/regress/sql/tsdicts.sql b/src/test/regress/sql/tsdicts.sqlindex ddc6c7f445..cf901ec08b 100644--- a/src/test/regress/sql/tsdicts.sql+++ b/src/test/regress/sql/tsdicts.sql@@ -19,6 +19,10 @@ SELECT ts_lexize('ispell', 'unbookings');
SELECT ts_lexize('ispell', 'unbooking');
SELECT ts_lexize('ispell', 'unbook');
+-- Test for hyphen escaping+SELECT ts_lexize('ispell', 'in-law');+SELECT ts_lexize('ispell', 'law');+
SELECT ts_lexize('ispell', 'footklubber');
SELECT ts_lexize('ispell', 'footballklubber');
SELECT ts_lexize('ispell', 'ballyklubber');
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Reply to all the recipients using the --to and --cc options:
reply via email
To: pgsql-bugs@postgresql.org
Cc: zaartur@gmail.com, horikyota.ntt@gmail.com, tomas.vondra@2ndquadrant.com, matti.linnanvuori@portalify.com, pgsql-bugs@lists.postgresql.org, noreply@postgresql.org
Subject: Re: BUG #16337: Finnish Ispell dictionary cannot be created
In-Reply-To: <7ce82fff-5368-47b8-671e-31ea340b0cde@gmail.com>
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
This inbox is served by DDX for PostgreSQL; see mirroring instructions
for how to clone and mirror all data and code used for this inbox