pg.ddx.io  pgsql-bugs@postgresql.org mailing list archive  
help / color / mirror / Atom feed
From: Artur Zakirov <zaartur@gmail.com>
To: Kyotaro Horiguchi <horikyota.ntt@gmail.com>
Cc: tomas.vondra@2ndquadrant.com
Cc: matti.linnanvuori@portalify.com
Cc: pgsql-bugs@lists.postgresql.org
Cc: noreply@postgresql.org
Subject: Re: BUG #16337: Finnish Ispell dictionary cannot be created
Date: Tue, 14 Apr 2020 12:44:44 +0900
Message-ID: <7ce82fff-5368-47b8-671e-31ea340b0cde@gmail.com> (raw)
In-Reply-To: <20200413.173610.1847967467851370073.horikyota.ntt@gmail.com>
References: <cae19e93-692f-d646-6ec7-3edc0ec27edb@gmail.com>
	<20200403085509.bwrvm7hfh6h4vgrl@development>
	<CAKNkYnxeHJJDkw3_s908oMgiv4pn0ODkqGXUxME0FvMDxhu0=g@mail.gmail.com>
	<20200413.173610.1847967467851370073.horikyota.ntt@gmail.com>

Hello Horiguchi-san,

On 4/13/2020 5:36 PM, Kyotaro Horiguchi wrote:
> Looking man 5 ispell, "Any character with special meaning to parser
> can be changed to an uniterpreted token by backslashing it". It
> depends on how we sholud be strict on that, but I think it is safer
> that we think that any character prefixed by a backslash is an word
> character.  (I don't understand how '-' can be in a word by the
> definition in the .affix file, though.)
> 
> Since a escaped character is intended to be a part of a word, there's
> no point in identifying minus-sign ad-hockerly, I think.

Thank you to pay attention to the patch.

I don't mind if the patch will work in more broad cases. But I tested 
ispell utility with other characters other than '-' before. It seems 
that it ignores such affixes or doesn't work properly. But in general 
maybe it is better to stick closer with the man page description.

I attached new version of the patch. It fixes only PAE_INFIND and 
PAE_INREPL cases. I think we shouldn't allow to escape all cases and it 
is safer to have some exceptions:
- In PAE_WAIT_MASK we shouldn't escape comment string which starts with '#'
- PAE_INMASK case is handled by regcomp.c separately and maybe it is 
better to leave the string as-is
- PAE_WAIT_FIND can start only with '-'
- I don't think that there is a sense in escaping PAE_WAIT_REPL

And in PAE_INFIND and PAE_INREPL I think we shouldn't allow to escape 
',' and '#'.

The condition:

if (t_iseq(str, '\\') && !isescaped &&
	(state == PAE_INFIND || state == PAE_INREPL))

maybe is not great, but I cannot come up with a better solution.

-- 
Artur

Attachments:

tsearch_escape_hyphen_v2.patch (text/plain, 3.4K) download diff --git a/src/backend/tsearch/dicts/ispell_sample.affix b/src/backend/tsearch/dicts/ispell_sample.affix index f29004ff1d..93b8d5435c 100644 --- a/src/backend/tsearch/dicts/ispell_sample.affix +++ b/src/backend/tsearch/dicts/ispell_sample.affix @@ -5,6 +5,9 @@ prefixes flag *B: . > RE # As in enter > reenter +flag E: + . > IN\- # As in law > in-law + flag U: . > UN # As in natural > unnatural diff --git a/src/backend/tsearch/dicts/ispell_sample.dict b/src/backend/tsearch/dicts/ispell_sample.dict index 44df1967a6..558c7154c1 100644 --- a/src/backend/tsearch/dicts/ispell_sample.dict +++ b/src/backend/tsearch/dicts/ispell_sample.dict @@ -5,4 +5,5 @@ foot/ZS football/Z ball/SZ\ klubber/Z +law/E sky/A diff --git a/src/backend/tsearch/spell.c b/src/backend/tsearch/spell.c index 8aab96d3b0..ccb46339e2 100644 --- a/src/backend/tsearch/spell.c +++ b/src/backend/tsearch/spell.c @@ -909,11 +909,25 @@ parse_affentry(char *str, char *mask, char *find, char *repl) char *pmask = mask, *pfind = find, *prepl = repl; + bool isescaped = false; *mask = *find = *repl = '\0'; while (*str) { + if (t_iseq(str, '\\') && !isescaped && + (state == PAE_INFIND || state == PAE_INREPL)) + { + /* + * Next character should be escaped. Currently it is applyed only to + * PAE_INFIND and PAE_INREPL. We expect that <find> can start only + * with '-' and <replace> can start only with an alphabet character. + */ + isescaped = true; + str += pg_mblen(str); + continue; + } + if (state == PAE_WAIT_MASK) { if (t_iseq(str, '#')) @@ -962,7 +976,7 @@ parse_affentry(char *str, char *mask, char *find, char *repl) *pfind = '\0'; state = PAE_WAIT_REPL; } - else if (t_isalpha(str)) + else if (t_isalpha(str) || isescaped) { COPYCHAR(pfind, str); pfind += pg_mblen(str); @@ -996,7 +1010,7 @@ parse_affentry(char *str, char *mask, char *find, char *repl) *prepl = '\0'; break; } - else if (t_isalpha(str)) + else if (t_isalpha(str) || isescaped) { COPYCHAR(prepl, str); prepl += pg_mblen(str); @@ -1009,6 +1023,7 @@ parse_affentry(char *str, char *mask, char *find, char *repl) else elog(ERROR, "unrecognized state in parse_affentry: %d", state); + isescaped = false; str += pg_mblen(str); } diff --git a/src/test/regress/expected/tsdicts.out b/src/test/regress/expected/tsdicts.out index c804293142..5a0353f552 100644 --- a/src/test/regress/expected/tsdicts.out +++ b/src/test/regress/expected/tsdicts.out @@ -71,6 +71,19 @@ SELECT ts_lexize('ispell', 'unbook'); {book} (1 row) +-- Test for hyphen escaping +SELECT ts_lexize('ispell', 'in-law'); + ts_lexize +----------- + {law} +(1 row) + +SELECT ts_lexize('ispell', 'law'); + ts_lexize +----------- + {law} +(1 row) + SELECT ts_lexize('ispell', 'footklubber'); ts_lexize ---------------- diff --git a/src/test/regress/sql/tsdicts.sql b/src/test/regress/sql/tsdicts.sql index ddc6c7f445..cf901ec08b 100644 --- a/src/test/regress/sql/tsdicts.sql +++ b/src/test/regress/sql/tsdicts.sql @@ -19,6 +19,10 @@ SELECT ts_lexize('ispell', 'unbookings'); SELECT ts_lexize('ispell', 'unbooking'); SELECT ts_lexize('ispell', 'unbook'); +-- Test for hyphen escaping +SELECT ts_lexize('ispell', 'in-law'); +SELECT ts_lexize('ispell', 'law'); + SELECT ts_lexize('ispell', 'footklubber'); SELECT ts_lexize('ispell', 'footballklubber'); SELECT ts_lexize('ispell', 'ballyklubber');

view thread (6+ messages)

Message-ID: <7ce82fff-5368-47b8-671e-31ea340b0cde@gmail.com>
Permalink:  ../7ce82fff-5368-47b8-671e-31ea340b0cde@gmail.com/
Also on:    postgresql.org/message-id/7ce82fff-5368-47b8-671e-31ea340b0cde@gmail.com

 ·  · 

reply

Reply instructions:

You may reply publicly to this message via plain-text email
using any one of the following methods:

* Reply to all the recipients using the --to and --cc options:
  reply via email

  To: pgsql-bugs@postgresql.org
  Cc: zaartur@gmail.com, horikyota.ntt@gmail.com, tomas.vondra@2ndquadrant.com, matti.linnanvuori@portalify.com, pgsql-bugs@lists.postgresql.org, noreply@postgresql.org
  Subject: Re: BUG #16337: Finnish Ispell dictionary cannot be created
  In-Reply-To: <7ce82fff-5368-47b8-671e-31ea340b0cde@gmail.com>

* Save the following mbox file, import it into your mail client,
  and reply-to-all from there: mbox

This inbox is served by DDX for PostgreSQL; see mirroring instructions
for how to clone and mirror all data and code used for this inbox