pg.ddx.io pgsql-bugs@postgresql.org mailing list archive
help / color / mirror / Atom feedText search prefix matching and stop words
5+ messages / 4 participants
[nested] [flat]
* Text search prefix matching and stop words
@ 2021-10-08 18:17 Matthew Nelson <mnelson@binarykeep.com>
2021-10-08 20:30 ` Re: Text search prefix matching and stop words Pavel Borisov <pashkin.elfe@gmail.com>
0 siblings, 1 reply; 5+ messages in thread
From: Matthew Nelson @ 2021-10-08 18:17 UTC (permalink / raw)
To: pgsql-bugs@lists.postgresql.org
Prefix matching should not omit stop words, as matching lexemes may legitimately begin with stop words.
# select to_tsquery('english', 'over:*') @@ to_tsvector('english', 'overhaul');
NOTICE: text-search query contains only stop words or doesn't contain lexemes, ignored
?column?
----------
f
(1 row)
I noticed this after implementing interactive, incremental search in an application. As the user typed "overhaul," with each successive character executing a search, "ove" and "overh" matched a particular document, but "over" did not.
Reproduced in PostgreSQL 11, 13, and 14.
^ permalink raw reply [nested|flat] 5+ messages in thread
* Re: Text search prefix matching and stop words
2021-10-08 18:17 Text search prefix matching and stop words Matthew Nelson <mnelson@binarykeep.com>
@ 2021-10-08 20:30 ` Pavel Borisov <pashkin.elfe@gmail.com>
2021-10-08 20:32 ` Re: Text search prefix matching and stop words Pavel Borisov <pashkin.elfe@gmail.com>
2021-10-08 21:06 ` Re: Text search prefix matching and stop words Tom Lane <tgl@sss.pgh.pa.us>
2021-10-11 14:50 ` Re: Text search prefix matching and stop words Artur Zakirov <zaartur@gmail.com>
0 siblings, 3 replies; 5+ messages in thread
From: Pavel Borisov @ 2021-10-08 20:30 UTC (permalink / raw)
To: mnelson@binarykeep.com; +Cc: PostgreSQL mailing lists <pgsql-bugs@lists.postgresql.org>
>
> Prefix matching should not omit stop words, as matching lexemes may
> legitimately begin with stop words.
>
> # select to_tsquery('english', 'over:*') @@ to_tsvector('english',
> 'overhaul');
> NOTICE: text-search query contains only stop words or doesn't contain
> lexemes, ignored
> ?column?
> ----------
> f
> (1 row)
>
> I noticed this after implementing interactive, incremental search in an
> application. As the user typed "overhaul," with each successive character
> executing a search, "ove" and "overh" matched a particular document, but
> "over" did not.
Big thanks for the reporting!
I am not sure that it is a bug. I think this is a way how to_tsquery
conversion work: stopwords first then template processing.
If you want to process successive characters typing, you can use casting to
tsvector type until input is not finished
'over:*'::tsquery;
and when the user finishes input then process the result via to_tsquery
with stop words.
if we do to_tsquery in a way you described I expect it will never apply the
stop-word filter on templated input as it can not be compared to stop words.
--
Best regards,
Pavel Borisov
Postgres Professional: http://postgrespro.com <http://www.postgrespro.com;
^ permalink raw reply [nested|flat] 5+ messages in thread
* Re: Text search prefix matching and stop words
2021-10-08 18:17 Text search prefix matching and stop words Matthew Nelson <mnelson@binarykeep.com>
2021-10-08 20:30 ` Re: Text search prefix matching and stop words Pavel Borisov <pashkin.elfe@gmail.com>
@ 2021-10-08 20:32 ` Pavel Borisov <pashkin.elfe@gmail.com>
2 siblings, 0 replies; 5+ messages in thread
From: Pavel Borisov @ 2021-10-08 20:32 UTC (permalink / raw)
To: mnelson@binarykeep.com; +Cc: PostgreSQL mailing lists <pgsql-bugs@lists.postgresql.org>
I must commect myself
"If you want to process successive characters typing, you can use casting
to *tsquery* type until input is not finished"
^ permalink raw reply [nested|flat] 5+ messages in thread
* Re: Text search prefix matching and stop words
2021-10-08 18:17 Text search prefix matching and stop words Matthew Nelson <mnelson@binarykeep.com>
2021-10-08 20:30 ` Re: Text search prefix matching and stop words Pavel Borisov <pashkin.elfe@gmail.com>
@ 2021-10-08 21:06 ` Tom Lane <tgl@sss.pgh.pa.us>
2 siblings, 0 replies; 5+ messages in thread
From: Tom Lane @ 2021-10-08 21:06 UTC (permalink / raw)
To: Pavel Borisov <pashkin.elfe@gmail.com>; +Cc: mnelson@binarykeep.com, PostgreSQL mailing lists <pgsql-bugs@lists.postgresql.org>
Pavel Borisov <pashkin.elfe@gmail.com> writes:
>> Prefix matching should not omit stop words, as matching lexemes may
>> legitimately begin with stop words.
> I am not sure that it is a bug. I think this is a way how to_tsquery
> conversion work: stopwords first then template processing.
I concur with the OP that this is a bug, or at least that it'd be nice
if it worked better. But I'm not sure we can make it better. The basic
design of our text search stuff combined the functions of normalization
and stop-word-suppression into a single dictionary stack, so that it's
impossible to ask for just one of those to happen. But if we skip
applying the dictionaries at all for a prefix item, then word
normalization doesn't happen, which would create a different set of
unexpected-failure-to-match conditions. (So your proposed workaround
of casting directly to tsquery just moves the problem somewhere else.)
I think we could only fix this with a dictionary API change that
allows telling the dictionaries not to suppress stopwords. Not
sure how practical that is. If we'd had the prefix-match feature
from the beginning, maybe it'd have occurred to us that we needed
that API option ... but we didn't.
regards, tom lane
^ permalink raw reply [nested|flat] 5+ messages in thread
* Re: Text search prefix matching and stop words
2021-10-08 18:17 Text search prefix matching and stop words Matthew Nelson <mnelson@binarykeep.com>
2021-10-08 20:30 ` Re: Text search prefix matching and stop words Pavel Borisov <pashkin.elfe@gmail.com>
@ 2021-10-11 14:50 ` Artur Zakirov <zaartur@gmail.com>
2 siblings, 0 replies; 5+ messages in thread
From: Artur Zakirov @ 2021-10-11 14:50 UTC (permalink / raw)
To: Pavel Borisov <pashkin.elfe@gmail.com>; +Cc: mnelson@binarykeep.com, PostgreSQL mailing lists <pgsql-bugs@lists.postgresql.org>
On Fri, Oct 8, 2021 at 10:31 PM Pavel Borisov <pashkin.elfe@gmail.com> wrote:
> If you want to process successive characters typing, you can use casting to tsvector type until input is not finished
>
> 'over:*'::tsquery;
Also it is possible to use a custom configuration without stop words
if you want normalization:
postgres=# select to_tsquery('english_wo_stop', 'over:*') &&
to_tsquery('english', 'foo');
?column?
------------------
'over':* & 'foo'
--
Artur
^ permalink raw reply [nested|flat] 5+ messages in thread
end of thread, other threads:[~2021-10-11 14:50 UTC | newest]
Thread overview: 5+ messages (download: mbox mbox.gz follow: Atom feed)
-- links below jump to the message on this page --
2021-10-08 18:17 Text search prefix matching and stop words Matthew Nelson <mnelson@binarykeep.com>
2021-10-08 20:30 ` Pavel Borisov <pashkin.elfe@gmail.com>
2021-10-08 20:32 ` Pavel Borisov <pashkin.elfe@gmail.com>
2021-10-08 21:06 ` Tom Lane <tgl@sss.pgh.pa.us>
2021-10-11 14:50 ` Artur Zakirov <zaartur@gmail.com>
This inbox is served by DDX for PostgreSQL; see mirroring instructions
for how to clone and mirror all data and code used for this inbox