public inbox for [email protected]
help / color / mirror / Atom feedFrom: Tom Lane <[email protected]>
To: PostgreSQL Hackers <[email protected]>
Cc: Егор Чиндяскин <[email protected]>
Cc: Richard Guo <[email protected]>
Cc: Alvaro Herrera <[email protected]>
Cc: mahendrakar s <[email protected]>
Subject: Re: Stack overflow issue
Date: Tue, 30 Aug 2022 18:57:06 -0400
Message-ID: <[email protected]> (raw)
In-Reply-To: <[email protected]>
References: <CABkiuWrDt75d1pFaahQ0AsqYxqx3AQRJ+ajd1Wv=CFTW8ZSJFQ@mail.gmail.com>
<[email protected]>
<CAMbWs4_p9rP=0XtVqzZf6dEFx0MHmPFOP4PhG8N8=_tqk2+yjA@mail.gmail.com>
<CAMbWs48iswb9cXXnq1Ocy0WLdov__1eR=qenRP+a3x1PJiphJg@mail.gmail.com>
<[email protected]>
<[email protected]>
<[email protected]>
I wrote:
> The upstream recommendation, which seems pretty sane to me, is to
> simply reject any string exceeding some threshold length as not
> possibly being a word. Apparently it's common to use thresholds
> as small as 64 bytes, but in the attached I used 1000 bytes.
On further thought: that coding treats anything longer than 1000
bytes as a stopword, but maybe we should just accept it unmodified.
The manual says "A Snowball dictionary recognizes everything, whether
or not it is able to simplify the word". While "recognizes" formally
includes the case of "recognizes as a stopword", people might find
this behavior surprising. We could alternatively do it as attached,
which accepts overlength words but does nothing to them except
case-fold. This is closer to the pre-patch behavior, but gives up
the opportunity to avoid useless downstream processing of long words.
regards, tom lane
Attachments:
[text/x-diff] limit-length-of-strings-passed-to-snowball-2.patch (1.2K, ../[email protected]/2-limit-length-of-strings-passed-to-snowball-2.patch)
download | inline diff:
diff --git a/src/backend/snowball/dict_snowball.c b/src/backend/snowball/dict_snowball.c
index 68c9213f69..1d5dfff5a0 100644
--- a/src/backend/snowball/dict_snowball.c
+++ b/src/backend/snowball/dict_snowball.c
@@ -275,8 +275,24 @@ dsnowball_lexize(PG_FUNCTION_ARGS)
char *txt = lowerstr_with_len(in, len);
TSLexeme *res = palloc0(sizeof(TSLexeme) * 2);
- if (*txt == '\0' || searchstoplist(&(d->stoplist), txt))
+ /*
+ * Do not pass strings exceeding 1000 bytes to the stemmer, as they're
+ * surely not words in any human language. This restriction avoids
+ * wasting cycles on stuff like base64-encoded data, and it protects us
+ * against possible inefficiency or misbehavior in the stemmer. (For
+ * example, the Turkish stemmer has an indefinite recursion, so it can
+ * crash on long-enough strings.) However, Snowball dictionaries are
+ * defined to recognize all strings, so we can't reject the string as an
+ * unknown word.
+ */
+ if (len > 1000)
+ {
+ /* return the lexeme lowercased, but otherwise unmodified */
+ res->lexeme = txt;
+ }
+ else if (*txt == '\0' || searchstoplist(&(d->stoplist), txt))
{
+ /* empty or stopword, so report as stopword */
pfree(txt);
}
else
view thread (15+ messages) latest in thread
reply
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Reply to all the recipients using the --to and --cc options:
reply via email
To: [email protected]
Cc: [email protected], [email protected], [email protected], [email protected], [email protected], [email protected]
Subject: Re: Stack overflow issue
In-Reply-To: <[email protected]>
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
This inbox is served by agora; see mirroring instructions
for how to clone and mirror all data and code used for this inbox