public inbox for [email protected]  

help / color / mirror / Atom feed

From: Tom Lane <[email protected]>
To: PostgreSQL Hackers <[email protected]>
Cc: Егор Чиндяскин <[email protected]>
Cc: Richard Guo <[email protected]>
Cc: Alvaro Herrera <[email protected]>
Cc: mahendrakar s <[email protected]>
Subject: Re: Stack overflow issue
Date: Tue, 30 Aug 2022 11:02:38 -0400
Message-ID: <[email protected]> (raw)
In-Reply-To: <[email protected]>
References: <CABkiuWrDt75d1pFaahQ0AsqYxqx3AQRJ+ajd1Wv=CFTW8ZSJFQ@mail.gmail.com>
	<[email protected]>
	<CAMbWs4_p9rP=0XtVqzZf6dEFx0MHmPFOP4PhG8N8=_tqk2+yjA@mail.gmail.com>
	<CAMbWs48iswb9cXXnq1Ocy0WLdov__1eR=qenRP+a3x1PJiphJg@mail.gmail.com>
	<[email protected]>
	<[email protected]>

I wrote:
>> I think most likely we should report this to Snowball upstream
>> and see what they think is an appropriate fix.

> Done at [1], and I pushed the other fixes.  Thanks again for the report!

The upstream recommendation, which seems pretty sane to me, is to
simply reject any string exceeding some threshold length as not
possibly being a word.  Apparently it's common to use thresholds
as small as 64 bytes, but in the attached I used 1000 bytes.

			regards, tom lane



Attachments:

  [text/x-diff] limit-length-of-strings-passed-to-snowball.patch (1.2K, ../[email protected]/2-limit-length-of-strings-passed-to-snowball.patch)
  download | inline diff:
diff --git a/src/backend/snowball/dict_snowball.c b/src/backend/snowball/dict_snowball.c
index 68c9213f69..aaf4ff72b6 100644
--- a/src/backend/snowball/dict_snowball.c
+++ b/src/backend/snowball/dict_snowball.c
@@ -272,11 +272,25 @@ dsnowball_lexize(PG_FUNCTION_ARGS)
 	DictSnowball *d = (DictSnowball *) PG_GETARG_POINTER(0);
 	char	   *in = (char *) PG_GETARG_POINTER(1);
 	int32		len = PG_GETARG_INT32(2);
-	char	   *txt = lowerstr_with_len(in, len);
 	TSLexeme   *res = palloc0(sizeof(TSLexeme) * 2);
+	char	   *txt;
 
+	/*
+	 * Reject strings exceeding 1000 bytes, as they're surely not words in any
+	 * human language.  This restriction avoids wasting cycles on stuff like
+	 * base64-encoded data, and it protects us against possible inefficiency
+	 * or misbehavior in the stemmers (for example, the Turkish stemmer has an
+	 * indefinite recursion so it can crash on long-enough strings).
+	 */
+	if (len <= 0 || len > 1000)
+		PG_RETURN_POINTER(res);
+
+	txt = lowerstr_with_len(in, len);
+
+	/* txt is probably not zero-length now, but we'll check anyway */
 	if (*txt == '\0' || searchstoplist(&(d->stoplist), txt))
 	{
+		/* empty or stopword, so reject */
 		pfree(txt);
 	}
 	else

view thread (15+ messages)  latest in thread

reply

Reply instructions:

You may reply publicly to this message via plain-text email
using any one of the following methods:

* Reply to all the recipients using the --to and --cc options:
  reply via email

  To: [email protected]
  Cc: [email protected], [email protected], [email protected], [email protected], [email protected], [email protected]
  Subject: Re: Stack overflow issue
  In-Reply-To: <[email protected]>

* Save the following mbox file, import it into your mail client,
  and reply-to-all from there: mbox

This inbox is served by agora; see mirroring instructions
for how to clone and mirror all data and code used for this inbox