pg.ddx.io  pgsql-committers@postgresql.org mailing list archive  
help / color / mirror / Atom feed
From: Tom Lane <tgl@sss.pgh.pa.us>
To: pgsql-committers@lists.postgresql.org
Subject: pgsql: In the Snowball dictionary, don't try to stem excessively-long w
Date: Wed, 31 Aug 2022 14:42:21 +0000
Message-ID: <E1oTOv7-000llL-1Y@gemulon.postgresql.org> (raw)

In the Snowball dictionary, don't try to stem excessively-long words.

If the input word exceeds 1000 bytes, don't pass it to the stemmer;
just return it as-is after case folding.  Such an input is surely
not a word in any human language, so whatever the stemmer might
do to it would be pretty dubious in the first place.  Adding this
restriction protects us against a known recursion-to-stack-overflow
problem in the Turkish stemmer, and it seems like good insurance
against any other safety or performance issues that may exist in
the Snowball stemmers.  (I note, for example, that they contain no
CHECK_FOR_INTERRUPTS calls, so we really don't want them running
for a long time.)  The threshold of 1000 bytes is arbitrary.

An alternative definition could have been to treat such words as
stopwords, but that seems like a bigger break from the old behavior.

Per report from Egor Chindyaskin and Alexander Lakhin.
Thanks to Olly Betts for the recommendation to fix it this way.

Discussion: https://postgr.es/m/1661334672.728714027@f473.i.mail.ru

Branch
------
REL_10_STABLE

Details
-------
https://git.postgresql.org/pg/commitdiff/cba393a10af870669a4d69248669179e28e4089b

Modified Files
--------------
src/backend/snowball/dict_snowball.c | 18 +++++++++++++++++-
1 file changed, 17 insertions(+), 1 deletion(-)



view thread (7+ messages)  latest in thread

Message-ID: <E1oTOv7-000llL-1Y@gemulon.postgresql.org>
Permalink:  ../E1oTOv7-000llL-1Y@gemulon.postgresql.org/
Also on:    postgresql.org/message-id/E1oTOv7-000llL-1Y@gemulon.postgresql.org

 · 

reply

Reply instructions:

You may reply publicly to this message via plain-text email
using any one of the following methods:

* Reply to all the recipients from your mail client; the link fills in
  the headers below: reply via email
  Some clients, webmail especially, drop In-Reply-To and References from
  a link, and the reply then starts a new thread: use the mbox below.

  To: pgsql-committers@postgresql.org
  Cc: tgl@sss.pgh.pa.us
  Subject: Re: pgsql: In the Snowball dictionary, don't try to stem excessively-long w
  In-Reply-To: <E1oTOv7-000llL-1Y@gemulon.postgresql.org>
  References: <E1oTOv7-000llL-1Y@gemulon.postgresql.org>

* Save the following mbox file, import it into your mail client,
  and reply-to-all from there: mbox

This inbox is served by DDX for PostgreSQL; see mirroring instructions
for how to clone and mirror all data and code used for this inbox