pg.ddx.io  pgsql-committers@postgresql.org mailing list archive  
help / color / mirror / Atom feed
pgsql: In the Snowball dictionary, don't try to stem excessively-long w
7+ messages / 1 participants
[nested] [flat]

* pgsql: In the Snowball dictionary, don't try to stem excessively-long w
@ 2022-08-31 14:42 Tom Lane <tgl@sss.pgh.pa.us>
  0 siblings, 0 replies; 7+ messages in thread

From: Tom Lane @ 2022-08-31 14:42 UTC (permalink / raw)
  To: pgsql-committers@lists.postgresql.org

In the Snowball dictionary, don't try to stem excessively-long words.

If the input word exceeds 1000 bytes, don't pass it to the stemmer;
just return it as-is after case folding.  Such an input is surely
not a word in any human language, so whatever the stemmer might
do to it would be pretty dubious in the first place.  Adding this
restriction protects us against a known recursion-to-stack-overflow
problem in the Turkish stemmer, and it seems like good insurance
against any other safety or performance issues that may exist in
the Snowball stemmers.  (I note, for example, that they contain no
CHECK_FOR_INTERRUPTS calls, so we really don't want them running
for a long time.)  The threshold of 1000 bytes is arbitrary.

An alternative definition could have been to treat such words as
stopwords, but that seems like a bigger break from the old behavior.

Per report from Egor Chindyaskin and Alexander Lakhin.
Thanks to Olly Betts for the recommendation to fix it this way.

Discussion: https://postgr.es/m/1661334672.728714027@f473.i.mail.ru

Branch
------
master

Details
-------
https://git.postgresql.org/pg/commitdiff/1058555a5eccffd17c47edf3e15dd7e26934b492

Modified Files
--------------
src/backend/snowball/dict_snowball.c | 18 +++++++++++++++++-
1 file changed, 17 insertions(+), 1 deletion(-)



^ permalink  raw  reply  [nested|flat] 7+ messages in thread

* pgsql: In the Snowball dictionary, don't try to stem excessively-long w
@ 2022-08-31 14:42 Tom Lane <tgl@sss.pgh.pa.us>
  0 siblings, 0 replies; 7+ messages in thread

From: Tom Lane @ 2022-08-31 14:42 UTC (permalink / raw)
  To: pgsql-committers@lists.postgresql.org

In the Snowball dictionary, don't try to stem excessively-long words.

If the input word exceeds 1000 bytes, don't pass it to the stemmer;
just return it as-is after case folding.  Such an input is surely
not a word in any human language, so whatever the stemmer might
do to it would be pretty dubious in the first place.  Adding this
restriction protects us against a known recursion-to-stack-overflow
problem in the Turkish stemmer, and it seems like good insurance
against any other safety or performance issues that may exist in
the Snowball stemmers.  (I note, for example, that they contain no
CHECK_FOR_INTERRUPTS calls, so we really don't want them running
for a long time.)  The threshold of 1000 bytes is arbitrary.

An alternative definition could have been to treat such words as
stopwords, but that seems like a bigger break from the old behavior.

Per report from Egor Chindyaskin and Alexander Lakhin.
Thanks to Olly Betts for the recommendation to fix it this way.

Discussion: https://postgr.es/m/1661334672.728714027@f473.i.mail.ru

Branch
------
REL_15_STABLE

Details
-------
https://git.postgresql.org/pg/commitdiff/61f6e5c2360382be99846be904af33e3406c938a

Modified Files
--------------
src/backend/snowball/dict_snowball.c | 18 +++++++++++++++++-
1 file changed, 17 insertions(+), 1 deletion(-)



^ permalink  raw  reply  [nested|flat] 7+ messages in thread

* pgsql: In the Snowball dictionary, don't try to stem excessively-long w
@ 2022-08-31 14:42 Tom Lane <tgl@sss.pgh.pa.us>
  0 siblings, 0 replies; 7+ messages in thread

From: Tom Lane @ 2022-08-31 14:42 UTC (permalink / raw)
  To: pgsql-committers@lists.postgresql.org

In the Snowball dictionary, don't try to stem excessively-long words.

If the input word exceeds 1000 bytes, don't pass it to the stemmer;
just return it as-is after case folding.  Such an input is surely
not a word in any human language, so whatever the stemmer might
do to it would be pretty dubious in the first place.  Adding this
restriction protects us against a known recursion-to-stack-overflow
problem in the Turkish stemmer, and it seems like good insurance
against any other safety or performance issues that may exist in
the Snowball stemmers.  (I note, for example, that they contain no
CHECK_FOR_INTERRUPTS calls, so we really don't want them running
for a long time.)  The threshold of 1000 bytes is arbitrary.

An alternative definition could have been to treat such words as
stopwords, but that seems like a bigger break from the old behavior.

Per report from Egor Chindyaskin and Alexander Lakhin.
Thanks to Olly Betts for the recommendation to fix it this way.

Discussion: https://postgr.es/m/1661334672.728714027@f473.i.mail.ru

Branch
------
REL_14_STABLE

Details
-------
https://git.postgresql.org/pg/commitdiff/e969f1ae2b01d7b69371332b839fa16e3b54e56d

Modified Files
--------------
src/backend/snowball/dict_snowball.c | 18 +++++++++++++++++-
1 file changed, 17 insertions(+), 1 deletion(-)



^ permalink  raw  reply  [nested|flat] 7+ messages in thread

* pgsql: In the Snowball dictionary, don't try to stem excessively-long w
@ 2022-08-31 14:42 Tom Lane <tgl@sss.pgh.pa.us>
  0 siblings, 0 replies; 7+ messages in thread

From: Tom Lane @ 2022-08-31 14:42 UTC (permalink / raw)
  To: pgsql-committers@lists.postgresql.org

In the Snowball dictionary, don't try to stem excessively-long words.

If the input word exceeds 1000 bytes, don't pass it to the stemmer;
just return it as-is after case folding.  Such an input is surely
not a word in any human language, so whatever the stemmer might
do to it would be pretty dubious in the first place.  Adding this
restriction protects us against a known recursion-to-stack-overflow
problem in the Turkish stemmer, and it seems like good insurance
against any other safety or performance issues that may exist in
the Snowball stemmers.  (I note, for example, that they contain no
CHECK_FOR_INTERRUPTS calls, so we really don't want them running
for a long time.)  The threshold of 1000 bytes is arbitrary.

An alternative definition could have been to treat such words as
stopwords, but that seems like a bigger break from the old behavior.

Per report from Egor Chindyaskin and Alexander Lakhin.
Thanks to Olly Betts for the recommendation to fix it this way.

Discussion: https://postgr.es/m/1661334672.728714027@f473.i.mail.ru

Branch
------
REL_13_STABLE

Details
-------
https://git.postgresql.org/pg/commitdiff/f204ad3a2b19a33a72f0401b8640f68527539bd1

Modified Files
--------------
src/backend/snowball/dict_snowball.c | 18 +++++++++++++++++-
1 file changed, 17 insertions(+), 1 deletion(-)



^ permalink  raw  reply  [nested|flat] 7+ messages in thread

* pgsql: In the Snowball dictionary, don't try to stem excessively-long w
@ 2022-08-31 14:42 Tom Lane <tgl@sss.pgh.pa.us>
  0 siblings, 0 replies; 7+ messages in thread

From: Tom Lane @ 2022-08-31 14:42 UTC (permalink / raw)
  To: pgsql-committers@lists.postgresql.org

In the Snowball dictionary, don't try to stem excessively-long words.

If the input word exceeds 1000 bytes, don't pass it to the stemmer;
just return it as-is after case folding.  Such an input is surely
not a word in any human language, so whatever the stemmer might
do to it would be pretty dubious in the first place.  Adding this
restriction protects us against a known recursion-to-stack-overflow
problem in the Turkish stemmer, and it seems like good insurance
against any other safety or performance issues that may exist in
the Snowball stemmers.  (I note, for example, that they contain no
CHECK_FOR_INTERRUPTS calls, so we really don't want them running
for a long time.)  The threshold of 1000 bytes is arbitrary.

An alternative definition could have been to treat such words as
stopwords, but that seems like a bigger break from the old behavior.

Per report from Egor Chindyaskin and Alexander Lakhin.
Thanks to Olly Betts for the recommendation to fix it this way.

Discussion: https://postgr.es/m/1661334672.728714027@f473.i.mail.ru

Branch
------
REL_12_STABLE

Details
-------
https://git.postgresql.org/pg/commitdiff/a53e0ea782c52da0fd8f9605a63aba276fc17c13

Modified Files
--------------
src/backend/snowball/dict_snowball.c | 18 +++++++++++++++++-
1 file changed, 17 insertions(+), 1 deletion(-)



^ permalink  raw  reply  [nested|flat] 7+ messages in thread

* pgsql: In the Snowball dictionary, don't try to stem excessively-long w
@ 2022-08-31 14:42 Tom Lane <tgl@sss.pgh.pa.us>
  0 siblings, 0 replies; 7+ messages in thread

From: Tom Lane @ 2022-08-31 14:42 UTC (permalink / raw)
  To: pgsql-committers@lists.postgresql.org

In the Snowball dictionary, don't try to stem excessively-long words.

If the input word exceeds 1000 bytes, don't pass it to the stemmer;
just return it as-is after case folding.  Such an input is surely
not a word in any human language, so whatever the stemmer might
do to it would be pretty dubious in the first place.  Adding this
restriction protects us against a known recursion-to-stack-overflow
problem in the Turkish stemmer, and it seems like good insurance
against any other safety or performance issues that may exist in
the Snowball stemmers.  (I note, for example, that they contain no
CHECK_FOR_INTERRUPTS calls, so we really don't want them running
for a long time.)  The threshold of 1000 bytes is arbitrary.

An alternative definition could have been to treat such words as
stopwords, but that seems like a bigger break from the old behavior.

Per report from Egor Chindyaskin and Alexander Lakhin.
Thanks to Olly Betts for the recommendation to fix it this way.

Discussion: https://postgr.es/m/1661334672.728714027@f473.i.mail.ru

Branch
------
REL_11_STABLE

Details
-------
https://git.postgresql.org/pg/commitdiff/f5aa855cd8a63cd695c5f9ff097c7bc445872cdf

Modified Files
--------------
src/backend/snowball/dict_snowball.c | 18 +++++++++++++++++-
1 file changed, 17 insertions(+), 1 deletion(-)



^ permalink  raw  reply  [nested|flat] 7+ messages in thread

* pgsql: In the Snowball dictionary, don't try to stem excessively-long w
@ 2022-08-31 14:42 Tom Lane <tgl@sss.pgh.pa.us>
  0 siblings, 0 replies; 7+ messages in thread

From: Tom Lane @ 2022-08-31 14:42 UTC (permalink / raw)
  To: pgsql-committers@lists.postgresql.org

In the Snowball dictionary, don't try to stem excessively-long words.

If the input word exceeds 1000 bytes, don't pass it to the stemmer;
just return it as-is after case folding.  Such an input is surely
not a word in any human language, so whatever the stemmer might
do to it would be pretty dubious in the first place.  Adding this
restriction protects us against a known recursion-to-stack-overflow
problem in the Turkish stemmer, and it seems like good insurance
against any other safety or performance issues that may exist in
the Snowball stemmers.  (I note, for example, that they contain no
CHECK_FOR_INTERRUPTS calls, so we really don't want them running
for a long time.)  The threshold of 1000 bytes is arbitrary.

An alternative definition could have been to treat such words as
stopwords, but that seems like a bigger break from the old behavior.

Per report from Egor Chindyaskin and Alexander Lakhin.
Thanks to Olly Betts for the recommendation to fix it this way.

Discussion: https://postgr.es/m/1661334672.728714027@f473.i.mail.ru

Branch
------
REL_10_STABLE

Details
-------
https://git.postgresql.org/pg/commitdiff/cba393a10af870669a4d69248669179e28e4089b

Modified Files
--------------
src/backend/snowball/dict_snowball.c | 18 +++++++++++++++++-
1 file changed, 17 insertions(+), 1 deletion(-)



^ permalink  raw  reply  [nested|flat] 7+ messages in thread


end of thread, other threads:[~2022-08-31 14:42 UTC | newest]

Thread overview: 7+ messages (download: mbox mbox.gz follow: Atom feed)
-- links below jump to the message on this page --
2022-08-31 14:42 pgsql: In the Snowball dictionary, don't try to stem excessively-long w Tom Lane <tgl@sss.pgh.pa.us>
2022-08-31 14:42 pgsql: In the Snowball dictionary, don't try to stem excessively-long w Tom Lane <tgl@sss.pgh.pa.us>
2022-08-31 14:42 pgsql: In the Snowball dictionary, don't try to stem excessively-long w Tom Lane <tgl@sss.pgh.pa.us>
2022-08-31 14:42 pgsql: In the Snowball dictionary, don't try to stem excessively-long w Tom Lane <tgl@sss.pgh.pa.us>
2022-08-31 14:42 pgsql: In the Snowball dictionary, don't try to stem excessively-long w Tom Lane <tgl@sss.pgh.pa.us>
2022-08-31 14:42 pgsql: In the Snowball dictionary, don't try to stem excessively-long w Tom Lane <tgl@sss.pgh.pa.us>
2022-08-31 14:42 pgsql: In the Snowball dictionary, don't try to stem excessively-long w Tom Lane <tgl@sss.pgh.pa.us>

This inbox is served by DDX for PostgreSQL; see mirroring instructions
for how to clone and mirror all data and code used for this inbox