pg.ddx.io pgsql-committers@postgresql.org mailing list archivehelp / color / mirror / Atom feed
pgsql: In the Snowball dictionary, don't try to stem excessively-long w 7+ messages / 1 participants [nested] [flat]
* pgsql: In the Snowball dictionary, don't try to stem excessively-long w @ 2022-08-31 14:42 Tom Lane <tgl@sss.pgh.pa.us> 0 siblings, 0 replies; 7+ messages in thread From: Tom Lane @ 2022-08-31 14:42 UTC (permalink / raw) To: pgsql-committers@lists.postgresql.org In the Snowball dictionary, don't try to stem excessively-long words. If the input word exceeds 1000 bytes, don't pass it to the stemmer; just return it as-is after case folding. Such an input is surely not a word in any human language, so whatever the stemmer might do to it would be pretty dubious in the first place. Adding this restriction protects us against a known recursion-to-stack-overflow problem in the Turkish stemmer, and it seems like good insurance against any other safety or performance issues that may exist in the Snowball stemmers. (I note, for example, that they contain no CHECK_FOR_INTERRUPTS calls, so we really don't want them running for a long time.) The threshold of 1000 bytes is arbitrary. An alternative definition could have been to treat such words as stopwords, but that seems like a bigger break from the old behavior. Per report from Egor Chindyaskin and Alexander Lakhin. Thanks to Olly Betts for the recommendation to fix it this way. Discussion: https://postgr.es/m/1661334672.728714027@f473.i.mail.ru Branch ------ master Details ------- https://git.postgresql.org/pg/commitdiff/1058555a5eccffd17c47edf3e15dd7e26934b492 Modified Files -------------- src/backend/snowball/dict_snowball.c | 18 +++++++++++++++++- 1 file changed, 17 insertions(+), 1 deletion(-) ^ permalink raw reply [nested|flat] 7+ messages in thread
* pgsql: In the Snowball dictionary, don't try to stem excessively-long w @ 2022-08-31 14:42 Tom Lane <tgl@sss.pgh.pa.us> 0 siblings, 0 replies; 7+ messages in thread From: Tom Lane @ 2022-08-31 14:42 UTC (permalink / raw) To: pgsql-committers@lists.postgresql.org In the Snowball dictionary, don't try to stem excessively-long words. If the input word exceeds 1000 bytes, don't pass it to the stemmer; just return it as-is after case folding. Such an input is surely not a word in any human language, so whatever the stemmer might do to it would be pretty dubious in the first place. Adding this restriction protects us against a known recursion-to-stack-overflow problem in the Turkish stemmer, and it seems like good insurance against any other safety or performance issues that may exist in the Snowball stemmers. (I note, for example, that they contain no CHECK_FOR_INTERRUPTS calls, so we really don't want them running for a long time.) The threshold of 1000 bytes is arbitrary. An alternative definition could have been to treat such words as stopwords, but that seems like a bigger break from the old behavior. Per report from Egor Chindyaskin and Alexander Lakhin. Thanks to Olly Betts for the recommendation to fix it this way. Discussion: https://postgr.es/m/1661334672.728714027@f473.i.mail.ru Branch ------ REL_15_STABLE Details ------- https://git.postgresql.org/pg/commitdiff/61f6e5c2360382be99846be904af33e3406c938a Modified Files -------------- src/backend/snowball/dict_snowball.c | 18 +++++++++++++++++- 1 file changed, 17 insertions(+), 1 deletion(-) ^ permalink raw reply [nested|flat] 7+ messages in thread
* pgsql: In the Snowball dictionary, don't try to stem excessively-long w @ 2022-08-31 14:42 Tom Lane <tgl@sss.pgh.pa.us> 0 siblings, 0 replies; 7+ messages in thread From: Tom Lane @ 2022-08-31 14:42 UTC (permalink / raw) To: pgsql-committers@lists.postgresql.org In the Snowball dictionary, don't try to stem excessively-long words. If the input word exceeds 1000 bytes, don't pass it to the stemmer; just return it as-is after case folding. Such an input is surely not a word in any human language, so whatever the stemmer might do to it would be pretty dubious in the first place. Adding this restriction protects us against a known recursion-to-stack-overflow problem in the Turkish stemmer, and it seems like good insurance against any other safety or performance issues that may exist in the Snowball stemmers. (I note, for example, that they contain no CHECK_FOR_INTERRUPTS calls, so we really don't want them running for a long time.) The threshold of 1000 bytes is arbitrary. An alternative definition could have been to treat such words as stopwords, but that seems like a bigger break from the old behavior. Per report from Egor Chindyaskin and Alexander Lakhin. Thanks to Olly Betts for the recommendation to fix it this way. Discussion: https://postgr.es/m/1661334672.728714027@f473.i.mail.ru Branch ------ REL_14_STABLE Details ------- https://git.postgresql.org/pg/commitdiff/e969f1ae2b01d7b69371332b839fa16e3b54e56d Modified Files -------------- src/backend/snowball/dict_snowball.c | 18 +++++++++++++++++- 1 file changed, 17 insertions(+), 1 deletion(-) ^ permalink raw reply [nested|flat] 7+ messages in thread
* pgsql: In the Snowball dictionary, don't try to stem excessively-long w @ 2022-08-31 14:42 Tom Lane <tgl@sss.pgh.pa.us> 0 siblings, 0 replies; 7+ messages in thread From: Tom Lane @ 2022-08-31 14:42 UTC (permalink / raw) To: pgsql-committers@lists.postgresql.org In the Snowball dictionary, don't try to stem excessively-long words. If the input word exceeds 1000 bytes, don't pass it to the stemmer; just return it as-is after case folding. Such an input is surely not a word in any human language, so whatever the stemmer might do to it would be pretty dubious in the first place. Adding this restriction protects us against a known recursion-to-stack-overflow problem in the Turkish stemmer, and it seems like good insurance against any other safety or performance issues that may exist in the Snowball stemmers. (I note, for example, that they contain no CHECK_FOR_INTERRUPTS calls, so we really don't want them running for a long time.) The threshold of 1000 bytes is arbitrary. An alternative definition could have been to treat such words as stopwords, but that seems like a bigger break from the old behavior. Per report from Egor Chindyaskin and Alexander Lakhin. Thanks to Olly Betts for the recommendation to fix it this way. Discussion: https://postgr.es/m/1661334672.728714027@f473.i.mail.ru Branch ------ REL_13_STABLE Details ------- https://git.postgresql.org/pg/commitdiff/f204ad3a2b19a33a72f0401b8640f68527539bd1 Modified Files -------------- src/backend/snowball/dict_snowball.c | 18 +++++++++++++++++- 1 file changed, 17 insertions(+), 1 deletion(-) ^ permalink raw reply [nested|flat] 7+ messages in thread
* pgsql: In the Snowball dictionary, don't try to stem excessively-long w @ 2022-08-31 14:42 Tom Lane <tgl@sss.pgh.pa.us> 0 siblings, 0 replies; 7+ messages in thread From: Tom Lane @ 2022-08-31 14:42 UTC (permalink / raw) To: pgsql-committers@lists.postgresql.org In the Snowball dictionary, don't try to stem excessively-long words. If the input word exceeds 1000 bytes, don't pass it to the stemmer; just return it as-is after case folding. Such an input is surely not a word in any human language, so whatever the stemmer might do to it would be pretty dubious in the first place. Adding this restriction protects us against a known recursion-to-stack-overflow problem in the Turkish stemmer, and it seems like good insurance against any other safety or performance issues that may exist in the Snowball stemmers. (I note, for example, that they contain no CHECK_FOR_INTERRUPTS calls, so we really don't want them running for a long time.) The threshold of 1000 bytes is arbitrary. An alternative definition could have been to treat such words as stopwords, but that seems like a bigger break from the old behavior. Per report from Egor Chindyaskin and Alexander Lakhin. Thanks to Olly Betts for the recommendation to fix it this way. Discussion: https://postgr.es/m/1661334672.728714027@f473.i.mail.ru Branch ------ REL_12_STABLE Details ------- https://git.postgresql.org/pg/commitdiff/a53e0ea782c52da0fd8f9605a63aba276fc17c13 Modified Files -------------- src/backend/snowball/dict_snowball.c | 18 +++++++++++++++++- 1 file changed, 17 insertions(+), 1 deletion(-) ^ permalink raw reply [nested|flat] 7+ messages in thread
* pgsql: In the Snowball dictionary, don't try to stem excessively-long w @ 2022-08-31 14:42 Tom Lane <tgl@sss.pgh.pa.us> 0 siblings, 0 replies; 7+ messages in thread From: Tom Lane @ 2022-08-31 14:42 UTC (permalink / raw) To: pgsql-committers@lists.postgresql.org In the Snowball dictionary, don't try to stem excessively-long words. If the input word exceeds 1000 bytes, don't pass it to the stemmer; just return it as-is after case folding. Such an input is surely not a word in any human language, so whatever the stemmer might do to it would be pretty dubious in the first place. Adding this restriction protects us against a known recursion-to-stack-overflow problem in the Turkish stemmer, and it seems like good insurance against any other safety or performance issues that may exist in the Snowball stemmers. (I note, for example, that they contain no CHECK_FOR_INTERRUPTS calls, so we really don't want them running for a long time.) The threshold of 1000 bytes is arbitrary. An alternative definition could have been to treat such words as stopwords, but that seems like a bigger break from the old behavior. Per report from Egor Chindyaskin and Alexander Lakhin. Thanks to Olly Betts for the recommendation to fix it this way. Discussion: https://postgr.es/m/1661334672.728714027@f473.i.mail.ru Branch ------ REL_11_STABLE Details ------- https://git.postgresql.org/pg/commitdiff/f5aa855cd8a63cd695c5f9ff097c7bc445872cdf Modified Files -------------- src/backend/snowball/dict_snowball.c | 18 +++++++++++++++++- 1 file changed, 17 insertions(+), 1 deletion(-) ^ permalink raw reply [nested|flat] 7+ messages in thread
* pgsql: In the Snowball dictionary, don't try to stem excessively-long w @ 2022-08-31 14:42 Tom Lane <tgl@sss.pgh.pa.us> 0 siblings, 0 replies; 7+ messages in thread From: Tom Lane @ 2022-08-31 14:42 UTC (permalink / raw) To: pgsql-committers@lists.postgresql.org In the Snowball dictionary, don't try to stem excessively-long words. If the input word exceeds 1000 bytes, don't pass it to the stemmer; just return it as-is after case folding. Such an input is surely not a word in any human language, so whatever the stemmer might do to it would be pretty dubious in the first place. Adding this restriction protects us against a known recursion-to-stack-overflow problem in the Turkish stemmer, and it seems like good insurance against any other safety or performance issues that may exist in the Snowball stemmers. (I note, for example, that they contain no CHECK_FOR_INTERRUPTS calls, so we really don't want them running for a long time.) The threshold of 1000 bytes is arbitrary. An alternative definition could have been to treat such words as stopwords, but that seems like a bigger break from the old behavior. Per report from Egor Chindyaskin and Alexander Lakhin. Thanks to Olly Betts for the recommendation to fix it this way. Discussion: https://postgr.es/m/1661334672.728714027@f473.i.mail.ru Branch ------ REL_10_STABLE Details ------- https://git.postgresql.org/pg/commitdiff/cba393a10af870669a4d69248669179e28e4089b Modified Files -------------- src/backend/snowball/dict_snowball.c | 18 +++++++++++++++++- 1 file changed, 17 insertions(+), 1 deletion(-) ^ permalink raw reply [nested|flat] 7+ messages in thread
end of thread, other threads:[~2022-08-31 14:42 UTC | newest] Thread overview: 7+ messages (download: mbox mbox.gz follow: Atom feed) -- links below jump to the message on this page -- 2022-08-31 14:42 pgsql: In the Snowball dictionary, don't try to stem excessively-long w Tom Lane <tgl@sss.pgh.pa.us> 2022-08-31 14:42 pgsql: In the Snowball dictionary, don't try to stem excessively-long w Tom Lane <tgl@sss.pgh.pa.us> 2022-08-31 14:42 pgsql: In the Snowball dictionary, don't try to stem excessively-long w Tom Lane <tgl@sss.pgh.pa.us> 2022-08-31 14:42 pgsql: In the Snowball dictionary, don't try to stem excessively-long w Tom Lane <tgl@sss.pgh.pa.us> 2022-08-31 14:42 pgsql: In the Snowball dictionary, don't try to stem excessively-long w Tom Lane <tgl@sss.pgh.pa.us> 2022-08-31 14:42 pgsql: In the Snowball dictionary, don't try to stem excessively-long w Tom Lane <tgl@sss.pgh.pa.us> 2022-08-31 14:42 pgsql: In the Snowball dictionary, don't try to stem excessively-long w Tom Lane <tgl@sss.pgh.pa.us>
This inbox is served by DDX for PostgreSQL; see mirroring instructions for how to clone and mirror all data and code used for this inbox