pg.ddx.io pgsql-committers@postgresql.org mailing list archive
help / color / mirror / Atom feedpgsql: Update to latest Snowball sources.
4+ messages / 2 participants
[nested] [flat]
* pgsql: Update to latest Snowball sources.
@ 2025-02-19 02:14 Tom Lane <tgl@sss.pgh.pa.us>
0 siblings, 0 replies; 4+ messages in thread
From: Tom Lane @ 2025-02-19 02:14 UTC (permalink / raw)
To: pgsql-committers@lists.postgresql.org
Update to latest Snowball sources.
It's been some time since we did this, partly because the upstream
snowball project hasn't formally tagged a new release since 2021.
The main motivation for doing it now is to absorb a bug fix
(their commit e322673a841d9abd69994ae8cd20e191090b6ef4), which
prevents a null pointer dereference crash if SN_create_env() gets
a malloc failure at just the wrong point. We'll patch the back
branches with only that change, but we might as well do the full
sync dance on HEAD.
Aside from a bunch of mostly-minor tweaks to existing stemmers, this
update adds a new stemmer for Estonian. It also removes the existing
stemmer for Romanian using ISO-8859-2 encoding. Upstream apparently
concluded that ISO-8859-2 doesn't provide an adequate representation
of some Romanian characters, and the UTF-8 implementation should be
used instead.
While at it, update the README's instructions for doing a sync,
which have not been adjusted during the addition of meson tooling.
Thanks to Maksim Korotkov for discovering the null-pointer
bug and submitting the fix to upstream snowball.
Reported-by: Maksim Korotkov <m.korotkov@postgrespro.ru>
Discussion: https://postgr.es/m/1d1a46-67ab1000-21-80c451@83151435
Branch
------
master
Details
-------
https://git.postgresql.org/pg/commitdiff/b464e51ab32fbf09cf5d9c911a8e26f491ad1f44
Modified Files
--------------
doc/src/sgml/textsearch.sgml | 1 +
src/backend/snowball/Makefile | 2 +-
src/backend/snowball/README | 22 +-
src/backend/snowball/dict_snowball.c | 4 +-
src/backend/snowball/libstemmer/api.c | 2 +-
.../snowball/libstemmer/stem_ISO_8859_1_basque.c | 38 +-
.../snowball/libstemmer/stem_ISO_8859_1_catalan.c | 29 +-
.../snowball/libstemmer/stem_ISO_8859_1_danish.c | 14 +-
.../snowball/libstemmer/stem_ISO_8859_1_dutch.c | 56 +-
.../snowball/libstemmer/stem_ISO_8859_1_english.c | 154 +-
.../snowball/libstemmer/stem_ISO_8859_1_finnish.c | 34 +-
.../snowball/libstemmer/stem_ISO_8859_1_french.c | 256 +--
.../snowball/libstemmer/stem_ISO_8859_1_german.c | 403 ++--
.../libstemmer/stem_ISO_8859_1_indonesian.c | 36 +-
.../snowball/libstemmer/stem_ISO_8859_1_irish.c | 31 +-
.../snowball/libstemmer/stem_ISO_8859_1_italian.c | 134 +-
.../libstemmer/stem_ISO_8859_1_norwegian.c | 14 +-
.../snowball/libstemmer/stem_ISO_8859_1_porter.c | 65 +-
.../libstemmer/stem_ISO_8859_1_portuguese.c | 48 +-
.../snowball/libstemmer/stem_ISO_8859_1_spanish.c | 51 +-
.../snowball/libstemmer/stem_ISO_8859_1_swedish.c | 57 +-
.../libstemmer/stem_ISO_8859_2_hungarian.c | 31 +-
.../snowball/libstemmer/stem_ISO_8859_2_romanian.c | 965 ---------
.../snowball/libstemmer/stem_KOI8_R_russian.c | 33 +-
.../snowball/libstemmer/stem_UTF_8_arabic.c | 124 +-
.../snowball/libstemmer/stem_UTF_8_armenian.c | 25 +-
.../snowball/libstemmer/stem_UTF_8_basque.c | 38 +-
.../snowball/libstemmer/stem_UTF_8_catalan.c | 29 +-
.../snowball/libstemmer/stem_UTF_8_danish.c | 14 +-
src/backend/snowball/libstemmer/stem_UTF_8_dutch.c | 58 +-
.../snowball/libstemmer/stem_UTF_8_english.c | 158 +-
.../snowball/libstemmer/stem_UTF_8_estonian.c | 1416 ++++++++++++++
.../snowball/libstemmer/stem_UTF_8_finnish.c | 34 +-
.../snowball/libstemmer/stem_UTF_8_french.c | 274 +--
.../snowball/libstemmer/stem_UTF_8_german.c | 405 ++--
src/backend/snowball/libstemmer/stem_UTF_8_greek.c | 376 ++--
src/backend/snowball/libstemmer/stem_UTF_8_hindi.c | 2 +-
.../snowball/libstemmer/stem_UTF_8_hungarian.c | 31 +-
.../snowball/libstemmer/stem_UTF_8_indonesian.c | 36 +-
src/backend/snowball/libstemmer/stem_UTF_8_irish.c | 31 +-
.../snowball/libstemmer/stem_UTF_8_italian.c | 134 +-
.../snowball/libstemmer/stem_UTF_8_lithuanian.c | 27 +-
.../snowball/libstemmer/stem_UTF_8_nepali.c | 8 +-
.../snowball/libstemmer/stem_UTF_8_norwegian.c | 14 +-
.../snowball/libstemmer/stem_UTF_8_porter.c | 65 +-
.../snowball/libstemmer/stem_UTF_8_portuguese.c | 48 +-
.../snowball/libstemmer/stem_UTF_8_romanian.c | 1104 ++++++-----
.../snowball/libstemmer/stem_UTF_8_russian.c | 33 +-
.../snowball/libstemmer/stem_UTF_8_serbian.c | 35 +-
.../snowball/libstemmer/stem_UTF_8_spanish.c | 51 +-
.../snowball/libstemmer/stem_UTF_8_swedish.c | 57 +-
src/backend/snowball/libstemmer/stem_UTF_8_tamil.c | 2064 ++++++++------------
.../snowball/libstemmer/stem_UTF_8_turkish.c | 243 +--
.../snowball/libstemmer/stem_UTF_8_yiddish.c | 91 +-
src/backend/snowball/libstemmer/utilities.c | 6 +-
src/backend/snowball/meson.build | 2 +-
src/backend/snowball/snowball_create.pl | 1 +
src/bin/initdb/initdb.c | 2 +
src/include/snowball/libstemmer/header.h | 2 +-
.../snowball/libstemmer/stem_ISO_8859_2_romanian.h | 15 -
.../snowball/libstemmer/stem_UTF_8_estonian.h | 15 +
61 files changed, 4970 insertions(+), 4578 deletions(-)
^ permalink raw reply [nested|flat] 4+ messages in thread
* pgsql: Update to latest Snowball sources.
@ 2026-01-05 20:23 Tom Lane <tgl@sss.pgh.pa.us>
2026-01-05 22:24 ` Re: pgsql: Update to latest Snowball sources. Jelte Fennema-Nio <postgres@jeltef.nl>
0 siblings, 1 reply; 4+ messages in thread
From: Tom Lane @ 2026-01-05 20:23 UTC (permalink / raw)
To: pgsql-committers@lists.postgresql.org
Update to latest Snowball sources.
It's been almost a year since we last did this, and upstream has
been busy. They've added stemmers for Polish and Esperanto,
and also deprecated their old Dutch stemmer in favor of the
Kraaij-Pohlmann algorithm. (The "dutch" stemmer is now the
latter, and "dutch_porter" is the old algorithm.)
Upstream also decided to rename their internal header "header.h"
to something less generic: "snowball_runtime.h". Seems like a good
thing, but it complicates this patch a bit because we were relying on
interposing our own version of "header.h" to control system header
inclusion order. (We're partially failing at that now, because now the
generated stemmer files include <stddef.h> before snowball_runtime.h.
I think that'll be okay, but if the buildfarm complains then we'll
have to do more-extensive editing of the generated files.)
I realized that we weren't documenting the available stemmers in
any user-visible place, except indirectly through sample \dFd output.
That's incomplete because we only provide built-in dictionaries for
the recommended stemmers for each language, not alternative stemmers
such as dutch_porter. So I added a list to the documentation.
I did not do anything with the stopword lists. If those are still
available from snowballstem.org, they are mighty well hidden.
Discussion: https://postgr.es/m/1185975.1767569534@sss.pgh.pa.us
Branch
------
master
Details
-------
https://git.postgresql.org/pg/commitdiff/7dc95cc3b94f2f558606e5ec307466a4e3dbc832
Modified Files
--------------
doc/src/sgml/textsearch.sgml | 52 +
src/backend/snowball/Makefile | 5 +
src/backend/snowball/README | 22 +-
src/backend/snowball/dict_snowball.c | 14 +-
src/backend/snowball/libstemmer/api.c | 48 +-
.../snowball/libstemmer/stem_ISO_8859_1_basque.c | 1132 +--
.../snowball/libstemmer/stem_ISO_8859_1_catalan.c | 1364 +--
.../snowball/libstemmer/stem_ISO_8859_1_danish.c | 324 +-
.../snowball/libstemmer/stem_ISO_8859_1_dutch.c | 2338 ++++-
.../libstemmer/stem_ISO_8859_1_dutch_porter.c | 665 ++
.../snowball/libstemmer/stem_ISO_8859_1_english.c | 1310 +--
.../snowball/libstemmer/stem_ISO_8859_1_finnish.c | 686 +-
.../snowball/libstemmer/stem_ISO_8859_1_french.c | 1493 +--
.../snowball/libstemmer/stem_ISO_8859_1_german.c | 547 +-
.../libstemmer/stem_ISO_8859_1_indonesian.c | 543 +-
.../snowball/libstemmer/stem_ISO_8859_1_irish.c | 388 +-
.../snowball/libstemmer/stem_ISO_8859_1_italian.c | 995 +-
.../libstemmer/stem_ISO_8859_1_norwegian.c | 420 +-
.../snowball/libstemmer/stem_ISO_8859_1_porter.c | 611 +-
.../libstemmer/stem_ISO_8859_1_portuguese.c | 901 +-
.../snowball/libstemmer/stem_ISO_8859_1_spanish.c | 1017 +-
.../snowball/libstemmer/stem_ISO_8859_1_swedish.c | 477 +-
.../libstemmer/stem_ISO_8859_2_hungarian.c | 1101 +-
.../snowball/libstemmer/stem_ISO_8859_2_polish.c | 520 +
.../snowball/libstemmer/stem_KOI8_R_russian.c | 602 +-
.../snowball/libstemmer/stem_UTF_8_arabic.c | 1554 +--
.../snowball/libstemmer/stem_UTF_8_armenian.c | 528 +-
.../snowball/libstemmer/stem_UTF_8_basque.c | 1135 +--
.../snowball/libstemmer/stem_UTF_8_catalan.c | 1367 +--
.../snowball/libstemmer/stem_UTF_8_danish.c | 326 +-
src/backend/snowball/libstemmer/stem_UTF_8_dutch.c | 2400 ++++-
.../snowball/libstemmer/stem_UTF_8_dutch_porter.c | 680 ++
.../snowball/libstemmer/stem_UTF_8_english.c | 1324 +--
.../snowball/libstemmer/stem_UTF_8_esperanto.c | 820 ++
.../snowball/libstemmer/stem_UTF_8_estonian.c | 2010 ++--
.../snowball/libstemmer/stem_UTF_8_finnish.c | 696 +-
.../snowball/libstemmer/stem_UTF_8_french.c | 1523 +--
.../snowball/libstemmer/stem_UTF_8_german.c | 554 +-
src/backend/snowball/libstemmer/stem_UTF_8_greek.c | 4218 ++++----
src/backend/snowball/libstemmer/stem_UTF_8_hindi.c | 308 +-
.../snowball/libstemmer/stem_UTF_8_hungarian.c | 1100 +-
.../snowball/libstemmer/stem_UTF_8_indonesian.c | 543 +-
src/backend/snowball/libstemmer/stem_UTF_8_irish.c | 388 +-
.../snowball/libstemmer/stem_UTF_8_italian.c | 1007 +-
.../snowball/libstemmer/stem_UTF_8_lithuanian.c | 1179 ++-
.../snowball/libstemmer/stem_UTF_8_nepali.c | 598 +-
.../snowball/libstemmer/stem_UTF_8_norwegian.c | 422 +-
.../snowball/libstemmer/stem_UTF_8_polish.c | 523 +
.../snowball/libstemmer/stem_UTF_8_porter.c | 620 +-
.../snowball/libstemmer/stem_UTF_8_portuguese.c | 910 +-
.../snowball/libstemmer/stem_UTF_8_romanian.c | 961 +-
.../snowball/libstemmer/stem_UTF_8_russian.c | 625 +-
.../snowball/libstemmer/stem_UTF_8_serbian.c | 10148 ++++++++++---------
.../snowball/libstemmer/stem_UTF_8_spanish.c | 1023 +-
.../snowball/libstemmer/stem_UTF_8_swedish.c | 479 +-
src/backend/snowball/libstemmer/stem_UTF_8_tamil.c | 1361 +--
.../snowball/libstemmer/stem_UTF_8_turkish.c | 2371 +++--
.../snowball/libstemmer/stem_UTF_8_yiddish.c | 1235 +--
src/backend/snowball/libstemmer/utilities.c | 205 +-
src/backend/snowball/meson.build | 7 +-
src/backend/snowball/snowball_create.pl | 2 +
src/bin/initdb/initdb.c | 2 +
src/include/snowball/libstemmer/api.h | 18 +-
src/include/snowball/libstemmer/header.h | 61 -
src/include/snowball/libstemmer/snowball_runtime.h | 109 +
.../snowball/libstemmer/stem_ISO_8859_1_basque.h | 3 +-
.../snowball/libstemmer/stem_ISO_8859_1_catalan.h | 3 +-
.../snowball/libstemmer/stem_ISO_8859_1_danish.h | 3 +-
.../snowball/libstemmer/stem_ISO_8859_1_dutch.h | 3 +-
.../libstemmer/stem_ISO_8859_1_dutch_porter.h | 14 +
.../snowball/libstemmer/stem_ISO_8859_1_english.h | 3 +-
.../snowball/libstemmer/stem_ISO_8859_1_finnish.h | 3 +-
.../snowball/libstemmer/stem_ISO_8859_1_french.h | 3 +-
.../snowball/libstemmer/stem_ISO_8859_1_german.h | 3 +-
.../libstemmer/stem_ISO_8859_1_indonesian.h | 3 +-
.../snowball/libstemmer/stem_ISO_8859_1_irish.h | 3 +-
.../snowball/libstemmer/stem_ISO_8859_1_italian.h | 3 +-
.../libstemmer/stem_ISO_8859_1_norwegian.h | 3 +-
.../snowball/libstemmer/stem_ISO_8859_1_porter.h | 3 +-
.../libstemmer/stem_ISO_8859_1_portuguese.h | 3 +-
.../snowball/libstemmer/stem_ISO_8859_1_spanish.h | 3 +-
.../snowball/libstemmer/stem_ISO_8859_1_swedish.h | 3 +-
.../libstemmer/stem_ISO_8859_2_hungarian.h | 3 +-
.../snowball/libstemmer/stem_ISO_8859_2_polish.h | 14 +
.../snowball/libstemmer/stem_KOI8_R_russian.h | 3 +-
.../snowball/libstemmer/stem_UTF_8_arabic.h | 3 +-
.../snowball/libstemmer/stem_UTF_8_armenian.h | 3 +-
.../snowball/libstemmer/stem_UTF_8_basque.h | 3 +-
.../snowball/libstemmer/stem_UTF_8_catalan.h | 3 +-
.../snowball/libstemmer/stem_UTF_8_danish.h | 3 +-
src/include/snowball/libstemmer/stem_UTF_8_dutch.h | 3 +-
.../snowball/libstemmer/stem_UTF_8_dutch_porter.h | 14 +
.../snowball/libstemmer/stem_UTF_8_english.h | 3 +-
.../snowball/libstemmer/stem_UTF_8_esperanto.h | 14 +
.../snowball/libstemmer/stem_UTF_8_estonian.h | 3 +-
.../snowball/libstemmer/stem_UTF_8_finnish.h | 3 +-
.../snowball/libstemmer/stem_UTF_8_french.h | 3 +-
.../snowball/libstemmer/stem_UTF_8_german.h | 3 +-
src/include/snowball/libstemmer/stem_UTF_8_greek.h | 3 +-
src/include/snowball/libstemmer/stem_UTF_8_hindi.h | 3 +-
.../snowball/libstemmer/stem_UTF_8_hungarian.h | 3 +-
.../snowball/libstemmer/stem_UTF_8_indonesian.h | 3 +-
src/include/snowball/libstemmer/stem_UTF_8_irish.h | 3 +-
.../snowball/libstemmer/stem_UTF_8_italian.h | 3 +-
.../snowball/libstemmer/stem_UTF_8_lithuanian.h | 3 +-
.../snowball/libstemmer/stem_UTF_8_nepali.h | 3 +-
.../snowball/libstemmer/stem_UTF_8_norwegian.h | 3 +-
.../snowball/libstemmer/stem_UTF_8_polish.h | 14 +
.../snowball/libstemmer/stem_UTF_8_porter.h | 3 +-
.../snowball/libstemmer/stem_UTF_8_portuguese.h | 3 +-
.../snowball/libstemmer/stem_UTF_8_romanian.h | 3 +-
.../snowball/libstemmer/stem_UTF_8_russian.h | 3 +-
.../snowball/libstemmer/stem_UTF_8_serbian.h | 3 +-
.../snowball/libstemmer/stem_UTF_8_spanish.h | 3 +-
.../snowball/libstemmer/stem_UTF_8_swedish.h | 3 +-
src/include/snowball/libstemmer/stem_UTF_8_tamil.h | 3 +-
.../snowball/libstemmer/stem_UTF_8_turkish.h | 3 +-
.../snowball/libstemmer/stem_UTF_8_yiddish.h | 3 +-
.../snowball/{header.h => snowball_runtime.h} | 22 +-
119 files changed, 36038 insertions(+), 27113 deletions(-)
^ permalink raw reply [nested|flat] 4+ messages in thread
* Re: pgsql: Update to latest Snowball sources.
2026-01-05 20:23 pgsql: Update to latest Snowball sources. Tom Lane <tgl@sss.pgh.pa.us>
@ 2026-01-05 22:24 ` Jelte Fennema-Nio <postgres@jeltef.nl>
2026-01-05 22:27 ` Re: pgsql: Update to latest Snowball sources. Tom Lane <tgl@sss.pgh.pa.us>
0 siblings, 1 reply; 4+ messages in thread
From: Jelte Fennema-Nio @ 2026-01-05 22:24 UTC (permalink / raw)
To: Tom Lane <tgl@sss.pgh.pa.us>; +Cc: pgsql-committers@lists.postgresql.org
On Mon, 5 Jan 2026 at 21:23, Tom Lane <tgl@sss.pgh.pa.us> wrote:
>
> Update to latest Snowball sources.
On my own CI I got some failures seemingly related to this:
https://cirrus-ci.com/task/5989391459418112
Might be build cache issues or something (that I don't have time to
look into right now), but wanted to call it out in case others run
into the same.
^ permalink raw reply [nested|flat] 4+ messages in thread
* Re: pgsql: Update to latest Snowball sources.
2026-01-05 20:23 pgsql: Update to latest Snowball sources. Tom Lane <tgl@sss.pgh.pa.us>
2026-01-05 22:24 ` Re: pgsql: Update to latest Snowball sources. Jelte Fennema-Nio <postgres@jeltef.nl>
@ 2026-01-05 22:27 ` Tom Lane <tgl@sss.pgh.pa.us>
0 siblings, 0 replies; 4+ messages in thread
From: Tom Lane @ 2026-01-05 22:27 UTC (permalink / raw)
To: Jelte Fennema-Nio <postgres@jeltef.nl>; +Cc: pgsql-committers@lists.postgresql.org
Jelte Fennema-Nio <postgres@jeltef.nl> writes:
> On my own CI I got some failures seemingly related to this:
> https://cirrus-ci.com/task/5989391459418112
If it's in a meson build, yeah ... fixed ...
regards, tom lane
^ permalink raw reply [nested|flat] 4+ messages in thread
end of thread, other threads:[~2026-01-05 22:27 UTC | newest]
Thread overview: 4+ messages (download: mbox mbox.gz follow: Atom feed)
-- links below jump to the message on this page --
2025-02-19 02:14 pgsql: Update to latest Snowball sources. Tom Lane <tgl@sss.pgh.pa.us>
2026-01-05 20:23 pgsql: Update to latest Snowball sources. Tom Lane <tgl@sss.pgh.pa.us>
2026-01-05 22:24 ` Jelte Fennema-Nio <postgres@jeltef.nl>
2026-01-05 22:27 ` Tom Lane <tgl@sss.pgh.pa.us>
This inbox is served by DDX for PostgreSQL; see mirroring instructions
for how to clone and mirror all data and code used for this inbox