From: Oleg Bartunov <oleg@sai.msu.su>
To: Rick Jansen <rick@rockingstone.nl>
Cc: Mike Rylander <mrylander@gmail.com>
Cc: pgsql-performance@postgresql.org
Subject: Re: Tsearch2 performance on big database
Date: Tue, 22 Mar 2005 19:38:05 +0300 (MSK)
Message-ID: <Pine.GSO.4.62.0503221907320.5508@ra.sai.msu.su> (raw)
In-Reply-To: <42404263.40604@rockingstone.nl>
References: <42400F57.3040604@rockingstone.nl>
<Pine.GSO.4.62.0503221533400.5508@ra.sai.msu.su>
<b918cf3d050322044847d3f9e1@mail.gmail.com>
<42401CBF.50403@rockingstone.nl>
<b918cf3d0503220730665fe92c@mail.gmail.com>
<Pine.GSO.4.62.0503221840480.5508@ra.sai.msu.su>
<42404263.40604@rockingstone.nl>
On Tue, 22 Mar 2005, Rick Jansen wrote:
> Oleg Bartunov wrote:
>> Mike,
>>
>> no comments before Rick post tsearch configs and increased buffers !
>> Union shouldn't be faster than (term1|term2).
>> tsearch2 internals description might help you understanding tsearch2
>> limitations.
>> See http://www.sai.msu.su/~megera/oddmuse/index.cgi/Tsearch_V2_internals
>> Also, don't miss my notes:
>> http://www.sai.msu.su/~megera/oddmuse/index.cgi/Tsearch_V2_Notes
>>
>> Oleg
>
> Thanks Oleg, i've seen those pages before :) I've set shared_buffers to 45000
> now (yes thats probably very much, isn't it?) and it already seems a lot
> quicker.
>
> How do I find out what my tsearch config is? I followed the intro
> (http://www.sai.msu.su/~megera/oddmuse/index.cgi/tsearch-v2-intro) and
> applied it to our books table, thats all, didnt change anything else about
> configs.
Hmm, default configuration is too eager, you index every lexem using
simple dictionary) ! Probably, it's too much. Here is what I have for my
russian configuration in dictionary database:
default_russian | lword | {en_ispell,en_stem}
default_russian | lpart_hword | {en_ispell,en_stem}
default_russian | lhword | {en_ispell,en_stem}
default_russian | nlword | {ru_ispell,ru_stem}
default_russian | nlpart_hword | {ru_ispell,ru_stem}
default_russian | nlhword | {ru_ispell,ru_stem}
Notice, I index only russian and english words, no numbers, url, etc.
You may just delete unwanted rows in pg_ts_cfgmap for your configuration,
but I'd recommend just update them setting dict_name to NULL.
For example, to not indexing integers:
update pg_ts_cfgmap set dict_name=NULL where ts_name='default_russian'
and tok_alias='int';
voc=# select token,dict_name,tok_type,tsvector from ts_debug('Do you have +70000 bucks');
token | dict_name | tok_type | tsvector
--------+---------------------+----------+----------
Do | {en_ispell,en_stem} | lword |
you | {en_ispell,en_stem} | lword |
have | {en_ispell,en_stem} | lword |
+70000 | | int |
bucks | {en_ispell,en_stem} | lword | 'buck'
Only 'bucks' gets indexed :)
Hmm, probably I should add this into documentation.
What about word statistics (# of unique words, for example).
Regards,
Oleg
_____________________________________________________________
Oleg Bartunov, sci.researcher, hostmaster of AstroNet,
Sternberg Astronomical Institute, Moscow University (Russia)
Internet: oleg@sai.msu.su, http://www.sai.msu.su/~megera/
phone: +007(095)939-16-83, +007(095)939-23-83
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Reply to all the recipients using the --to and --cc options:
reply via email
To: pgsql-performance@postgresql.org
Cc: oleg@sai.msu.su, rick@rockingstone.nl, mrylander@gmail.com
Subject: Re: Tsearch2 performance on big database
In-Reply-To: <Pine.GSO.4.62.0503221907320.5508@ra.sai.msu.su>
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
This inbox is served by DDX for PostgreSQL; see mirroring instructions
for how to clone and mirror all data and code used for this inbox