From: Rick Jansen <rick@rockingstone.nl>
To: Oleg Bartunov <oleg@sai.msu.su>
Cc: Mike Rylander <mrylander@gmail.com>
Cc: pgsql-performance@postgresql.org
Subject: Re: Tsearch2 performance on big database
Date: Wed, 23 Mar 2005 09:52:27 +0100
Message-ID: <42412E4B.4010407@rockingstone.nl> (raw)
In-Reply-To: <Pine.GSO.4.62.0503221907320.5508@ra.sai.msu.su>
References: <42400F57.3040604@rockingstone.nl>
<Pine.GSO.4.62.0503221533400.5508@ra.sai.msu.su>
<b918cf3d050322044847d3f9e1@mail.gmail.com>
<42401CBF.50403@rockingstone.nl>
<b918cf3d0503220730665fe92c@mail.gmail.com>
<Pine.GSO.4.62.0503221840480.5508@ra.sai.msu.su>
<42404263.40604@rockingstone.nl>
<Pine.GSO.4.62.0503221907320.5508@ra.sai.msu.su>
Oleg Bartunov wrote:
> On Tue, 22 Mar 2005, Rick Jansen wrote:
>
> Hmm, default configuration is too eager, you index every lexem using
> simple dictionary) ! Probably, it's too much. Here is what I have for my
> russian configuration in dictionary database:
>
> default_russian | lword | {en_ispell,en_stem}
> default_russian | lpart_hword | {en_ispell,en_stem}
> default_russian | lhword | {en_ispell,en_stem}
> default_russian | nlword | {ru_ispell,ru_stem}
> default_russian | nlpart_hword | {ru_ispell,ru_stem}
> default_russian | nlhword | {ru_ispell,ru_stem}
>
> Notice, I index only russian and english words, no numbers, url, etc.
> You may just delete unwanted rows in pg_ts_cfgmap for your configuration,
> but I'd recommend just update them setting dict_name to NULL.
> For example, to not indexing integers:
>
> update pg_ts_cfgmap set dict_name=NULL where ts_name='default_russian'
> and tok_alias='int';
>
> voc=# select token,dict_name,tok_type,tsvector from ts_debug('Do you
> have +70000 bucks');
> token | dict_name | tok_type | tsvector
> --------+---------------------+----------+----------
> Do | {en_ispell,en_stem} | lword |
> you | {en_ispell,en_stem} | lword |
> have | {en_ispell,en_stem} | lword |
> +70000 | | int |
> bucks | {en_ispell,en_stem} | lword | 'buck'
>
> Only 'bucks' gets indexed :)
> Hmm, probably I should add this into documentation.
>
> What about word statistics (# of unique words, for example).
>
I'm now following the guide to add the ispell dictionary and I've
updated most of the rows setting dict_name to NULL:
ts_name | tok_alias | dict_name
-----------------+--------------+-----------
default | lword | {en_stem}
default | nlword | {simple}
default | word | {simple}
default | part_hword | {simple}
default | nlpart_hword | {simple}
default | lpart_hword | {en_stem}
default | hword | {simple}
default | lhword | {en_stem}
default | nlhword | {simple}
These are left, but I have no idea what a 'hword' or 'nlhword' or any
other of these tokens are.
Anyway, how do I find out the number of unique words or other word
statistics?
Rick
--
Systems Administrator for Rockingstone IT
http://www.rockingstone.comhttp://www.megabooksearch.com - Search many book listing sites at once
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Reply to all the recipients using the --to and --cc options:
reply via email
To: pgsql-performance@postgresql.org
Cc: rick@rockingstone.nl, oleg@sai.msu.su, mrylander@gmail.com
Subject: Re: Tsearch2 performance on big database
In-Reply-To: <42412E4B.4010407@rockingstone.nl>
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
This inbox is served by DDX for PostgreSQL; see mirroring instructions
for how to clone and mirror all data and code used for this inbox