pg.ddx.io  pgsql-performance@postgresql.org mailing list archive  
help / color / mirror / Atom feed
From: Oleg Bartunov <oleg@sai.msu.su>
To: Rick Jansen <rick@rockingstone.nl>
Cc: Mike Rylander <mrylander@gmail.com>
Cc: pgsql-performance@postgresql.org
Subject: Re: Tsearch2 performance on big database
Date: Wed, 23 Mar 2005 12:40:03 +0300 (MSK)
Message-ID: <Pine.GSO.4.62.0503231236390.5508@ra.sai.msu.su> (raw)
In-Reply-To: <42412E4B.4010407@rockingstone.nl>
References: <42400F57.3040604@rockingstone.nl>
	<Pine.GSO.4.62.0503221533400.5508@ra.sai.msu.su>
	<b918cf3d050322044847d3f9e1@mail.gmail.com>
	<42401CBF.50403@rockingstone.nl>
	<b918cf3d0503220730665fe92c@mail.gmail.com>
	<Pine.GSO.4.62.0503221840480.5508@ra.sai.msu.su>
	<42404263.40604@rockingstone.nl>
	<Pine.GSO.4.62.0503221907320.5508@ra.sai.msu.su>
	<42412E4B.4010407@rockingstone.nl>

On Wed, 23 Mar 2005, Rick Jansen wrote:

> Oleg Bartunov wrote:
>> On Tue, 22 Mar 2005, Rick Jansen wrote:
>> 
>> Hmm, default configuration is too eager, you index every lexem using simple 
>> dictionary) ! Probably, it's too much. Here is what I have for my russian 
>> configuration in dictionary database:
>> 
>>  default_russian | lword        | {en_ispell,en_stem}
>>  default_russian | lpart_hword  | {en_ispell,en_stem}
>>  default_russian | lhword       | {en_ispell,en_stem}
>>  default_russian | nlword       | {ru_ispell,ru_stem}
>>  default_russian | nlpart_hword | {ru_ispell,ru_stem}
>>  default_russian | nlhword      | {ru_ispell,ru_stem}
>> 
>> Notice, I index only russian and english words, no numbers, url, etc.
>> You may just delete unwanted rows in pg_ts_cfgmap for your configuration,
>> but I'd recommend just update them setting dict_name to NULL.
>> For example, to not indexing integers:
>> 
>> update pg_ts_cfgmap set dict_name=NULL where ts_name='default_russian' and 
>> tok_alias='int';
>> 
>> voc=# select token,dict_name,tok_type,tsvector from ts_debug('Do you have 
>> +70000 bucks');
>>  token  |      dict_name      | tok_type | tsvector 
>> --------+---------------------+----------+----------
>>  Do     | {en_ispell,en_stem} | lword    |
>>  you    | {en_ispell,en_stem} | lword    |
>>  have   | {en_ispell,en_stem} | lword    |
>>  +70000 |                     | int      |
>>  bucks  | {en_ispell,en_stem} | lword    | 'buck'
>> 
>> Only 'bucks' gets indexed :)
>> Hmm, probably I should add this into documentation.
>> 
>> What about word statistics (# of unique words, for example).
>> 
>
> I'm now following the guide to add the ispell dictionary and I've updated 
> most of the rows setting dict_name to NULL:
>
>     ts_name     |  tok_alias   | dict_name
> -----------------+--------------+-----------
> default         | lword        | {en_stem}
> default         | nlword       | {simple}
> default         | word         | {simple}
> default         | part_hword   | {simple}
> default         | nlpart_hword | {simple}
> default         | lpart_hword  | {en_stem}
> default         | hword        | {simple}
> default         | lhword       | {en_stem}
> default         | nlhword      | {simple}
>
> These are left, but I have no idea what a 'hword' or 'nlhword' or any other 
> of these tokens are.

from my notes http://www.sai.msu.su/~megera/oddmuse/index.cgi/Tsearch_V2_Notes
  I've asked how to know token types supported by parser. Actually, there is function token_type(parser), so you just use:

 	select * from token_type();

>
> Anyway, how do I find out the number of unique words or other word 
> statistics?


from my notes http://www.sai.msu.su/~megera/oddmuse/index.cgi/Tsearch_V2_Notes

It's usefull to see words statistics, for example, to check how good your 
dictionaries work or how did you configure pg_ts_cfgmap. Also, you may notice 
probable stop words relevant for your collection. 
Tsearch provides stat() function:

.......................

Don't hesitate to read it and if you find some bugs or know better wording
I'd be glad to improve my notes.

>
> Rick
>

 	Regards,
 		Oleg
_____________________________________________________________
Oleg Bartunov, sci.researcher, hostmaster of AstroNet,
Sternberg Astronomical Institute, Moscow University (Russia)
Internet: oleg@sai.msu.su, http://www.sai.msu.su/~megera/
phone: +007(095)939-16-83, +007(095)939-23-83



view thread (14+ messages)  latest in thread

Message-ID: <Pine.GSO.4.62.0503231236390.5508@ra.sai.msu.su>
Permalink:  ../Pine.GSO.4.62.0503231236390.5508@ra.sai.msu.su/
Also on:    postgresql.org/message-id/Pine.GSO.4.62.0503231236390.5508@ra.sai.msu.su

 · 

reply

Reply instructions:

You may reply publicly to this message via plain-text email
using any one of the following methods:

* Reply to all the recipients from your mail client; the link fills in
  the headers below: reply via email
  Some clients, webmail especially, drop In-Reply-To and References from
  a link, and the reply then starts a new thread: use the mbox below.

  To: pgsql-performance@postgresql.org
  Cc: oleg@sai.msu.su, rick@rockingstone.nl, mrylander@gmail.com
  Subject: Re: Tsearch2 performance on big database
  In-Reply-To: <Pine.GSO.4.62.0503231236390.5508@ra.sai.msu.su>
  References: <42400F57.3040604@rockingstone.nl> <Pine.GSO.4.62.0503221533400.5508@ra.sai.msu.su> <b918cf3d050322044847d3f9e1@mail.gmail.com> <42401CBF.50403@rockingstone.nl> <b918cf3d0503220730665fe92c@mail.gmail.com> <Pine.GSO.4.62.0503221840480.5508@ra.sai.msu.su> <42404263.40604@rockingstone.nl> <Pine.GSO.4.62.0503221907320.5508@ra.sai.msu.su> <42412E4B.4010407@rockingstone.nl> <Pine.GSO.4.62.0503231236390.5508@ra.sai.msu.su>

* Save the following mbox file, import it into your mail client,
  and reply-to-all from there: mbox

This inbox is served by DDX for PostgreSQL; see mirroring instructions
for how to clone and mirror all data and code used for this inbox