X-Original-To: pgsql-performance-postgresql.org@localhost.postgresql.org Received: from localhost (unknown [200.46.204.144]) by svr1.postgresql.org (Postfix) with ESMTP id 8A05F56440 for ; Tue, 22 Mar 2005 16:38:35 +0000 (GMT) Received: from svr1.postgresql.org ([200.46.204.71]) by localhost (av.hub.org [200.46.204.144]) (amavisd-new, port 10024) with ESMTP id 91559-01 for ; Tue, 22 Mar 2005 16:38:25 +0000 (GMT) Received: from ra.sai.msu.su (ra.sai.msu.su [158.250.29.2]) by svr1.postgresql.org (Postfix) with ESMTP id 6A4BC52D42 for ; Tue, 22 Mar 2005 16:38:17 +0000 (GMT) Received: from ra (ra [158.250.29.2]) by ra.sai.msu.su (8.12.10/8.12.10) with ESMTP id j2MGc55Q005475; Tue, 22 Mar 2005 19:38:06 +0300 (MSK) Date: Tue, 22 Mar 2005 19:38:05 +0300 (MSK) From: Oleg Bartunov X-X-Sender: megera@ra.sai.msu.su To: Rick Jansen Cc: Mike Rylander , pgsql-performance@postgresql.org Subject: Re: Tsearch2 performance on big database In-Reply-To: <42404263.40604@rockingstone.nl> Message-ID: References: <42400F57.3040604@rockingstone.nl> <42401CBF.50403@rockingstone.nl> <42404263.40604@rockingstone.nl> MIME-Version: 1.0 Content-Type: TEXT/PLAIN; charset=US-ASCII; format=flowed X-Virus-Scanned: by amavisd-new at hub.org X-Spam-Status: No, hits=0.343 tagged_above=0 required=5 tests=AWL, DNS_FROM_RFC_ABUSE X-Spam-Level: X-Archive-Number: 200503/454 X-Sequence-Number: 11159 On Tue, 22 Mar 2005, Rick Jansen wrote: > Oleg Bartunov wrote: >> Mike, >> >> no comments before Rick post tsearch configs and increased buffers ! >> Union shouldn't be faster than (term1|term2). >> tsearch2 internals description might help you understanding tsearch2 >> limitations. >> See http://www.sai.msu.su/~megera/oddmuse/index.cgi/Tsearch_V2_internals >> Also, don't miss my notes: >> http://www.sai.msu.su/~megera/oddmuse/index.cgi/Tsearch_V2_Notes >> >> Oleg > > Thanks Oleg, i've seen those pages before :) I've set shared_buffers to 45000 > now (yes thats probably very much, isn't it?) and it already seems a lot > quicker. > > How do I find out what my tsearch config is? I followed the intro > (http://www.sai.msu.su/~megera/oddmuse/index.cgi/tsearch-v2-intro) and > applied it to our books table, thats all, didnt change anything else about > configs. Hmm, default configuration is too eager, you index every lexem using simple dictionary) ! Probably, it's too much. Here is what I have for my russian configuration in dictionary database: default_russian | lword | {en_ispell,en_stem} default_russian | lpart_hword | {en_ispell,en_stem} default_russian | lhword | {en_ispell,en_stem} default_russian | nlword | {ru_ispell,ru_stem} default_russian | nlpart_hword | {ru_ispell,ru_stem} default_russian | nlhword | {ru_ispell,ru_stem} Notice, I index only russian and english words, no numbers, url, etc. You may just delete unwanted rows in pg_ts_cfgmap for your configuration, but I'd recommend just update them setting dict_name to NULL. For example, to not indexing integers: update pg_ts_cfgmap set dict_name=NULL where ts_name='default_russian' and tok_alias='int'; voc=# select token,dict_name,tok_type,tsvector from ts_debug('Do you have +70000 bucks'); token | dict_name | tok_type | tsvector --------+---------------------+----------+---------- Do | {en_ispell,en_stem} | lword | you | {en_ispell,en_stem} | lword | have | {en_ispell,en_stem} | lword | +70000 | | int | bucks | {en_ispell,en_stem} | lword | 'buck' Only 'bucks' gets indexed :) Hmm, probably I should add this into documentation. What about word statistics (# of unique words, for example). Regards, Oleg _____________________________________________________________ Oleg Bartunov, sci.researcher, hostmaster of AstroNet, Sternberg Astronomical Institute, Moscow University (Russia) Internet: oleg@sai.msu.su, http://www.sai.msu.su/~megera/ phone: +007(095)939-16-83, +007(095)939-23-83