X-Original-To: pgsql-performance-postgresql.org@localhost.postgresql.org Received: from localhost (unknown [200.46.204.144]) by svr1.postgresql.org (Postfix) with ESMTP id 20C1953546 for ; Wed, 23 Mar 2005 08:52:38 +0000 (GMT) Received: from svr1.postgresql.org ([200.46.204.71]) by localhost (av.hub.org [200.46.204.144]) (amavisd-new, port 10024) with ESMTP id 87957-05 for ; Wed, 23 Mar 2005 08:52:30 +0000 (GMT) Received: from mx1.rockingstone.nl (mx1.rockingstone.nl [82.94.241.100]) by svr1.postgresql.org (Postfix) with ESMTP id D81D353537 for ; Wed, 23 Mar 2005 08:52:29 +0000 (GMT) Received: from localhost (web1 [127.0.0.1]) by mx1.rockingstone.nl (Postfix) with ESMTP id 5CB20FB0B1; Wed, 23 Mar 2005 09:58:33 +0100 (CET) Received: from mx1.rockingstone.nl ([127.0.0.1]) by localhost (web1 [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id 17323-07; Wed, 23 Mar 2005 09:58:32 +0100 (CET) Received: from [10.10.20.219] (217-19-26-53.dsl.cambrium.nl [217.19.26.53]) by mx1.rockingstone.nl (Postfix) with ESMTP id EC41EFB0A8; Wed, 23 Mar 2005 09:58:31 +0100 (CET) Message-ID: <42412E4B.4010407@rockingstone.nl> Date: Wed, 23 Mar 2005 09:52:27 +0100 From: Rick Jansen User-Agent: Mozilla Thunderbird 1.0 (X11/20041206) X-Accept-Language: en-us, en MIME-Version: 1.0 To: Oleg Bartunov Cc: Mike Rylander , pgsql-performance@postgresql.org Subject: Re: Tsearch2 performance on big database References: <42400F57.3040604@rockingstone.nl> <42401CBF.50403@rockingstone.nl> <42404263.40604@rockingstone.nl> In-Reply-To: X-Enigmail-Version: 0.90.0.0 X-Enigmail-Supports: pgp-inline, pgp-mime Content-Type: text/plain; charset=UTF-8; format=flowed Content-Transfer-Encoding: 7bit X-Virus-Scanned: by amavisd-new at rockingstone.nl X-Virus-Scanned: by amavisd-new at hub.org X-Spam-Status: No, hits=0 tagged_above=0 required=5 tests= X-Spam-Level: X-Archive-Number: 200503/485 X-Sequence-Number: 11190 Oleg Bartunov wrote: > On Tue, 22 Mar 2005, Rick Jansen wrote: > > Hmm, default configuration is too eager, you index every lexem using > simple dictionary) ! Probably, it's too much. Here is what I have for my > russian configuration in dictionary database: > > default_russian | lword | {en_ispell,en_stem} > default_russian | lpart_hword | {en_ispell,en_stem} > default_russian | lhword | {en_ispell,en_stem} > default_russian | nlword | {ru_ispell,ru_stem} > default_russian | nlpart_hword | {ru_ispell,ru_stem} > default_russian | nlhword | {ru_ispell,ru_stem} > > Notice, I index only russian and english words, no numbers, url, etc. > You may just delete unwanted rows in pg_ts_cfgmap for your configuration, > but I'd recommend just update them setting dict_name to NULL. > For example, to not indexing integers: > > update pg_ts_cfgmap set dict_name=NULL where ts_name='default_russian' > and tok_alias='int'; > > voc=# select token,dict_name,tok_type,tsvector from ts_debug('Do you > have +70000 bucks'); > token | dict_name | tok_type | tsvector > --------+---------------------+----------+---------- > Do | {en_ispell,en_stem} | lword | > you | {en_ispell,en_stem} | lword | > have | {en_ispell,en_stem} | lword | > +70000 | | int | > bucks | {en_ispell,en_stem} | lword | 'buck' > > Only 'bucks' gets indexed :) > Hmm, probably I should add this into documentation. > > What about word statistics (# of unique words, for example). > I'm now following the guide to add the ispell dictionary and I've updated most of the rows setting dict_name to NULL: ts_name | tok_alias | dict_name -----------------+--------------+----------- default | lword | {en_stem} default | nlword | {simple} default | word | {simple} default | part_hword | {simple} default | nlpart_hword | {simple} default | lpart_hword | {en_stem} default | hword | {simple} default | lhword | {en_stem} default | nlhword | {simple} These are left, but I have no idea what a 'hword' or 'nlhword' or any other of these tokens are. Anyway, how do I find out the number of unique words or other word statistics? Rick -- Systems Administrator for Rockingstone IT http://www.rockingstone.com http://www.megabooksearch.com - Search many book listing sites at once