X-Original-To: pgsql-performance-postgresql.org@localhost.postgresql.org Received: from localhost (unknown [200.46.204.144]) by svr1.postgresql.org (Postfix) with ESMTP id 1D57E5367A for ; Thu, 24 Mar 2005 10:52:00 +0000 (GMT) Received: from svr1.postgresql.org ([200.46.204.71]) by localhost (av.hub.org [200.46.204.144]) (amavisd-new, port 10024) with ESMTP id 66913-02 for ; Thu, 24 Mar 2005 10:51:58 +0000 (GMT) Received: from ra.sai.msu.su (ra.sai.msu.su [158.250.29.2]) by svr1.postgresql.org (Postfix) with ESMTP id 892D153641 for ; Thu, 24 Mar 2005 10:51:50 +0000 (GMT) Received: from ra (ra [158.250.29.2]) by ra.sai.msu.su (8.12.10/8.12.10) with ESMTP id j2OApg5Q011395; Thu, 24 Mar 2005 13:51:42 +0300 (MSK) Date: Thu, 24 Mar 2005 13:51:42 +0300 (MSK) From: Oleg Bartunov X-X-Sender: megera@ra.sai.msu.su To: Rick Jansen Cc: Mike Rylander , pgsql-performance@postgresql.org Subject: Re: Tsearch2 performance on big database In-Reply-To: <42429940.8040607@rockingstone.nl> Message-ID: References: <42400F57.3040604@rockingstone.nl> <42401CBF.50403@rockingstone.nl> <42404263.40604@rockingstone.nl> <42412E4B.4010407@rockingstone.nl> <42429940.8040607@rockingstone.nl> MIME-Version: 1.0 Content-Type: TEXT/PLAIN; charset=US-ASCII; format=flowed X-Virus-Scanned: by amavisd-new at hub.org X-Spam-Status: No, hits=0.341 tagged_above=0 required=5 tests=AWL, DNS_FROM_RFC_ABUSE X-Spam-Level: X-Archive-Number: 200503/502 X-Sequence-Number: 11207 On Thu, 24 Mar 2005, Rick Jansen wrote: > Oleg Bartunov wrote: >> from my notes >> http://www.sai.msu.su/~megera/oddmuse/index.cgi/Tsearch_V2_Notes >> >> It's usefull to see words statistics, for example, to check how good >> your dictionaries work or how did you configure pg_ts_cfgmap. Also, you >> may notice probable stop words relevant for your collection. Tsearch >> provides stat() function: >> >> ....................... >> >> Don't hesitate to read it and if you find some bugs or know better wording >> I'd be glad to improve my notes. >> > > Thanks, but that stat() query takes way too long.. I let it run for like > 4 hours and still nothing. The database I am testing tsearch2 on is also > the production database (mysql) server so I have to be careful not to > use too many resources :o stat() is indeed a bigdog, it was designed for developers needs, so we recommend to save results in table. > > Anyway, here's my pg_ts_cfgmap now (well the relevant bits): > > default_english | lhword | {en_ispell,en_stem} > default_english | lpart_hword | {en_ispell,en_stem} > default_english | lword | {en_ispell,en_stem} > > Is it normal that queries for single words (or perhaps they are words > that are common) take a really long time? Like this: > 'hispanic' isn't common, I see you get only 674 rows and 'buckingham & palace' returns 185 rows. Did you run 'vacuum analyze' ? I see a big discrepancy between estimated rows (8041) and actual rows. > ilab=# explain analyze select count(*) from books where description_fti @@ > to_tsquery('default', 'hispanic'); > QUERY PLAN > ------------------------------------------------------------------------------------------------------------------------------------------ > Aggregate (cost=20369.81..20369.81 rows=1 width=0) (actual > time=261512.031..261512.031 rows=1 loops=1) > -> Index Scan using idxfti_idx on books (cost=0.00..20349.70 rows=8041 > width=0) (actual time=45777.760..261509.288 rows=674 loops=1) > Index Cond: (description_fti @@ '\'hispan\''::tsquery) > Total runtime: 261518.529 ms > (4 rows) > > ilab=# explain analyze select titel from books where description_fti @@ > to_tsquery('default', 'buckingham & palace'); > QUERY PLAN > ------------------------------------------------------------------------------------------------------------------------------------ > Index Scan using idxfti_idx on books (cost=0.00..20349.70 rows=8041 > width=57) (actual time=18992.045..48863.385 rows=185 loops=1) > Index Cond: (description_fti @@ '\'buckingham\' & \'palac\''::tsquery) > Total runtime: 48863.874 ms > (3 rows) > > > I dont know what happened, these queries were a lot faster 2 days ago..what > the feck is going on?! > > Rick > > Regards, Oleg _____________________________________________________________ Oleg Bartunov, sci.researcher, hostmaster of AstroNet, Sternberg Astronomical Institute, Moscow University (Russia) Internet: oleg@sai.msu.su, http://www.sai.msu.su/~megera/ phone: +007(095)939-16-83, +007(095)939-23-83