X-Original-To: pgsql-admin-postgresql.org@localhost.postgresql.org Received: from localhost (unknown [200.46.204.144]) by svr1.postgresql.org (Postfix) with ESMTP id 2278432A11E for ; Wed, 29 Sep 2004 16:08:51 +0100 (BST) Received: from svr1.postgresql.org ([200.46.204.71]) by localhost (av.hub.org [200.46.204.144]) (amavisd-new, port 10024) with ESMTP id 76609-10 for ; Wed, 29 Sep 2004 15:08:27 +0000 (GMT) Received: from ra.sai.msu.su (ra.sai.msu.su [158.250.29.2]) by svr1.postgresql.org (Postfix) with ESMTP id 5C72232A0F9 for ; Wed, 29 Sep 2004 16:08:19 +0100 (BST) Received: from ra (ra [158.250.29.2]) by ra.sai.msu.su (8.12.10/8.12.10) with ESMTP id i8TF83QT002518; Wed, 29 Sep 2004 19:08:03 +0400 (MSD) Date: Wed, 29 Sep 2004 19:08:03 +0400 (MSD) From: Oleg Bartunov X-X-Sender: megera@ra.sai.msu.su To: Kris Kiger Cc: pgsql-admin@postgresql.org Subject: Re: tsearch2 poor performance In-Reply-To: <415AC7A2.9020906@musicrebellion.com> Message-ID: References: <71E37EF6B7DCC1499CEA0316A2568328DC9DF0@loki.wc.globexplorer.net> <4158409E.8060109@musicrebellion.com> <41584E49.5040301@musicrebellion.com> <41586264.5020302@musicrebellion.com> <415AC7A2.9020906@musicrebellion.com> MIME-Version: 1.0 Content-Type: TEXT/PLAIN; charset=US-ASCII X-Virus-Scanned: by amavisd-new at hub.org X-Spam-Status: No, hits=0.0 tagged_above=0.0 required=5.0 tests= X-Spam-Level: X-Archive-Number: 200409/274 X-Sequence-Number: 14977 Kris, is't possible to share your "paragraph generator" ? It'd be better than downloading the whole dump :) On Wed, 29 Sep 2004, Kris Kiger wrote: > Oleg, the data I have right now was generated using a random paragraph > generator. The words are real words, but there are only 508 distinct > keywords in the 3,000,000 records that tsearch2 will pick up, using > default settings. I was using this data set for the purpose of testing > tsearch2's capabilities, so it's not real world data. If you still want > it, let me know where to send it and I will send you a dump of the DB. > > Kris > > > > Oleg Bartunov wrote: > > >Kris, > > > >we're working on prototype of tsearchd - full text search daemon, which > >maintain static inverted index outside of postgresql using the same > >parser, dictionary tsearch2 does. This approach could scale up > >fts capability preserving access to metadata, so yo may have > >"archive" part of your collection (tsearchd) and "online", which could be > >searchable with tsearch2. > > > >Here is what we have right now: > > > >pages ( tid integer, fts_index tsvector) > > > >1. Create index > >select count(tdindex(tid,fts_index)) from pages; > >2. Flush index > >select tdflush(); > >3. Search > >select pages.tid, rank(fts_index,to_tsquery('supernovae & magellan')) as rank > >from pages, tdsearch(to_tsquery('supernovae & magellan')) as idx where > >tid=idx order by rank desc; > > > >If it's possible, you could share your data, so we could test our > >prototype on real data. > > > > > > Oleg > > > > > > > ---------------------------(end of broadcast)--------------------------- > TIP 7: don't forget to increase your free space map settings > Regards, Oleg _____________________________________________________________ Oleg Bartunov, sci.researcher, hostmaster of AstroNet, Sternberg Astronomical Institute, Moscow University (Russia) Internet: oleg@sai.msu.su, http://www.sai.msu.su/~megera/ phone: +007(095)939-16-83, +007(095)939-23-83