X-Original-To: pgsql-admin-postgresql.org@localhost.postgresql.org Received: from localhost (unknown [200.46.204.144]) by svr1.postgresql.org (Postfix) with ESMTP id 9D38932A06F for ; Wed, 29 Sep 2004 15:34:06 +0100 (BST) Received: from svr1.postgresql.org ([200.46.204.71]) by localhost (av.hub.org [200.46.204.144]) (amavisd-new, port 10024) with ESMTP id 63923-01 for ; Wed, 29 Sep 2004 14:33:27 +0000 (GMT) Received: from mail.musicrebellion.com (dsl093-203-229.ind1.dsl.speakeasy.net [66.93.203.229]) by svr1.postgresql.org (Postfix) with ESMTP id 0BD5A32A0A0 for ; Wed, 29 Sep 2004 15:33:25 +0100 (BST) Received: from [10.0.1.102] (helo=musicrebellion.com ident=kris) by mail.musicrebellion.com with asmtp (Exim 4.34 #1 (Debian)) id 1CCfVx-0002qm-IB for ; Wed, 29 Sep 2004 09:33:07 -0500 Message-ID: <415AC7A2.9020906@musicrebellion.com> Date: Wed, 29 Sep 2004 09:33:06 -0500 From: Kris Kiger User-Agent: Mozilla/5.0 (X11; U; Linux i686; en-US; rv:1.0.0) Gecko/20020623 Debian/1.0.0-0.woody.1 X-Accept-Language: en MIME-Version: 1.0 To: pgsql-admin@postgresql.org Subject: Re: tsearch2 poor performance References: <71E37EF6B7DCC1499CEA0316A2568328DC9DF0@loki.wc.globexplorer.net> <4158409E.8060109@musicrebellion.com> <41584E49.5040301@musicrebellion.com> <41586264.5020302@musicrebellion.com> Content-Type: text/plain; charset=us-ascii; format=flowed Content-Transfer-Encoding: 7bit X-Virus-Scanned: by amavisd-new at hub.org X-Spam-Status: No, hits=0.0 tagged_above=0.0 required=5.0 tests= X-Spam-Level: X-Archive-Number: 200409/273 X-Sequence-Number: 14976 Oleg, the data I have right now was generated using a random paragraph generator. The words are real words, but there are only 508 distinct keywords in the 3,000,000 records that tsearch2 will pick up, using default settings. I was using this data set for the purpose of testing tsearch2's capabilities, so it's not real world data. If you still want it, let me know where to send it and I will send you a dump of the DB. Kris Oleg Bartunov wrote: >Kris, > >we're working on prototype of tsearchd - full text search daemon, which >maintain static inverted index outside of postgresql using the same >parser, dictionary tsearch2 does. This approach could scale up >fts capability preserving access to metadata, so yo may have >"archive" part of your collection (tsearchd) and "online", which could be >searchable with tsearch2. > >Here is what we have right now: > >pages ( tid integer, fts_index tsvector) > >1. Create index >select count(tdindex(tid,fts_index)) from pages; >2. Flush index >select tdflush(); >3. Search >select pages.tid, rank(fts_index,to_tsquery('supernovae & magellan')) as rank >from pages, tdsearch(to_tsquery('supernovae & magellan')) as idx where >tid=idx order by rank desc; > >If it's possible, you could share your data, so we could test our >prototype on real data. > > > Oleg > >