X-Original-To: pgsql-general-postgresql.org@localhost.postgresql.org Received: from localhost (unknown [200.46.204.144]) by svr1.postgresql.org (Postfix) with ESMTP id CCA0D329C70 for ; Thu, 9 Sep 2004 16:33:25 +0100 (BST) Received: from svr1.postgresql.org ([200.46.204.71]) by localhost (av.hub.org [200.46.204.144]) (amavisd-new, port 10024) with ESMTP id 78989-06 for ; Thu, 9 Sep 2004 15:33:22 +0000 (GMT) Received: from 10.hostcorporate.com (unknown [69.41.231.186]) by svr1.postgresql.org (Postfix) with SMTP id E8C0D329C6B for ; Thu, 9 Sep 2004 16:33:18 +0100 (BST) Received: (qmail 20935 invoked from network); 9 Sep 2004 15:21:53 -0000 Received: from localhost (HELO localhost.localdomain) (127.0.0.1) by localhost.10.hostcorporate.com with SMTP; 9 Sep 2004 15:21:53 -0000 Message-ID: <414077B7.5080808@zara.6.isreserved.com> Date: Thu, 09 Sep 2004 22:33:11 +0700 From: David Garamond User-Agent: Mozilla/5.0 (X11; U; Linux i686; en-US; rv:1.7b) Gecko/20040421 X-Accept-Language: en-us, en MIME-Version: 1.0 To: Steve Atkins Cc: pgsql-general@postgresql.org Subject: Re: Text Search vs MYSQL vs Lucene References: <414052F4.5050000@portalvu.com> <20040909142006.GA24396@gp.word-to-the-wise.com> In-Reply-To: <20040909142006.GA24396@gp.word-to-the-wise.com> Content-Type: text/plain; charset=us-ascii; format=flowed Content-Transfer-Encoding: 7bit X-Virus-Scanned: by amavisd-new at hub.org X-Spam-Status: No, hits=0.0 tagged_above=0.0 required=5.0 tests= X-Spam-Level: X-Archive-Number: 200409/397 X-Sequence-Number: 65603 Steve Atkins wrote: >>What would be performance of pgSQL text search vs MySQL vs Lucene (flat >>file) for a 2 terabyte db? >>thanks for any comments. > > My experience with tsearch2 has been that indexing even moderately > large chunks of data is too slow to be feasible. Moderately large > meaning tens of megabytes. My experience with MySQL's full text search as well as the various MySQL-based text indexing programs (forgot the names, it's been a while) for some 10-20GB of mail archives has been pretty disappointing too. My biggest gripe is with the indexing speed. It literally takes days to index less than a million documents. I ended up using Swish++. Microsoft's CHM compiler also has pretty amazing indexing speed (though it crashes quite often when encountering bad HTML). -- dave