Received: from maia.hub.org (maia-5.hub.org [200.46.204.29]) by mail.postgresql.org (Postfix) with ESMTP id 949EFB5D944 for ; Thu, 16 Jun 2011 00:14:18 -0300 (ADT) Received: from mail.postgresql.org ([200.46.204.86]) by maia.hub.org (mx1.hub.org [200.46.204.29]) (amavisd-maia, port 10024) with ESMTP id 24471-06 for ; Thu, 16 Jun 2011 03:14:08 +0000 (UTC) X-Greylist: from auto-whitelisted by SQLgrey-1.7.6 Received: from sss.pgh.pa.us (sss.pgh.pa.us [66.207.139.130]) by mail.postgresql.org (Postfix) with ESMTP id 14E4AB5D8BD for ; Thu, 16 Jun 2011 00:14:07 -0300 (ADT) Received: from sss2.sss.pgh.pa.us (tgl@localhost [127.0.0.1]) by sss.pgh.pa.us (8.14.2/8.14.2) with ESMTP id p5G3E62K014752; Wed, 15 Jun 2011 23:14:06 -0400 (EDT) To: Tim cc: pgsql-admin@postgresql.org Subject: Re: tsvector limitations In-reply-to: References: <18955.1308162688@sss.pgh.pa.us> Comments: In-reply-to Tim message dated "Wed, 15 Jun 2011 22:34:01 -0400" Date: Wed, 15 Jun 2011 23:14:06 -0400 Message-ID: <14751.1308194046@sss.pgh.pa.us> From: Tom Lane X-Virus-Scanned: Maia Mailguard 1.0.1 X-Spam-Status: No, hits=-1.91 tagged_above=-10 required=5 tests=BAYES_00=-1.9, T_RP_MATCHES_RCVD=-0.01 X-Spam-Level: X-Archive-Number: 201106/86 X-Sequence-Number: 37712 Tim writes: > My limited testing indicates tsvector size has an approximately linear > (binomial) correlation with "number of unique words", and "word count". Yes, it would be linear in the number of unique words (assuming the average word length stays constant). It's not linear in word count, because tsvector only keeps track of distinct word positions for the first 16K words in the text. So for large documents only the number of unique words is really going to matter, and that grows *much* less than linearly with document size for real-world documents. regards, tom lane