Received: from maia.hub.org (maia-3.hub.org [200.46.204.243]) by mail.postgresql.org (Postfix) with ESMTP id 4880AB5DBDA for ; Mon, 13 Jun 2011 22:43:44 -0300 (ADT) Received: from mail.postgresql.org ([200.46.204.86]) by maia.hub.org (mx1.hub.org [200.46.204.243]) (amavisd-maia, port 10024) with ESMTP id 82272-07 for ; Tue, 14 Jun 2011 01:43:37 +0000 (UTC) X-Greylist: from auto-whitelisted by SQLgrey-1.7.6 Received: from omr15.networksolutionsemail.com (omr15.networksolutionsemail.com [205.178.146.65]) by mail.postgresql.org (Postfix) with ESMTP id 826B1B5DBC6 for ; Mon, 13 Jun 2011 22:43:37 -0300 (ADT) Received: from cm-omr3 (mail.networksolutionsemail.com [205.178.146.50]) by omr15.networksolutionsemail.com (8.13.6/8.13.6) with ESMTP id p5E1hbsZ024980 for ; Mon, 13 Jun 2011 21:43:37 -0400 X-Authenticated-IP: 205.178.146.50 Received: from [205.178.146.50] ([205.178.146.50:51792] helo=webmail13) by cm-omr3 (envelope-from ) (ecelerity 2.2.2.41 r(31179/31189)) with ESMTP id C5/6E-27068-9CCB6FD4; Mon, 13 Jun 2011 21:43:37 -0400 Received: from 71.171.97.233 (mark@remingtondatabasesolutions.com [71.171.97.233]) by webmail13 (Netsol 11.2.30) with WEBMAIL id 8074; Tue, 14 Jun 2011 01:43:37 +0000 From: "Mark Johnson" To: "Tim" , pgsql-admin@postgresql.org Importance: Normal Message-ID: X-Mailer: Network Solutions Webmail, Build 11.2.30 X-Originating-IP: [71.171.97.233] X-Forwarded-For: [(null)] Date: Tue, 14 Jun 2011 01:43:37 +0000 Subject: Re: tsvector limitations MIME-Version: 1.0 Content-Type: multipart/alternative; boundary="--=_vm_0011_W155343112_8074_1308015817" X-Virus-Scanned: Maia Mailguard 1.0.1 X-Spam-Status: No, hits=-0.735 tagged_above=-10 required=5 tests=BAYES_00=-1.9, HTML_MESSAGE=0.001, RCVD_IN_DNSWL_NONE=-0.0001, RCVD_NUMERIC_HELO=1.164 X-Spam-Level: X-Archive-Number: 201106/65 X-Sequence-Number: 37691 ----=_vm_0011_W155343112_8074_1308015817 Content-Type: text/plain; charset="iso-8859-1" Content-Transfer-Encoding: 7bit I found another post where you asked the same questions 5 months ago. Have you tested in that time? http://www.spinics.net/lists/pgsql-admin/msg19438.html A text search vector is an array of distinct lexemes (less any stopwords) and their positions. Taking your example we get ... select to_tsvector('the lord of the rings.txt') "answer"; answer ------------------- 'lord':2, 'rings.txt':5 You can put the length() function around it to just get the number of lexemes. This is the size in terms of number of distinct lexemes, not size in terms of space utilization. select length(to_tsvector('the lord of the rings.txt')) "answer"; answer -------- 2 You might find the tsvector data consumes 2x the space required by the input text. It will depend on your configuration and your input data. Test it and let us know what you find. -Mark -----Original Message----- From: Tim [mailto:elatllat@gmail.com] Sent: Monday, June 13, 2011 03:19 PM To: pgsql-admin@postgresql.org Subject: [ADMIN] tsvector limitations Dear list, How big of a file would one need to fill the 1MB limit of a tsvector? Reading http://www.postgresql.org/docs/9.0/static/textsearch-limitations.html seems to hint that filling a tsvector is improbable. Is there an easy way of query the bytes of a tsvector? something like length(tsvector) but bytes(tsvector). If there no easy method to query the bytes of a tsvector I realize the answer is highly dependent on the contents of the file, so here are 2 random examples: How many bytes of a tsvector would a 32MB ascii english unique word list make? How many bytes of a tsvector would something like "The Lord of the Rings.txt" make? If this limitation is ever hit is there a common practice for using more than one tsvector? Using a separate "one to many" table seems like an obvious solution piece, but I would not know how to detect or calculate how much text to give each tsvector. Assuming tsvectors can't be linked maybe they would need some overlap. Thanks in advance. ----=_vm_0011_W155343112_8074_1308015817 Content-Type: text/html; charset="iso-8859-1" Content-Transfer-Encoding: quoted-printable
 

I found another post where you asked= the same questions 5 months ago.  Have you tested in that time?&nbs= p; http://www.spinics.net/lists/pgsql-admin/msg19438.html


A text search vector is an array of distinct lexemes (less any= stopwords) and their positions.  Taking your example we get ...

select to_tsvector('the lord of the rings.txt') "answ= er";
      answer
= -------------------
'lord':2, 'rings.txt':5

You can put the length() function around it to just g= et the number of lexemes.  This is the size in terms of number of di= stinct lexemes, not size in terms of space utilization.

select length(to_tsvector('the lord of the rings.txt'= )) "answer";
  answer
--------        2

You might find the tsvector data consumes 2x the sp= ace required by the input text.  It will depend on your configuratio= n and your input data.  Test it and let us know what you find.

-Mark

-----Original Message-----
= From: Tim [mailto:elatllat@gmail.com]
Sent: Monday, June 13= , 2011 03:19 PM
To: pgsql-admin@postgresql.org
Subject: [ADMIN] tsvector limitations

Dear list,

How big of a fil= e would one need to fill the 1MB limit of a tsvector?
Reading http://www.postgresql.org/docs/9.0/static/text= search-limitations.html seems to hint that filling a tsvector is impr= obable.

Is there an easy way of query the bytes of a tsvector?
= something like length(tsvector) but bytes(tsvector).

If there no e= asy method to query the bytes of a tsvector
I realize the answer is hi= ghly dependent on the contents of the file, so here are 2 random examples= :
How many bytes of a tsvector would a 32MB ascii english unique word = list make?
How many bytes of a tsvector would something like "Th= e Lord of the Rings.txt" make?

If this limitation is ever h= it is there a common practice for using more than one tsvector?
Using = a separate "one to many" table seems like an obvious solution piece,
b= ut I would not know how to detect or calculate how much text to give each= tsvector.
Assuming tsvectors can't be linked maybe they would need so= me overlap.


Thanks in advance.
----=_vm_0011_W155343112_8074_1308015817--