From: Tom Lane <tgl@sss.pgh.pa.us>
To: Mark Johnson <mark@remingtondatabasesolutions.com>
Cc: Kevin Grittner <Kevin.Grittner@wicourts.gov>
Cc: Tim <elatllat@gmail.com>
Cc: pgsql-admin@postgresql.org, "Greg Williamson" <gwilliamson39@yahoo.com>
Subject: Re: tsvector limitations
Date: Wed, 15 Jun 2011 14:31:28 -0400
Message-ID: <18955.1308162688@sss.pgh.pa.us> (raw)
In-Reply-To: <W565151360813081308158921@webmail12>
References: <W565151360813081308158921@webmail12>
"Mark Johnson" <mark@remingtondatabasesolutions.com> writes:
> When this discussion first started, I immediately thought about people
> who full text index their server's log files. As a test I copied
> /var/log/messages to $PGDATA and then used the same pg_read_file()
> function you mentioned earlier to pull the data into a column of type
> text. The original file was 4.3 MB, and the db column had length
> 4334920 and the function pg_column_size reported a size of 1058747. I
> then added a column named tsv of type tsvector, and populated it using
> to_tsvector(). The function pg_column_size reported 201557. So in this
> test a 4.2 MB text file produced a tsvector of size 200 KB. If this
> scales linearly,
... which it won't. There is no real-world full text indexing
application where there aren't many duplications of words. (The OP
eventually admitted that his "test case" was a dictionary word list
and not an actual document.) Any discussion of required tsvector
sizes that doesn't account for the actual, nonlinear scaling behavior
isn't worth the electrons it's printed on.
regards, tom lane
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Reply to all the recipients using the --to and --cc options:
reply via email
To: pgsql-admin@postgresql.org
Cc: tgl@sss.pgh.pa.us, mark@remingtondatabasesolutions.com, Kevin.Grittner@wicourts.gov, elatllat@gmail.com, gwilliamson39@yahoo.com
Subject: Re: tsvector limitations
In-Reply-To: <18955.1308162688@sss.pgh.pa.us>
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
This inbox is served by DDX for PostgreSQL; see mirroring instructions
for how to clone and mirror all data and code used for this inbox