From: "Matt Clark" <matt@ymogen.net>
To: "Scott Cain" <cain@cshl.org>, "Richard Huxton" <dev@archonet.com>
Cc: "PgSQL Performance ML" <pgsql-performance@postgresql.org>,
	<pgsql-sql@postgresql.org>
Subject: Re: [SQL] EXTERNAL storage and substring on long strings
Date: Mon, 4 Aug 2003 17:56:00 +0100
Message-ID: <OAEAKHEHCMLBLIDGAFELEEDDDGAA.matt@ymogen.net>
MIME-Version: 1.0
Content-Type: text/plain;
	charset="iso-8859-1"
Content-Transfer-Encoding: 7bit
Importance: Normal
In-Reply-To: <1060013646.1434.40.camel@localhost.localdomain>

> > 2. If you want to search for a sequence you'll need to deal with the case
> > where it starts in one chunk and ends in another.
>
> I forgot about searching--I suspect that application is why I faced
> opposition for shredding in my schema development group.  Maybe I should
> push that off to the file system and use grep (or BLAST).  Otherwise, I
> could write a function that would search the chunks first, then after
> failing to find the substring in those, I could start sewing the chunks
> together to look for the query string.  That could get ugly (and
> slow--but if the user knows that and expects it to be slow, I'm ok with
> that).

If you know the max length of the sequences being searched for, and this is much less than the chunk size, then you could simply
have the chunks overlap by that much, thus guaranteeing every substring will be found in its entirety in at least one chunk.