Message-ID: <44F3CAE6.4040204@wildenhain.de>
Date: Tue, 29 Aug 2006 07:04:38 +0200
From: Tino Wildenhain <tino@wildenhain.de>
User-Agent: Mail/News 1.5 (X11/20060228)
MIME-Version: 1.0
To: "Joshua D. Drake" <jd@commandprompt.com>
Cc: PostgreSQL WWW <pgsql-www@postgresql.org>
Subject: Re: A counter productive conversation about search.
References: <44F3B09C.3010104@commandprompt.com>
In-Reply-To: <44F3B09C.3010104@commandprompt.com>
Content-Type: text/plain; charset=ISO-8859-1
Content-Transfer-Encoding: 7bit

Joshua D. Drake wrote:
...
> Rolling our own really wouldn't be that hard "if" we can create a
> reasonably smart web page grabber. We have all the tools (tsearch2 and
> pg_pgtrm) to easily do the searches.
> 
> So is anyone up for helping develop a page grabber?

Thats not the hardest part but why do we need to grab if the contents
of the pages could be in the database? But admittedly, I don't know
any good CMS w/ postgresql backend. But anyway, grabbing the sources
of the pages while they are published (like the docbook stuff
for the documentation) makes a lot more sense imho. Ditto for the
archives. Its much easier to get an idea of the structure and nature
of the data when you dont have to deal with the final result (e.g. HTML)

So a couple of scripts that fire when mail comes in, documentation
is compiled and when some other publishing takes place could
really help to keep the index in sync w/o having to crawl all sites
over and over again.

Regards
Tino Wildenhain