Subject: New PostgreSQL search resource

From: "Joshua D. Drake" <[email-masked]>
Date: Fri, 16 Jan 2004 16:33:38 -0800

Hello,

Took an hour today and made the 7.3.4, 7.4.1 and Practical PostgreSQL 
documentation
all searchable using OpenFTS and Tsearch2. You can take a look at:

http://www.commandprompt.com/community/

Sincerely,

Joshua Drake

-- 
Command Prompt, Inc., home of Mammoth PostgreSQL - S/ODBC and S/JDBC
Postgresql support, programming shared hosting and dedicated hosting.
+1-503-667-4564 - [email-masked] - http://www.commandprompt.com
Mammoth PostgreSQL Replicator. Integrated Replication for PostgreSQL

---

From: Robert Treat <[email-masked]>
Date: Fri, 16 Jan 2004 22:46:33 -0500

Ok, this is now the second site that has come to be in the last few days that 
is using FTS and Tsearch for site search, and providing something that seems 
a lot better than the search available on the main website... what is 
stopping us from implementing this and dumping mnogosearch?  at least for the 
main site if not for the archives?  

Robert Treat

-- 
Build A Brighter Lamp :: Linux Apache {middleware} PostgreSQL

---

From: "Marc G. Fournier" <[email-masked]>
Date: Fri, 16 Jan 2004 23:51:35 -0400 (AST)

> main site if not for the archives?

actually, Dave is working on an improved search ... but, Oleg just
announced a crawler using tsearch that I'm going to take a look at
implementing as well ...

> ---------------------------(end of broadcast)---------------------------
> TIP 4: Don't 'kill -9' the postmaster
>

----
Marc G. Fournier           Hub.Org Networking Services (http://www.hub.org)
Email: [email-masked]           Yahoo!: yscrappy              ICQ: 7615664

---

From: Oleg Bartunov <[email-masked]>
Date: Sat, 17 Jan 2004 07:48:10 +0300 (MSK)

> http://www.commandprompt.com/community/

I'd appreciate if you mention somewhere OpenFTS utilization, so people
could recognize it. But I'm not insisting ;)

> Joshua Drake

	Regards,
		Oleg
_____________________________________________________________
Oleg Bartunov, sci.researcher, hostmaster of AstroNet,
Sternberg Astronomical Institute, Moscow University (Russia)
Internet: [email-masked], http://www.sai.msu.su/~megera/
phone: +007(095)939-16-83, +007(095)939-23-83

---

From: "Joshua D. Drake" <[email-masked]>
Date: Fri, 16 Jan 2004 20:52:22 -0800

>could recognize it. But I'm not insisting ;)
Full intent to do so... want to get it finished up first. :)

-- 
Command Prompt, Inc., home of Mammoth PostgreSQL - S/ODBC and S/JDBC
Postgresql support, programming shared hosting and dedicated hosting.
+1-503-667-4564 - [email-masked] - http://www.commandprompt.com
PostgreSQL Replicator -- production quality replication for PostgreSQL

---

From: Oleg Bartunov <[email-masked]>
Date: Sat, 17 Jan 2004 07:57:36 +0300 (MSK)

> announced a crawler using tsearch that I'm going to take a look at

I suggest to follow Roberts suggestion about indexing main site.
Archives is a bit another thing. I already wrote it needs to be
optimized for crawlers (headers).

Also, It would be much better to be able to index just content without
headers/footers etc. If you have access to files :), you may index
much faster without any crawler ! OpenFTS distribution contains
example scripts to index file collections. It's very very easy.

I quoting from "Survival Guide":

   APOD collection is consists of 1757 articles (about 7 Mb) and ideally suited
   for OpenFTS. Indexing tooks about 29 seconds on my IBM ThinkPad T21 notebook
   ( Linux, 2.4.17, 256 Mb RAM, 20 Gb IDE HD). Total number of lexems is
   131310, while the number of unique lexemes is only 8,806
  ( using Porter's stemmer ).

Official PostgreSQL documentation is about the same size.

> TIP 5: Have you checked our extensive FAQ?
>
>                http://www.postgresql.org/docs/faqs/FAQ.html
>

	Regards,
		Oleg

---

From: "scott.marlowe" <[email-masked]>
Date: Mon, 19 Jan 2004 10:38:04 -0700 (MST)

Is it possible to institute phrase searching with that system?

I.e.:

"create unique index"

"select distinct"

etc...

???

---

From: Josh Berkus <[email-masked]>
Date: Mon, 19 Jan 2004 09:57:54 -0800

Scott,

> "select distinct"

I believe that one does this by adding to the word root dictionary and 
rebuilding the index.   The problem is that then anything indexed as "select 
distinct" shows up under searches for "select distinct" but not for 
"distinct".

Try searching on SELECT & DISTINCT

-- 
-Josh Berkus
 Aglio Database Solutions
 San Francisco

---

From: "Joshua D. Drake" <[email-masked]>
Date: Mon, 19 Jan 2004 10:21:58 -0800

>Is it possible to institute phrase searching with that system?

It seems to work fine???

Sincerely,

Joshua D. Drake

---

From: "scott.marlowe" <[email-masked]>
Date: Mon, 19 Jan 2004 11:38:41 -0700 (MST)

> >Is it possible to institute phrase searching with that system?

Not really.  with htdig, I can do "select distinct" on the 7.3 docs and I 
get exactly 7 responses back, and they all have select followed 
immediately by distinct, with only space, CR / LF etc in between.

And the & symbol josh recommended shows as being ignored, so that didn't 
work either.

Word combos like "bulk loads" or "run-time configuration" are quite handy 
filters.

---

From: "Joshua D. Drake" <[email-masked]>
Date: Mon, 19 Jan 2004 11:03:00 -0800

>immediately by distinct, with only space, CR / LF etc in between.
Well, if you search for select distinct... OpenFTS automatically searches
for select & distinct. My search returned:

5 results with the first one being this:

http://www.commandprompt.com/community//search.lxp?lxpwrap=734/queries-select-lists.html

Which seems fairly accurate ;). Perhaps a better description of what the 
problem actually is or
the behavior you are expecting?

SIncerely,

Joshua D. Drake

---

From: Jussi Mikkola <[email-masked]>
Date: Mon, 19 Jan 2004 21:33:59 +0200

I think most of the people use http://www.postgresql.org/search.cgi

Default settings, search for select distinct returns in about 10 
minutes, with "You haven't typed any word(s) to search for."

Same query with match full phrase... (I did not wait long enough. But 
several minutes in any case. Don't know about the result.)

Same query with match full phrase and 7.4 documentation. 2 results in 5 
seconds. Results are select into and pg_select.

Could we do something to this search? (Change the defaults, divide it to 
smaller parts, use different search engine...)

Rgs,

Jussi

Joshua D. Drake wrote:

>
>> Not really.  with htdig, I can do "select distinct" on the 7.3 docs 
>> and I get exactly 7 responses back, and they all have select followed 
> Which seems fairly accurate ;). Perhaps a better description of what 
> the problem actually is or
>> And the & symbol josh recommended shows as being ignored, so that 
>> didn't work either.
>>
>> Word combos like "bulk loads" or "run-time configuration" are quite 
>> handy filters.
>>  
>>
>
>

---

From: "Joshua D. Drake" <[email-masked]>
Date: Mon, 19 Jan 2004 11:42:15 -0800

> I think most of the people use http://www.postgresql.org/search.cgi

Yep... Supposedly it is being worked on to make it faster.

J

> Same query with match full phrase and 7.4 documentation. 2 results in 
> 5 seconds. Results are select into and pg_select.
>
> Could we do something to this search? (Change the defaults, divide it 
> to smaller parts, use different search engine...)
>
> Rgs,
>
> Jussi
>
>
>
>
>
> Joshua D. Drake wrote:
>
>>
>>> and I get exactly 7 responses back, and they all have select 
>>> followed immediately by distinct, with only space, CR / LF etc in 
>>> between.
>>>
>>>  
>>>

---

From: "scott.marlowe" <[email-masked]>
Date: Mon, 19 Jan 2004 13:27:13 -0700 (MST)

> the behavior you are expecting?

Yes, phrase searching generally means you enter an exact string of words, 
in a particular order, and it searches on those strings in that order.

this comes in handy when looking for strings where the words together mean 
one thing, and apart may mean something else, like

set transaction isolation level

Many of those words appear throughout the docs, and may well all appear on 
the same page but not be tied together with each other on each page.

If we do a phrase search we only see the places where those words are 
together.  Individual word searching gives me 11 results in htdig, whereas 
the phrase gives me 3 hits, the second of which is obviously the correct 
one (the first is release notes for 6.5...  hmmm.)  With the 11 results, 
moves it down to number 7.

This is especially useful when you can search on a couple of small phrases 
that might go together.  Anyway, it's no big deal, just wondering if it 
supported phrase searching.