Subject: A counter productive conversation about search.

From: "Joshua D. Drake" <[email-masked]>
Date: Mon, 28 Aug 2006 20:12:28 -0700

Hello,

Now that I have effectively slapped myself silly by being rude to Tom 
about search. Let me bring up some points about search and see if there 
is a way to resolve them.

The problem:

Search really isn't that good. Tom has good results with it, but I am 
guessing that because he is looking for specific things, likely just in 
archives as I doubt he often searches the documentation ;).

A quick search on google:

site:archives.postgresql.org index bloat

archives.postgresql.org/pgsql-performance/2005-04/msg00617.php
archives.postgresql.org/pgsql-performance/2005-04/msg00594.php
archives.postgresql.org/pgsql-performance/2005-04/msg00608.php

archives.postgresql.org:

http://archives.postgresql.org/pgsql-performance/2005-04/msg00575.php
http://archives.postgresql.org/pgsql-general/2004-12/msg00288.php
http://archives.postgresql.org/pgsql-general/2005-07/msg00186.php

site:www.postgresql.org create index
www.postgresql.org/docs/7.4/static/sql-createindex.html
www.postgresql.org/docs/8.1/static/sql-createindex.html
www.postgresql.org/files/documentation/books/aw_pgsql/node216.html

search.postgresql.org:
http://www.postgresql.org/files/documentation/books/aw_pgsql/node216.html
http://www.postgresql.org/files/documentation/books/pghandbuch/html/sql-createindex.html
http://developer.postgresql.org/~petere/past-events/lsm2003-slides/foil20.html

The first search is "reasonable" between the two, although it does not 
appear to correctly follow the thread path.

The second search to me is completely wrong. CREATE INDEX should always 
return the current documentation first. I can forgive google for showing 
7.4 first because it has been around longer and yet is still widely in use.

I have on multiple occasions brought up the idea of another search 
engine. I wrote the pgsql.ru guys and asked if they would share their 
code. To their benefit they said they would be willing but didn't have 
the time to install it for us. I told them I would be happy to muscle 
through it if they would just answer some emails. I never heard back.

Other options include lucene, and rolling our own.

Rolling our own really wouldn't be that hard "if" we can create a 
reasonably smart web page grabber. We have all the tools (tsearch2 and 
pg_pgtrm) to easily do the searches.

So is anyone up for helping develop a page grabber?

Sincerely,

Joshua D. Drake

-- 

    === The PostgreSQL Company: Command Prompt, Inc. ===
Sales/Support: +1.503.667.4564 || 24x7/Emergency: +1.800.492.2240
    Providing the most comprehensive  PostgreSQL solutions since 1997
              http://www.commandprompt.com/

---

From: Tino Wildenhain <[email-masked]>
Date: Tue, 29 Aug 2006 07:04:38 +0200

Joshua D. Drake wrote:
...
> So is anyone up for helping develop a page grabber?

Thats not the hardest part but why do we need to grab if the contents
of the pages could be in the database? But admittedly, I don't know
any good CMS w/ postgresql backend. But anyway, grabbing the sources
of the pages while they are published (like the docbook stuff
for the documentation) makes a lot more sense imho. Ditto for the
archives. Its much easier to get an idea of the structure and nature
of the data when you dont have to deal with the final result (e.g. HTML)

So a couple of scripts that fire when mail comes in, documentation
is compiled and when some other publishing takes place could
really help to keep the index in sync w/o having to crawl all sites
over and over again.

Regards
Tino Wildenhain

---

From: Oleg Bartunov <[email-masked]>
Date: Tue, 29 Aug 2006 09:37:04 +0400 (MSD)

> really help to keep the index in sync w/o having to crawl all sites

This is exactly what we have on pgsql.ru/db/mw. We use procmail to fire
our backend to process incoming message. This is not a problem, the
most complex thing is a backend.

>
> Regards
> Tino Wildenhain
>
>
> ---------------------------(end of broadcast)---------------------------
> TIP 1: if posting/reading through Usenet, please send an appropriate
>       subscribe-nomail command to [email-masked] so that your
>       message can get through to the mailing list cleanly
>

 	Regards,
 		Oleg
_____________________________________________________________
Oleg Bartunov, Research Scientist, Head of AstroNet (www.astronet.ru),
Sternberg Astronomical Institute, Moscow University, Russia
Internet: [email-masked], http://www.sai.msu.su/~megera/
phone: +007(495)939-16-83, +007(495)939-23-83

---

From: "Dave Page" <[email-masked]>
Date: Tue, 29 Aug 2006 08:45:10 +0100

> -----Original Message-----
> From: [email-masked] 
> [mailto:[email-masked]] On Behalf Of Joshua D. Drake
> Sent: 29 August 2006 04:12
> To: PostgreSQL WWW
> Subject: [pgsql-www] A counter productive conversation about search.
> 
> Hello,
> 

The search engine has no site specific knowledge - it (like any other
generic search engine) simply doesn't know about threading.

> 7.4 first because it has been around longer and yet is still 

That should be fixable by tweaking weighting values, however last time I
suggested that I got shot down.

> Other options include lucene, and rolling our own.

Is Lucene capable of handling the size of our index? This has always
been the problem we've had with other projects like MnogoSearch. They
work well until you load them up with the archives after which they
simply can't cope without ridiculous amounts of hardware.

> So is anyone up for helping develop a page grabber?

We have one - it builds the static version of the main site by spidering
it hourly.

Regards, Dave.

---

From: "John Hansen" <[email-masked]>
Date: Tue, 29 Aug 2006 17:51:32 +1000

Dave Page Wrote:

> That should be fixable by tweaking weighting values, however 
> last time I suggested that I got shot down.

Not so much shot down, but that it isn't possible. (at least not without rewriting code)

Siteweights are for sites, not for parts of a site.

However, if you browse to the documentation you want to search first, then only that part of the website will be searched.

Example:

Go to http://www.postgresql.org/docs/8.1/static/index.html and search for 'create index'

http://search.postgresql.org/www.search?ul=http%3A%2F%2Fwww.postgresql.org%2Fdocs%2F8.1%2Fstatic%2F%25&fm=on&cs=utf-8&q=create+index

---

From: "Dave Page" <[email-masked]>
Date: Tue, 29 Aug 2006 09:11:32 +0100

> From: John Hansen [mailto:[email-masked]] 
> Sent: 29 August 2006 08:52
> To: Dave Page; Joshua D. Drake; PostgreSQL WWW
> Subject: RE: [pgsql-www] A counter productive conversation 
> about search.
> 
> Dave Page Wrote:
> 

Yeah, I was thinking of Mnogosearch where a server can include
subsections so you can do things like:

ServerWeight 100
Server http://www.postgresql.org/docs/8.1/
ServerWeight 50
Server http://www.postgresql.org/docs/7.4/

I did get shot down as well though - iirc it was Oleg who was
essentially saying that the search engine should not have any knowledge
of the site beyond what it crawled.

> However, if you browse to the documentation you want to 
> search first, then only that part of the website will be searched.
> 
> Example:
> 
> http://search.postgresql.org/www.search?ul=http%3A%2F%2Fwww.po
> stgresql.org%2Fdocs%2F8.1%2Fstatic%2F%25&fm=on&cs=utf-8&q=create+index

Yeah.

/D

---

From: "Joshua D. Drake" <[email-masked]>
Date: Tue, 29 Aug 2006 07:28:05 -0700

> Is Lucene capable of handling the size of our index? This has always

I am going to say, "yes" without any actual knowledge because of Lucene 
but that is because I am putting more trust in the fact that it is an 
Apache project then anything. I will check.

> We have one - it builds the static version of the main site by spidering

Should we look at that then?

--

---

From: "Joshua D. Drake" <[email-masked]>
Date: Tue, 29 Aug 2006 07:31:45 -0700

> of the site beyond what it crawled.

That won't work in aspseek. I looked into that too. It applies the 
weight to the whole site.

Joshua D. Drake

--

---

From: "Dave Page" <[email-masked]>
Date: Tue, 29 Aug 2006 15:54:10 +0100

> From: Joshua D. Drake [mailto:[email-masked]] 
> Sent: 29 August 2006 15:28
> To: Dave Page
> Cc: PostgreSQL WWW
> Subject: Re: [pgsql-www] A counter productive conversation 
> about search.
> 
> 

Is that the "Royal" we? I'm currently in 'no more project atm' mode, but
as far as I'm concerned you're welcome to work on it yourself on the
understanding that whatever you come up with will only be accepted to
replace the current solution if the community (-www) agrees that it is
a) better from a user perspective, b) more maintainable and c) matches
the main site look'n'feel.

FWIW, one of the problems with building a true online index of the
archives is that we don't know what URL a message might have until it
has been indexed by the archives site. Currently that is then indexed by
ASPSeek sometime later. It might be worth considering rewriting both the
archives and the search to make it all truly realtime. That shouldn't be
terribly difficult apart from generating thread indexes/forward/back
links, and dealing with the historic URL problem. A SMOP you might
say...

:-p

Regards, Dave.

---

From: "Dave Page" <[email-masked]>
Date: Tue, 29 Aug 2006 15:54:50 +0100

> Sent: 29 August 2006 15:32
> To: Dave Page
> Cc: John Hansen; PostgreSQL WWW

Yeah, I was confusing my self with options in Mnogosearch (which is very
similar in many ways).

Regards, Dave.

---

From: Oleg Bartunov <[email-masked]>
Date: Tue, 29 Aug 2006 21:57:08 +0400 (MSD)

Hi there,

On Mon, 28 Aug 2006, Joshua D. Drake wrote:

>
> I have on multiple occasions brought up the idea of another search engine. I 
> wrote the pgsql.ru guys and asked if they would share their code. To their 
> benefit they said they would be willing but didn't have the time to install 
> it for us. I told them I would be happy to muscle through it if they would 
> just answer some emails. I never heard back.

Joshua, we'd be happy to help PostgreSQL community and actually we tried
in past developing pgsql.ru, but we have families and we're in situation we
need money to live.  We don't want to promise something we could break.
On pgsql.ru we have 2 search engines, one is a commercial version which 
crawl pages, index them and provide search. I and Teodor are not the only
owners, so there is a problem with it. Also, I don't like the idea to 
use it, since it's not fully online indexing. The second SE, based on 
tsearch2, is what we actually needed. Several years ago (fts.postgresql.org)
tsearch2 was slow, but now, when we have GiN support I see no
real problem to have fully online indexing. We plan to renew pgsql.ru after
releasing 8.2 and then we'll see how it's working.

Another problem, is how documents are getting indexed. We have special user,
called robot, which subscribed to almost all mailing list, and procmail
entry instructed to process incoming message using our CMS. This worked
nice and allows to be fully in sync. Of course, we depend on what messages
come to the robot. This is not a problem on arhives.postgresql.org, which
has full control on the mailing lists.

To index wwww.postgresql.org I see two alternatives:
1. periodically run script, which crawl the site
2. Have a real CMS with hook to indexer.

I suspect, that second way is a complex thing for the current state of art,
so I'd stay with the first one. Giving, that documentation changed slow and
only news pages require indexing, it's not a bad approximation.

Hmm,  looks like a mess :( The entire system needs to be rewritten !

It's my opinion, that without understanding what to index/search and 
financial support current thread is useless. Do we have any financing 
for that ?

Regards,
 	Oleg

btw, we have simple crawler for OpenFTS, available from 
http://openfts.sourceforge.net/contributions.shtml
Using it, it's possible to write simple script to index collections of
documents, like documentation. See examples on
http://mira.sai.msu.su/~megera/pgsql

> Rolling our own really wouldn't be that hard "if" we can create a reasonably 
> smart web page grabber. We have all the tools (tsearch2 and pg_pgtrm) to 
> easily do the searches.
>

 	Regards,
 		Oleg

---

From: Bruce Momjian <[email-masked]>
Date: Tue, 29 Aug 2006 16:12:42 -0400 (EDT)

> financial support current thread is useless. Do we have any financing 

EnterpriseDB created a PostgreSQL fund at the anniversary.  Perhaps we
can use some of that.  I am one of the people who control the fund.

-- 
  Bruce Momjian   [email-masked]
  EnterpriseDB    http://www.enterprisedb.com

  + If your life is a hard drive, Christ can be your backup. +

---

From: "Joshua D. Drake" <[email-masked]>
Date: Tue, 29 Aug 2006 13:14:21 -0700

> can use some of that.  I am one of the people who control the fund.

Well I already offered them money yesterday... :) However this really 
should go through the PostgreSQL Fundraising Group don't you think?

Sincerely,

Joshua D. Drake

--

---

From: "Joshua D. Drake" <[email-masked]>
Date: Tue, 29 Aug 2006 13:18:14 -0700

Oleg Bartunov wrote:
> need money to live.

Of course :) and I understand that. Which is why I sent you a 
sponsorship suggestion yesterday.

Let's take this off list and talk about some of the financial requirements.

Sincerely,

Joshua D. Drake

--

---

From: Bruce Momjian <[email-masked]>
Date: Tue, 29 Aug 2006 18:15:33 -0400 (EDT)

> should go through the PostgreSQL Fundraising Group don't you think?

Yes, it should go through the Fundraising Group, if that is possible.