Subject: Re: Postgresql.org search engine.

From: "Dave Page" <[email-masked]>
Date: Fri, 30 Jan 2004 20:53:35 -0000

> -----Original Message-----
> From: Marc G. Fournier [mailto:[email-masked]] 
> Sent: 30 January 2004 20:43
> To: Dave Page
> Cc: Oleg Bartunov; [email-masked]; [email-masked]
> Subject: Re: [pgsql-www] Postgresql.org search engine.
> 
> 
> k, before I regenerate the lists, is this stuff you want me 
> to add to the META DATA part?

There's not much point I don't think. It's the XML feed that might make
use of it, not the standard indexer.

What I really want to see is the absolute bare minimum in the msg files
(not even the titles that are there at the moment - speacking of which,
might be worth including them as a php var we can pickup from the
top_config.php)  - as per the example I emailed you. Then, we should be
able to do anything by editting the header and footer php include files.

Regards, Dave.

---

From: "Marc G. Fournier" <[email-masked]>
Date: Fri, 30 Jan 2004 17:01:58 -0400 (AST)

> able to do anything by editting the header and footer php include files.

D'oh ... I was going to say that I didn't think taht was possible, but, it
just might be ... seems I have a section declared twice (note that someone
else wrote this originally, I've only just begun to understand it to
modify it), so the second section is overriding the first, but I was only
ever seeing the first ...

Let me play with this over the weekend, I'll do a 'small sample set' that
you can look at the messages in, and we can go from there ...

----
Marc G. Fournier           Hub.Org Networking Services (http://www.hub.org)
Email: [email-masked]           Yahoo!: yscrappy              ICQ: 7615664

---

From: Oleg Bartunov <[email-masked]>
Date: Sat, 31 Jan 2004 08:47:34 +0300 (MSK)

> able to do anything by editting the header and footer php include files.

I don't understand waht's the problem having postings in raw format stored
in filesystem, metadatt - in postgres and show component which combines
both sources to nice html page. Dave could get raw postings from filesystem
using metadata and index them without any problem. Marc could change
html wrapping everyday and everybody are happy :)

>
> Regards, Dave.
>
> ---------------------------(end of broadcast)---------------------------
> TIP 5: Have you checked our extensive FAQ?
>
>                http://www.postgresql.org/docs/faqs/FAQ.html
>

	Regards,
		Oleg
_____________________________________________________________
Oleg Bartunov, sci.researcher, hostmaster of AstroNet,
Sternberg Astronomical Institute, Moscow University (Russia)
Internet: [email-masked], http://www.sai.msu.su/~megera/
phone: +007(095)939-16-83, +007(095)939-23-83

---

From: "Marc G. Fournier" <[email-masked]>
Date: Sat, 31 Jan 2004 01:49:09 -0400 (AST)

On Sat, 31 Jan 2004, Oleg Bartunov wrote:

> I don't understand waht's the problem having postings in raw format
> stored in filesystem, metadatt - in postgres and show component which
> combines both sources to nice html page. Dave could get raw postings
> from filesystem using metadata and index them without any problem. Marc
> could change html wrapping everyday and everybody are happy :)

Do you have software to do this, including all the inter-posting
references and followups?  Or do you propose we write this all from
scratch?

----

---

From: Josh Berkus <[email-masked]>
Date: Fri, 30 Jan 2004 22:01:15 -0800

Guys,

> references and followups?  Or do you propose we write this all from

Robert Bernier apparently wrote something to break up mail for inclusion in a 
database, and should be able to help in a couple months.  Josh Drake is also 
willing to help, and has already done a prototype wiithout header searching.

-- 
-Josh Berkus
 Aglio Database Solutions
 San Francisco

---

From: "Marc G. Fournier" <[email-masked]>
Date: Sat, 31 Jan 2004 02:14:34 -0400 (AST)

> willing to help, and has already done a prototype wiithout header searching.

Dumping mail into a database isn't that hard to do ... there are several
projects on the 'Net right now doing that, including one that connects a
POP3 daemon into the database to download the mail ... in fact, from what
I recall of fts.postgresql.org, isn't that what Oleg/Teodor's stuff does?

I'm kinda curious here ... exactly what problem are we trying to solve
here?

Me, I'm just trying to clean up the archives so that when someone gets
their search results, they don't all show the same 'text', which I've
already accomplished ... Dave is working on improving the speed of the
searches, which he has accomplished with ASPseek ...

If I can figure out how to get the Date: of the posting into the
Last-Modified field (I know *how* it should work, but last time I tried it
ended up generating a whack of errors), then that should satisfy Oleg's
beef ...

Oleg, one question ... what do you recommend setting max-age to for
Cache-control?  Right now, I have it set to 30 days ... too long?  not
long enough?

----

---

From: Oleg Bartunov <[email-masked]>
Date: Sat, 31 Jan 2004 15:45:28 +0300 (MSK)

> Cache-control?  Right now, I have it set to 30 days ... too long?  not

in my experience Cache-control is not effective, because it's
HTTP/1.1 feature and a lot of users come through proxy which still
doesn't support HTTP/1.1
Last-Modified header is the most universal way.
Check http://www.mnot.net/cache_docs/#CACHE-CONTROL

	Regards,
		Oleg

---

From: "Marc G. Fournier" <[email-masked]>
Date: Sun, 1 Feb 2004 18:12:02 -0400 (AST)

> > ended up generating a whack of errors), then that should satisfy Oleg's

'k, figured out my error with the mhonarc resource file, and now have
posting date in as last-modified ... I'm doing this off to the side right
now, while I work out the noindex stuff for Dave, but check out:

http://archives.postgresql.org/dev

And let me know if the headers look right to you ... I took out the
Cache-control stuff ...

Let me know if there is anything else you'd like to see in there ...

----

---

From: Oleg Bartunov <[email-masked]>
Date: Mon, 2 Feb 2004 15:57:33 +0300 (MSK)

> Let me know if there is anything else you'd like to see in there ...

http headers looks fine !

	Regards,
		Oleg