Subject: Re: Suggestion for improving Archives

From: "John Hansen" <[email-masked]>
Date: Sun, 5 Sep 2004 18:47:31 +1000

> Marc again dropped last time modification header, so it's 
> impossible to sort results by date (in general case ) without 
> specific parser.

Yes, that is unfortunate, but the code required to make this happen puts
stress on the archives to some degree.

> Also, he changed template for message. These changes cause 
> recrawling the whole archive each time and overloading 
> archives.postgresql.org More specific search engine could use 
> another source of information which messages to crawl, but 
> one we use at pgsql.ru is a general search engine and it 
> can't get modification date without proper header.

There should be no need to reindex the entire archive because of a
template change, since if you honor the embedded
<!--noindex-->..<!--/noindex--> tags, the body text never changes.
Unless of course, you want to keep an up-to-date cached copy.

> 
> I suggest:
> 
> 1. Use 3-server architecture (image server, frontend, backend) which
>    could be reduced to 2 servers (image+frontend, backend) -
>    frontend could be plain apache+mod_accel and serve/cache 
> all backends
>    outputs, backend is a modperl or/and php enabled apache.
> 2. return last modification header - be friendly to crawlers 
> and browsers 

Tho an accellerator would only work if last-modified header is returned
by the backend, this might be worth looking into.

> 3. stop changing message template
> 

Template changes are inevitable, they're part of progress :)

... John

---

From: Oleg Bartunov <[email-masked]>
Date: Sun, 5 Sep 2004 17:25:52 +0400 (MSD)

> stress on the archives to some degree.

What code ? I've seen that last modified header and now it's gone.
No stress on the archives, it's pure question of several lines of code

> Unless of course, you want to keep an up-to-date cached copy.

Hmm, this is rather non-standard feature of archives.postgresql.org.
The problem is not with index/reindex ! The problem with crawler which
doesn't have enough information to make a right decision.
I don't like non-standard solution/hack when there are standard and
reliable solutions.

> by the backend, this might be worth looking into.

I don't see a problem to return that header. But we'll have standard
solution for database driven site with dynamic content. Note,
one frontend could serve/hide many backends.

> Template changes are inevitable, they're part of progress :)

it's not a portal page, it's just a message, why should it changed so
often. I think I should teach our crawler to recognize if changes were
cosmetic using fuzzy checksum.

> ... John

	Regards,
		Oleg
_____________________________________________________________
Oleg Bartunov, sci.researcher, hostmaster of AstroNet,
Sternberg Astronomical Institute, Moscow University (Russia)
Internet: [email-masked], http://www.sai.msu.su/~megera/
phone: +007(095)939-16-83, +007(095)939-23-83

---

From: "Marc G. Fournier" <[email-masked]>
Date: Sun, 5 Sep 2004 11:16:14 -0300 (ADT)

> Unless of course, you want to keep an up-to-date cached copy.

I think what Oleg is referring to is that search engines generally compare 
the Last-Modified header before pulling in the whole file, to see if they 
are the same or not ... php, unfortunately, sets that to now(), so as far 
as SE's are concerned, every time they index is a new file :(

I'm going to play with mhonarc this week to see if I can get it to 
properly set Last-Modified to Date based on the message itself ... that 
will clean up that mess ...

Oleg, is there anything that I can put into <HEAD></HEAD> for this?  To 
avoid having to use PHP to do it?

----
Marc G. Fournier           Hub.Org Networking Services (http://www.hub.org)
Email: [email-masked]           Yahoo!: yscrappy              ICQ: 7615664