Subject: Re: Suggestion for improving Archives

From: "Dave Page" <[email-masked]>
Date: Sat, 4 Sep 2004 11:42:55 +0100

-----Original Message-----
From: [email-masked] on behalf of Josh Berkus
Sent: Fri 9/3/2004 5:19 PM
To: PostgreSQL WWW Mailing List
Subject: [pgsql-www] Suggestion for improving Archives 

> Both of those resources are faster and better search engines than the Mhonarc 
> search could ever be, and neither eats CPU time on hub.org.   Yes?

Um, Mhonarc is the mail to html program which will still be required, and none of the searches run on hub.org currently anyway.

Regards, Dave

---

From: "John Hansen" <[email-masked]>
Date: Sun, 5 Sep 2004 04:53:13 +1000

> Note that the search functions haven't chewed up CPU on our 
> servers in over a month now ... John Hansen has been running 
> search.postgresql.org off of his server(s) for about that long now ...
> 

... It's been more than 2 months, but who's counting? :)

... John

---

From: "John Hansen" <[email-masked]>
Date: Sun, 5 Sep 2004 05:14:54 +1000

> the followign was searching "nested transaction support":
> 
> "Documents 1-10 of total 383 found.   Searching in 390035 
> documents took 
> 4.714 seconds."
> 
> Not what I'd consider "slow" ... granted, that last one on 
> Google too .2 seconds, but when we can build a server farm 
> like them, then I'll be worried about 4secs vs .2 :)
> 

And the archives is currently being crawled, and there was a vacuum
running (heavy db load).

nested transaction support:
Documents 1-10 of total 383 found.   Searching in 390035 documents took
0.104 seconds.

... John

---

From: "Dave Page" <[email-masked]>
Date: Sat, 4 Sep 2004 23:06:44 +0100

-----Original Message-----
From: [email-masked] on behalf of Greg Sabino Mullane
Sent: Sat 9/4/2004 8:57 PM
To: [email-masked]
Subject: Re: [pgsql-www] Suggestion for improving Archives

 
> "Documents 1-10 of total 383 found. Searching in 390035 documents took
> 4.714 seconds."
 
> The self-reported time should really not be used. I just ran a query, for
> example, that took 8 seconds as measured by my local clock, but reported
> searching in under 2 seconds: so obviously there are some other factors
> here. (I'll give maybe half a second for network times on my end).

Factors that would apply equally if we used pgsql.ru - what you are most likely seeing it the ads and other graphics loading. If we used pgsql.ru, you would still have those coming from hub.org.
 
> pgsql.ru: very fast, searches all sites at once, no advertisements,
> nice "group by site" feature, cool Mozilla plugin, BSD-licensed tech,
> written by PG developers

Would be linked from main site with ads if we started using it 'officially'.

> search.postgresql.org: linked from main site?

Yes, should we leave it hidden? :-) Note that in comparison to pgsql.ru, search.pg.org also groups by site, searches all sites at once and is seriously hacked by PG developers (not written I grant you). It is also open source (GPL, not BSD - not that I think that is particularly important to any of the users).
 
Regards, Dave.

---

From: "John Hansen" <[email-masked]>
Date: Sun, 5 Sep 2004 23:42:29 +1000

> 
> What code ? I've seen that last modified header and now it's gone.
> No stress on the archives, it's pure question of several lines of code
> 

Yes, which is exactly what we wanted to avoid, more php code.

> it's not a portal page, it's just a message, why should it 
> changed so often. I think I should teach our crawler to 
> recognize if changes were cosmetic using fuzzy checksum.
> 

No, but even something as simple as adding a new mailing list would then
cause you to recrawl the entire site.
I agree that the last-modified header is the best solution. (the value
of it being equal to the message date, that is)

... John

---

From: "John Hansen" <[email-masked]>
Date: Mon, 6 Sep 2004 03:19:29 +1000

> 
> k, check out:
> 
> http://archives.postgresql.org/sfpug/2004-09/msg00003.php
> 
> I have the meta tag in place ... please confirm that the 
> format is okay, as that is what mhonarc is getting from the 
> message itself ... I can reformat it if I have to using PHP, 
> but would like to avoid it if at all possible ... basically, 
> if the default will work, I'd like to leave it as is ...
> 

Last-Modified: Sun,  5 Sep 2004 04:38:32 +0100 (BST)

The date format for last-modified is not defined afaik, at least various
web servers seem to have different formats, so I'm guessing this would
be acceptable.

... John

---

From: "John Hansen" <[email-masked]>
Date: Wed, 8 Sep 2004 07:16:56 +1000

> 
> My apologies; using the Archive search was so painfully slow 
> that I'd stopped 
> using it months ago.    I didn't know that you'd speeded it up.
> 

:)

... John