Subject: Infrastructure monitoring

From: "Jim C. Nasby" <[email-masked]>
Date: Fri, 13 Jan 2006 19:05:02 -0600

Search has been down for at least 2 days now, and this certainly isn't
the first time it's happened. There's also been cases of archives
getting stuck, and probably other outages besides those that went on
until someone email'd about it.

Would it be difficult to setup something to monitor these various
services? I know there's at least one OSS tool to do it, though I have
no idea how hard it would be to tie that into the current
infrastructure.
-- 
Jim C. Nasby, Sr. Engineering Consultant      [email-masked]
Pervasive Software      http://pervasive.com    work: 512-231-6117
vcard: http://jim.nasby.net/pervasive.vcf       cell: 512-569-9461

---

From: Josh Berkus <[email-masked]>
Date: Fri, 13 Jan 2006 17:14:47 -0800

Jim,

> no idea how hard it would be to tie that into the current

We have an open offer of Hyperic licenses, and they support FreeBSD now.

--Josh

-- 
__Aglio Database Solutions_______________
Josh Berkus		       Consultant
[email-masked]	 www.agliodbs.com
Ph: 415-752-2500	Fax: 415-752-2387
2166 Hayes Suite 200	San Francisco, CA

---

From: "Marc G. Fournier" <[email-masked]>
Date: Fri, 13 Jan 2006 22:16:59 -0400 (AST)

> We have an open offer of Hyperic licenses, and they support FreeBSD now.

Not to discount the offer ... but, what exactly would that provide us?  We 
already monitor the *servers*, its what is inside of the servers that 
needs better monitoring ... knowing nothing about Hyperic, does that 
provide something for that?

In the case of the archives, for instance, the problem was a perl process 
that for some unknown reason got stuck randomly ... removed that in favor 
of an awk script, and it hasn't done it since ... i also redirected cron's 
email to [email-masked], so that any errors show up in my mailbox 
instead of roots, so I get an hourly reminder that things are running well 
...

In the case of search ... John would be better at answering that, but when 
he and I talked this past week, he mentioned that he was moving it all 
over to two new servers, which I changed the DNS for on Wednesday ...

As I've said above ... physical servers are being monitored, so if anyone 
has some ideas on how we can improve "content monitoring", for lack of a 
better word, I know I'm all ears ...

Again, if Hyperic can offer something for this, let me know ...

----
Marc G. Fournier           Hub.Org Networking Services (http://www.hub.org)
Email: [email-masked]           Yahoo!: yscrappy              ICQ: 7615664

---

From: Guido Barosio <[email-masked]>
Date: Sat, 14 Jan 2006 02:51:35 +0000

Actually it seems to be as easy as requesting a GET to search.postgresql.org
If a script is able to handle the http codes, then alerts could be triggered
upon events.

The search failure was due to a 503 error being dispatched from the server.

GET http://search.postgresql.org

Am I wrong?

Tho, thinking about content,  there is an opensource doing a job such as
siteconf (http://www.siteconfidence.com) but I can't remember the name atm.

But I understand that the search problem was not a *content* problem itself.

G.-

On 1/14/06, Marc G. Fournier <[email-masked]> wrote:
> ---------------------------(end of broadcast)---------------------------
> TIP 2: Don't 'kill -9' the postmaster
>

--
/"\   ASCII Ribbon Campaign  .
\ / - NO HTML/RTF in e-mail  .
X  - NO Word docs in e-mail .
/ \ -----------------------------------------------------------------

---

From: "Jim C. Nasby" <[email-masked]>
Date: Mon, 16 Jan 2006 17:17:43 -0600

> instead of roots, so I get an hourly reminder that things are running well 

I would definately recommend that root@ be forwarded to at least one
(preferably more) people that will read it; especially on FBSD there's a
lot of problems that can be easily identified just be reading those
emails. portupgrade -N portaudit is a good idea, too.
-- 
Jim C. Nasby, Sr. Engineering Consultant      [email-masked]
Pervasive Software      http://pervasive.com    work: 512-231-6117
vcard: http://jim.nasby.net/pervasive.vcf       cell: 512-569-9461