Subject: Re: Infrastructure monitoring

From: "Magnus Hagander" <[email-masked]>
Date: Sat, 14 Jan 2006 12:16:25 +0100

> >> Search has been down for at least 2 days now, and this certainly 
> >> isn't the first time it's happened. There's also been cases of 
> >> archives getting stuck, and probably other outages besides 
> those that 
> >> went on until someone email'd about it.
> >>
> >> Would it be difficult to setup something to monitor these various 
> >> services? I know there's at least one OSS tool to do it, though I 
> >> have no idea how hard it would be to tie that into the current 
> >> infrastructure.
> >
> > We have an open offer of Hyperic licenses, and they support 
> FreeBSD now.
> 
> Not to discount the offer ... but, what exactly would that 
> provide us?  We already monitor the *servers*, its what is 
> inside of the servers that needs better monitoring ... 
> knowing nothing about Hyperic, does that provide something for that?

I assume you talk about the nagios monitoring? Or are there perhaps even
now multiple sets of monitoring? (Dave has a nagios installation up at
least).

We could easily extend that to monitor much more detailed. It's just
that someone has to define what we need to monitor. And in either case,
I see no reason we should require commercial software to do it - that's
still going to need the definition of what has to be monitored. Let's
stick to opensource when we can...

BTW, we already do content monitoring on the actual website mirrors. If
a mirror does not answer, *or* does not update properly, it will
automatically be removed from the DNS record, and thus get out of
"public view" after 10-30 minutes.

> In the case of the archives, for instance, the problem was a 
> perl process that for some unknown reason got stuck randomly 
> ... removed that in favor of an awk script, and it hasn't 
> done it since ... i also redirected cron's email to 
> [email-masked], so that any errors show up in my 
> mailbox instead of roots, so I get an hourly reminder that 
> things are running well ...

Right. What we could do to easily enhance this is to have the update
script update a timestamp file somewhere on the system when it's done,
and then monitor that file using existing tools (the file should be
accessible through http://archives.postgresql.org/ the same way it is
for the general website). Then you can just define a "can get <nn>
minutes out of sync before we scream"..

> In the case of search ... John would be better at answering 
> that, but when he and I talked this past week, he mentioned 
> that he was moving it all over to two new servers, which I 
> changed the DNS for on Wednesday ...

What I think would be good in cases like this is just information -
AFAIK nobody on the web team knew hte servers were being moved. (I may
be wrong here - I know I didn't know and I also spoke to Dave about it,
but those are the only ones I polled. Anyway, -www should know)

That would also make it possible to do the standard fiddling with DNS
TTLs to make the problem much smaller.

//Magnus

---

From: Josh Berkus <[email-masked]>
Date: Sat, 14 Jan 2006 12:17:12 -0800

> now multiple sets of monitoring? (Dave has a nagios installation up at

For those of you who haven't seen Hyperic, think of Nagios with a fancy web UI 
including notification management, scheduled tasks, historical reporting, and 
specific monitioring tools for PostgreSQL databases and other common 
applications.    As a comparison,  Nagios::Hyperic --> ed::vi  or  
amanda::Arkieka

The one hitch there is that all of this functionality would require that 
Hyperic have a server to collect data and run the web interface -- it has 
substantial resource consumption.  Possibly Hyperic LLC would supply this 
too, in exchange for a case study, but we'd have to ask them.

-- 
Josh Berkus
Aglio Database Solutions
San Francisco

---

From: Dave Page <[email-masked]>
Date: Sat, 14 Jan 2006 20:55:52 +0000

> applications.    As a comparison,  Nagios::Hyperic --> ed::vi  or

Which is all very nice but what would any of those features give us that we
have any use for?

Regards, Dave.

---

From: Josh Berkus <[email-masked]>
Date: Sat, 14 Jan 2006 13:49:58 -0800

Dave,

> > For those of you who haven't seen Hyperic, think of Nagios with a fancy
> > web UI including notification management, scheduled tasks, historical
> > reporting, and specific monitioring tools for PostgreSQL databases and
> > other common applications.    As a comparison,  Nagios::Hyperic -->
> > ed::vi  or amanda::Arkieka
>

Hey, it's your bag ... if you don't want it, I won't ask about setting it up.

-- 
Josh Berkus
Aglio Database Solutions
San Francisco

---

From: Dave Page <[email-masked]>
Date: Sat, 14 Jan 2006 21:55:17 +0000

> Hey, it's your bag ... if you don't want it, I won't ask about setting it up.

OK, well unless anyone else disagrees - thanks, but no thanks. I honestly
can't see what we would gain.

Regards, Dave.

---

From: "Marc G. Fournier" <[email-masked]>
Date: Sat, 14 Jan 2006 22:43:08 -0400 (AST)

> minutes out of sync before we scream"..

'k, done ... http://archives.postgresql.org/timestamp ... should be there 
in about 40 minutes or so, and is updated hourly ... let's say if its more 
then 6 hours out ... ?

----
Marc G. Fournier           Hub.Org Networking Services (http://www.hub.org)
Email: [email-masked]           Yahoo!: yscrappy              ICQ: 7615664

---

From: Bruce Momjian <[email-masked]>
Date: Mon, 16 Jan 2006 14:35:52 -0500 (EST)

> For those of you who haven't seen Hyperic, think of Nagios with a fancy web UI 

Hyper-nachos?  :-)

-- 
  Bruce Momjian                        |  http://candle.pha.pa.us
  [email-masked]               |  (610) 359-1001
  +  If your life is a hard drive,     |  13 Roberts Road
  +  Christ can be your backup.        |  Newtown Square, Pennsylvania 19073

---

From: "Jim C. Nasby" <[email-masked]>
Date: Mon, 16 Jan 2006 18:12:45 -0600

> "public view" after 10-30 minutes.

And this is how all the services should work, at least from a monitoring
standpoint. If any public service (any of the websites, search,
archives, email, ftp, etc) goes down, multiple people should get pages.
Along those lines, disk space should also be monitored to make sure
nothing fills up.
 
> but those are the only ones I polled. Anyway, -www should know)

And info is one of the other keys to keeping things running smoothly...
ISTM any changes in service/outages should certainly be posted someplace
where those monitoring things know what's going on.
-- 
Jim C. Nasby, Sr. Engineering Consultant      [email-masked]
Pervasive Software      http://pervasive.com    work: 512-231-6117
vcard: http://jim.nasby.net/pervasive.vcf       cell: 512-569-9461