pg.ddx.io  pgsql-admin@postgresql.org mailing list archive  
help / color / mirror / Atom feed
From: Andrew Biagioni <andrew.biagioni@e-greek.net>
To: scott.marlowe <scott.marlowe@ihs.com>
Cc: pgsql-admin@postgresql.org
Subject: Re: Spontaneous PostgreSQL Server Reboot?
Date: Wed, 31 Mar 2004 23:19:26 -0500
Message-ID: <406B984E.7030109@e-greek.net> (raw)
In-Reply-To: <Pine.LNX.4.33.0403310854320.26243-100000@css120.ihs.com>
References: <Pine.LNX.4.33.0403310854320.26243-100000@css120.ihs.com>

scott.marlowe wrote:

> On Tue, 30 Mar 2004, Andrew Biagioni wrote:
> 
> 
>>Alex,
>>
>>the answer is "no" to all of these.  We are a tiny start-up (2 guys, and 
>>we do our own cleaning);  ambient temperature varies significantly but 
>>is not related to the failure, and one machine starts beeping when it 
>>gets too hot (then we added an extra case fan);  no fancy watchdogs 
>>(maybe someday...  One can only dream :-> );  three different cases, 
>>power supplies, motherboards, etc., etc. (one power supply is 
>>extra-large, and that's the machine that started failing first!).
>>
>>We originally blamed the problem on hardware failure (first machine); 
>>then on OS version/configuration (second machine);  now we're out of 
>>things to blame, except maybe unusually bad luck...
> 
> 
> What did memtest86 say?
> 
> Did the same person build all the machines?  I've seen plenty of folks 
> build machines and zap the memory when installing it.  >95% of all ESD 
> failures are partial / delayed failures, so just because a computer boots 
> up doesn't mean proper ESD procedures were followed, and if not, and if 
> you're in a dry environment like I am (I live in Denver) then it's quite 
> possible all three have bad CPU/mobo/memory or something like that.

Two different people built the machines;  we're both electrical 
engineers with plenty of familiarity and experience with static issues, 
so that particular issue is not likely.

As for memtest86 - I haven't been able to run it on two of the machines 
yet (they are in production), and I have to restart the third one (it 
was "retired" after the third time it died on us).

Meanwhile I found out some more details:
- the first machine had a software raid system that may have been unreliable
- the second machine had a much older kernel and sloppily-updated 
modules, and it would hang -- not reboot
- the last machine to reboot MAY have been a line power issue (the whole 
building lost power a few hours later, so I lost some info on other 
machines' restarting -- I'll dig more).

So -- it's memtest86 and badblocks for all three (as soon as I can), 
better UPS-ing, updated kernel(s), and checking more machines' logs; 
then we'll see...

Thanks to you all for the suggestions -- keep them coming!

		Andrew




view thread (10+ messages)

Message-ID: <406B984E.7030109@e-greek.net>
Permalink:  ../406B984E.7030109@e-greek.net/
Also on:    postgresql.org/message-id/406B984E.7030109@e-greek.net

 · 

reply

Reply instructions:

You may reply publicly to this message via plain-text email
using any one of the following methods:

* Reply to all the recipients from your mail client; the link fills in
  the headers below: reply via email
  Some clients, webmail especially, drop In-Reply-To and References from
  a link, and the reply then starts a new thread: use the mbox below.

  To: pgsql-admin@postgresql.org
  Cc: andrew.biagioni@e-greek.net, scott.marlowe@ihs.com
  Subject: Re: Spontaneous PostgreSQL Server Reboot?
  In-Reply-To: <406B984E.7030109@e-greek.net>
  References: <Pine.LNX.4.33.0403310854320.26243-100000@css120.ihs.com> <406B984E.7030109@e-greek.net>

* Save the following mbox file, import it into your mail client,
  and reply-to-all from there: mbox

This inbox is served by DDX for PostgreSQL; see mirroring instructions
for how to clone and mirror all data and code used for this inbox