X-Original-To: pgsql-admin-postgresql.org@localhost.postgresql.org Received: from localhost (unknown [200.46.204.2]) by svr1.postgresql.org (Postfix) with ESMTP id D1D1CD1BCB9 for ; Thu, 1 Apr 2004 04:19:37 +0000 (GMT) Received: from svr1.postgresql.org ([200.46.204.71]) by localhost (neptune.hub.org [200.46.204.2]) (amavisd-new, port 10024) with ESMTP id 36772-03 for ; Thu, 1 Apr 2004 00:19:35 -0400 (AST) Received: from sycorax.affordablehost.com (sycorax.affordablehost.com [216.46.192.84]) by svr1.postgresql.org (Postfix) with ESMTP id 79F3FD1E15E for ; Thu, 1 Apr 2004 00:19:34 -0400 (AST) Received: from h0040054d8ee4.ne.client2.attbi.com ([66.31.57.228] helo=e-greek.net) by sycorax.affordablehost.com with esmtp (TLSv1:RC4-MD5:128) (Exim 4.24) id 1B8tfp-0008Tk-Ln; Wed, 31 Mar 2004 23:19:25 -0500 Message-ID: <406B984E.7030109@e-greek.net> Date: Wed, 31 Mar 2004 23:19:26 -0500 From: Andrew Biagioni User-Agent: Mozilla/5.0 (Windows; U; Windows NT 5.0; en-US; rv:1.6) Gecko/20040113 X-Accept-Language: en-us, en, it, it-ch, fr-fr, fr-ch, de-de, de-ch MIME-Version: 1.0 To: "scott.marlowe" Cc: pgsql-admin@postgresql.org Subject: Re: Spontaneous PostgreSQL Server Reboot? References: In-Reply-To: Content-Type: text/plain; charset=us-ascii; format=flowed Content-Transfer-Encoding: 7bit X-AntiAbuse: This header was added to track abuse, please include it with any abuse report X-AntiAbuse: Primary Hostname - sycorax.affordablehost.com X-AntiAbuse: Original Domain - postgresql.org X-AntiAbuse: Originator/Caller UID/GID - [47 12] / [47 12] X-AntiAbuse: Sender Address Domain - e-greek.net X-Virus-Scanned: by amavisd-new at postgresql.org X-Archive-Number: 200404/1 X-Sequence-Number: 12974 scott.marlowe wrote: > On Tue, 30 Mar 2004, Andrew Biagioni wrote: > > >>Alex, >> >>the answer is "no" to all of these. We are a tiny start-up (2 guys, and >>we do our own cleaning); ambient temperature varies significantly but >>is not related to the failure, and one machine starts beeping when it >>gets too hot (then we added an extra case fan); no fancy watchdogs >>(maybe someday... One can only dream :-> ); three different cases, >>power supplies, motherboards, etc., etc. (one power supply is >>extra-large, and that's the machine that started failing first!). >> >>We originally blamed the problem on hardware failure (first machine); >>then on OS version/configuration (second machine); now we're out of >>things to blame, except maybe unusually bad luck... > > > What did memtest86 say? > > Did the same person build all the machines? I've seen plenty of folks > build machines and zap the memory when installing it. >95% of all ESD > failures are partial / delayed failures, so just because a computer boots > up doesn't mean proper ESD procedures were followed, and if not, and if > you're in a dry environment like I am (I live in Denver) then it's quite > possible all three have bad CPU/mobo/memory or something like that. Two different people built the machines; we're both electrical engineers with plenty of familiarity and experience with static issues, so that particular issue is not likely. As for memtest86 - I haven't been able to run it on two of the machines yet (they are in production), and I have to restart the third one (it was "retired" after the third time it died on us). Meanwhile I found out some more details: - the first machine had a software raid system that may have been unreliable - the second machine had a much older kernel and sloppily-updated modules, and it would hang -- not reboot - the last machine to reboot MAY have been a line power issue (the whole building lost power a few hours later, so I lost some info on other machines' restarting -- I'll dig more). So -- it's memtest86 and badblocks for all three (as soon as I can), better UPS-ing, updated kernel(s), and checking more machines' logs; then we'll see... Thanks to you all for the suggestions -- keep them coming! Andrew