Received: from malur.postgresql.org ([217.196.149.56])
 by arkaria.postgresql.org with esmtp (Exim 4.80) (envelope-from
 <pgsql-admin-owner+M42731=pgsql+2Dadmin=arkaria.postgresql.org@postgresql.org>)
 id 1XEhIu-0008IG-Od
 for pgsql-admin@arkaria.postgresql.org; Tue, 05 Aug 2014 16:09:36 +0000
Received: from localhost ([127.0.0.1] helo=postgresql.org)
 by malur.postgresql.org with smtp (Exim 4.80) (envelope-from
 <pgsql-admin-owner+M42731=pgsql+2Dadmin=arkaria.postgresql.org@postgresql.org>)
 id 1XEhIt-0002rl-W3
 for pgsql-admin@arkaria.postgresql.org; Tue, 05 Aug 2014 16:09:36 +0000
Received: from magus.postgresql.org ([2a02:c0:301:0:ffff::29])
 by malur.postgresql.org with esmtps (TLS1.2:DHE_RSA_AES_256_CBC_SHA256:256)
 (Exim 4.80) (envelope-from <jayknowsunix@gmail.com>)
 id 1XEhIs-0002re-SW
 for pgsql-admin@postgresql.org; Tue, 05 Aug 2014 16:09:34 +0000
Received: from mail-yh0-x22e.google.com ([2607:f8b0:4002:c01::22e])
 by magus.postgresql.org with esmtps (TLS1.0:RSA_AES_256_CBC_SHA1:256)
 (Exim 4.80) (envelope-from <jayknowsunix@gmail.com>)
 id 1XEhIj-0000qA-T0
 for pgsql-admin@postgresql.org; Tue, 05 Aug 2014 16:09:33 +0000
Received: by mail-yh0-f46.google.com with SMTP id a41so786167yho.19
 for <pgsql-admin@postgresql.org>; Tue, 05 Aug 2014 09:09:22 -0700 (PDT)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113;
 h=references:mime-version:in-reply-to:content-type
 :content-transfer-encoding:message-id:cc:from:subject:date:to;
 bh=O/lSMNXTEJyJ80Zd9adQOhaFbWJsQIDgQZKwfqj7+po=;
 b=w9O0oRw836nFH50OMWucx8G30SKTtQo/haPQ5pRcnbkh5sJYnCkiCeLOS0W8UpO6RM
 7Eq+1bOKD3CSrTLe8T2eWD+wGnbRmrc5vLIIKSvQn8sar2+nH6QpWMPFTPQE7yV9azHr
 Mw3Ll5J49hH+ELOk+bx67hQBwaNUbVNEqerOKcyxNEq4qfa4z0cvNu4T1UtKe/pVdl3H
 e8BmXrlZFY2fmHRsHI1PtaoU39Gpn2PoctxX5keAg+hURk9LvuQTvJvIULf4EuPH8UTQ
 zWr7kgOIsRyS3EtvgQM69BoMZ7HxYsZcolkORX4Aqj8Qr6meGYOi3h8gBiVUgL+zFzi6
 vdlQ==
X-Received: by 10.236.30.105 with SMTP id j69mr7217350yha.19.1407254962582;
 Tue, 05 Aug 2014 09:09:22 -0700 (PDT)
Received: from [192.168.254.1] (h77.89.102.166.static.ip.windstream.net.
 [166.102.89.77])
 by mx.google.com with ESMTPSA id k7sm3597590yhl.26.2014.08.05.09.09.21
 for <multiple recipients>
 (version=TLSv1 cipher=ECDHE-RSA-RC4-SHA bits=128/128);
 Tue, 05 Aug 2014 09:09:21 -0700 (PDT)
References: 
 <CAFwQ8rfsBZUSVEyhyZinBDJOQRkCjveVhLiKmHUkQEh2K+61nQ@mail.gmail.com>
Mime-Version: 1.0 (1.0)
In-Reply-To: 
 <CAFwQ8rfsBZUSVEyhyZinBDJOQRkCjveVhLiKmHUkQEh2K+61nQ@mail.gmail.com>
Content-Type: text/plain;
	charset=us-ascii
Content-Transfer-Encoding: quoted-printable
Message-Id: <9E9EF3AD-A2EC-4EEE-836C-860FECBD5F42@gmail.com>
Cc: "pgsql-admin@postgresql.org" <pgsql-admin@postgresql.org>
X-Mailer: iPad Mail (11D257)
From: jayknowsunix@gmail.com
Subject: Re: Stuck LSI 9650SE-12 RAID Controller
Date: Tue, 5 Aug 2014 12:09:21 -0400
To: Craig James <cjames@emolecules.com>
X-Pg-Spam-Score: -2.0 (--)
List-Archive: <http://archives.postgresql.org/pgsql-admin>
List-Help: <mailto:majordomo@postgresql.org?body=help>
List-ID: <pgsql-admin.postgresql.org>
List-Owner: <mailto:pgsql-admin-owner@postgresql.org>
List-Post: <mailto:pgsql-admin@postgresql.org>
List-Subscribe: <mailto:majordomo@postgresql.org?body=sub%20pgsql-admin>
List-Unsubscribe: <mailto:majordomo@postgresql.org?body=unsub%20pgsql-admin>
X-Mailing-List: pgsql-admin
Precedence: bulk
Sender: pgsql-admin-owner@postgresql.org

Certainly doesn't sound like a PostgreSQL issue. Is there any sort of advan=
ced diagnostics for the raid controller? I certainly would want to thrash i=
t top to bottom before I trusted it enough to put it back in service.
--
Jay

Sent from my iPad

> On Aug 5, 2014, at 12:00 PM, Craig James <cjames@emolecules.com> wrote:
>=20
> Has anyone seen anything like this?
>=20
> Our LSI 9650SE-12 RAID Controller dropped the main Postgres disk offline =
... it just disappeared as though the disk wasn't there.  It was an 8-disk =
RAID10 unit. The other unit (RAID1 for Linux & pg_xlog) was still functiona=
l.
>=20
> Using tw_cli, it showed the array as "DEGRADED" and claimed to be verifyi=
ng it. One disk in the array was "DEGRADED". There was no /dev entry for th=
e device; Linux couldn't see it at all.
>=20
> There were two hot spares, but it didn't use them. Worse, there was nothi=
ng I could do to make it do anything. Every command reported "Failed" and n=
o further explanation. Booting into the RAID BIOS gave the same problem: if=
 I selected "rebuild" or "verify", it said "You must select an array..." ev=
en though I had selected the array. It was as though the array didn't exist=
, yet it was shown.
>=20
> I shut off the computer, unplugged the BBU from the RAID card and plugged=
 it back in, unplugged and reinserted all the SATA cables, and then restart=
ed. Exact same symptoms.
>=20
> I finally gave up trying to recover the database (we had a backup server)=
. The RAID controller let me delete and recreate the degraded array, and no=
w everything seems fine. I can rebuild the Postgres database on the new uni=
t. But I've lost a HUGE amount of trust in the LSI 9650-SE RAID controller =
card.
>=20
> Thanks,
> Craig
>=20


--=20
Sent via pgsql-admin mailing list (pgsql-admin@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-admin