Date: Tue, 5 Aug 2014 11:14:20 -0500 (CDT)
From: Scott Whitney <scott@journyx.com>
Reply-To: Scott Whitney <swhitney@journyx.com>
To: jayknowsunix@gmail.com
Cc: Craig James <cjames@emolecules.com>, pgsql-admin@postgresql.org
Message-ID: 
 <1611759767.760183.1407255260652.JavaMail.zimbra@mail.int.journyx.com>
In-Reply-To: <9E9EF3AD-A2EC-4EEE-836C-860FECBD5F42@gmail.com>
References: 
 <CAFwQ8rfsBZUSVEyhyZinBDJOQRkCjveVhLiKmHUkQEh2K+61nQ@mail.gmail.com>
 <9E9EF3AD-A2EC-4EEE-836C-860FECBD5F42@gmail.com>
Subject: Re: Stuck LSI 9650SE-12 RAID Controller
MIME-Version: 1.0
Content-Type: multipart/alternative;
 boundary="----=_Part_760182_208238390.1407255260651"
Thread-Topic: Stuck LSI 9650SE-12 RAID Controller
Thread-Index: qAbud70fQ56WwDI1lZGslXGv6uipkg==
Precedence: bulk
Sender: pgsql-admin-owner@postgresql.org

------=_Part_760182_208238390.1407255260651
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit

Unfortunately, yes, I have seen similar situations. On Adaptec, IBM ServeRAID and Perc cards. 

I would replace that card, personally, with a new one. Likely the card itself is going flaky. 

Usually when I have seen this, swapping the card for a like card and importing the RAID config from the drives resolves it, unless the card went REAL bad and actually damaged the RAID itself (which I have also seen). 

----- Original Message -----

> Certainly doesn't sound like a PostgreSQL issue. Is there any sort of
> advanced diagnostics for the raid controller? I certainly would want to
> thrash it top to bottom before I trusted it enough to put it back in
> service.
> --
> Jay

> Sent from my iPad

> > On Aug 5, 2014, at 12:00 PM, Craig James <cjames@emolecules.com> wrote:
> >
> > Has anyone seen anything like this?
> >
> > Our LSI 9650SE-12 RAID Controller dropped the main Postgres disk offline
> > ... it just disappeared as though the disk wasn't there. It was an 8-disk
> > RAID10 unit. The other unit (RAID1 for Linux & pg_xlog) was still
> > functional.
> >
> > Using tw_cli, it showed the array as "DEGRADED" and claimed to be verifying
> > it. One disk in the array was "DEGRADED". There was no /dev entry for the
> > device; Linux couldn't see it at all.
> >
> > There were two hot spares, but it didn't use them. Worse, there was nothing
> > I could do to make it do anything. Every command reported "Failed" and no
> > further explanation. Booting into the RAID BIOS gave the same problem: if
> > I selected "rebuild" or "verify", it said "You must select an array..."
> > even though I had selected the array. It was as though the array didn't
> > exist, yet it was shown.
> >
> > I shut off the computer, unplugged the BBU from the RAID card and plugged
> > it back in, unplugged and reinserted all the SATA cables, and then
> > restarted. Exact same symptoms.
> >
> > I finally gave up trying to recover the database (we had a backup server).
> > The RAID controller let me delete and recreate the degraded array, and now
> > everything seems fine. I can rebuild the Postgres database on the new
> > unit. But I've lost a HUGE amount of trust in the LSI 9650-SE RAID
> > controller card.
> >
> > Thanks,
> > Craig
> >

> --
> Sent via pgsql-admin mailing list (pgsql-admin@postgresql.org)
> To make changes to your subscription:
> http://www.postgresql.org/mailpref/pgsql-admin

------=_Part_760182_208238390.1407255260651
Content-Type: text/html; charset=utf-8
Content-Transfer-Encoding: quoted-printable

<html><body><div style=3D"font-family: times new roman, new york, times, se=
rif; font-size: 12pt; color: #000000"><div>Unfortunately, yes, I have seen =
similar situations. On Adaptec, IBM ServeRAID and Perc cards.</div><div><br=
></div><div>I would replace that card, personally, with a new one. Likely t=
he card itself is going flaky.</div><div><br></div><div>Usually when I have=
 seen this, swapping the card for a like card and importing the RAID config=
 from the drives resolves it, unless the card went REAL bad and actually da=
maged the RAID itself (which I have also seen).</div><div><br></div><div><b=
r></div><hr id=3D"zwchr"><blockquote style=3D"border-left:2px solid #1010FF=
;margin-left:5px;padding-left:5px;color:#000;font-weight:normal;font-style:=
normal;text-decoration:none;font-family:Helvetica,Arial,sans-serif;font-siz=
e:12pt;">Certainly doesn't sound like a PostgreSQL issue. Is there any sort=
 of advanced diagnostics for the raid controller? I certainly would want to=
 thrash it top to bottom before I trusted it enough to put it back in servi=
ce.<br>--<br>Jay<br><div><br></div>Sent from my iPad<br><div><br></div>&gt;=
 On Aug 5, 2014, at 12:00 PM, Craig James &lt;cjames@emolecules.com&gt; wro=
te:<br>&gt; <br>&gt; Has anyone seen anything like this?<br>&gt; <br>&gt; O=
ur LSI 9650SE-12 RAID Controller dropped the main Postgres disk offline ...=
 it just disappeared as though the disk wasn't there. &nbsp;It was an 8-dis=
k RAID10 unit. The other unit (RAID1 for Linux &amp; pg_xlog) was still fun=
ctional.<br>&gt; <br>&gt; Using tw_cli, it showed the array as "DEGRADED" a=
nd claimed to be verifying it. One disk in the array was "DEGRADED". There =
was no /dev entry for the device; Linux couldn't see it at all.<br>&gt; <br=
>&gt; There were two hot spares, but it didn't use them. Worse, there was n=
othing I could do to make it do anything. Every command reported "Failed" a=
nd no further explanation. Booting into the RAID BIOS gave the same problem=
: if I selected "rebuild" or "verify", it said "You must select an array...=
" even though I had selected the array. It was as though the array didn't e=
xist, yet it was shown.<br>&gt; <br>&gt; I shut off the computer, unplugged=
 the BBU from the RAID card and plugged it back in, unplugged and reinserte=
d all the SATA cables, and then restarted. Exact same symptoms.<br>&gt; <br=
>&gt; I finally gave up trying to recover the database (we had a backup ser=
ver). The RAID controller let me delete and recreate the degraded array, an=
d now everything seems fine. I can rebuild the Postgres database on the new=
 unit. But I've lost a HUGE amount of trust in the LSI 9650-SE RAID control=
ler card.<br>&gt; <br>&gt; Thanks,<br>&gt; Craig<br>&gt; <br><div><br></div=
><br>-- <br>Sent via pgsql-admin mailing list (pgsql-admin@postgresql.org)<=
br>To make changes to your subscription:<br>http://www.postgresql.org/mailp=
ref/pgsql-admin<br></blockquote><div><br></div></div></body></html>
------=_Part_760182_208238390.1407255260651--