pg.ddx.io  pgsql-admin@postgresql.org mailing list archive  
help / color / mirror / Atom feed
From: Alex Turner <armtuk@gmail.com>
To: Tom Lane <tgl@sss.pgh.pa.us>
Cc: pgsql-admin@postgresql.org, Alex Turner <aturner@neteconomist.com>
Subject: Re: IO Timeout
Date: Fri, 11 Mar 2005 09:47:31 -0500
Message-ID: <33c6269f050311064772fac861@mail.gmail.com> (raw)
In-Reply-To: <28300.1110514147@sss.pgh.pa.us>
References: <33c6269f050310111892ae175@mail.gmail.com>
	<27055.1110499469@sss.pgh.pa.us>
	<33c6269f05031018197ae6206b@mail.gmail.com>
	<28300.1110514147@sss.pgh.pa.us>

Thanks very much Tom for you input - 

The guys at AMCC are suggesting that the firmware on the controller
card crashed, causing the card to basicaly stop IO operations. This
would explain why postgres could not recover and re-read WAL, because
/dev/sdc and sdd were inaccessible at that time.

I think this puzzle is mostly solved - all we need to do now is
figured out what the heck happened on the controller card!

Thanks,

Alex Turner


On Thu, 10 Mar 2005 23:09:07 -0500, Tom Lane <tgl@sss.pgh.pa.us> wrote:
> Alex Turner <armtuk@gmail.com> writes:
> > Well - I am sort of trying to piece together exactly what happened.
> > Here's what I know.
> 
> > Around 02:52 I get messages in my syslog stating that there were
> > problems writing to a controler channel:
> > [ various hardware errors snipped ]
> 
> > At around 07:30 all connections were failing giving the error:
> > InternalError: FATAL:  the database system is in recovery mode
> 
> I think what happened here is that Postgres got a write error on WAL,
> which would probably cause a PANIC, and then the ensuing database reboot
> got hung up trying to re-read WAL.  Client connection requests would be
> refused with messages like the above until the recovery process
> completed.  The fact that this was still going on 4+ hours later shows
> that Postgres is *not* timing out on stuck disk operations ... very much
> the reverse in fact.
> 
> You'd be best off to take the matter up with some kernel hackers.
> If there's anything to be done to improve the behavior, it's at
> the kernel device driver level.
> 
>                         regards, tom lane
> 
>



view thread (7+ messages)

Message-ID: <33c6269f050311064772fac861@mail.gmail.com>
Permalink:  ../33c6269f050311064772fac861@mail.gmail.com/
Also on:    postgresql.org/message-id/33c6269f050311064772fac861@mail.gmail.com

 · 

reply

Reply instructions:

You may reply publicly to this message via plain-text email
using any one of the following methods:

* Reply to all the recipients from your mail client; the link fills in
  the headers below: reply via email
  Some clients, webmail especially, drop In-Reply-To and References from
  a link, and the reply then starts a new thread: use the mbox below.

  To: pgsql-admin@postgresql.org
  Cc: armtuk@gmail.com, tgl@sss.pgh.pa.us, aturner@neteconomist.com
  Subject: Re: IO Timeout
  In-Reply-To: <33c6269f050311064772fac861@mail.gmail.com>
  References: <33c6269f050310111892ae175@mail.gmail.com> <27055.1110499469@sss.pgh.pa.us> <33c6269f05031018197ae6206b@mail.gmail.com> <28300.1110514147@sss.pgh.pa.us> <33c6269f050311064772fac861@mail.gmail.com>

* Save the following mbox file, import it into your mail client,
  and reply-to-all from there: mbox

This inbox is served by DDX for PostgreSQL; see mirroring instructions
for how to clone and mirror all data and code used for this inbox