X-Original-To: pgsql-admin-postgresql.org@localhost.postgresql.org Received: from localhost (unknown [200.46.204.144]) by svr1.postgresql.org (Postfix) with ESMTP id 50E62565D9 for ; Fri, 11 Mar 2005 14:47:53 +0000 (GMT) Received: from svr1.postgresql.org ([200.46.204.71]) by localhost (av.hub.org [200.46.204.144]) (amavisd-new, port 10024) with ESMTP id 53511-02 for ; Fri, 11 Mar 2005 14:47:44 +0000 (GMT) Received: from wproxy.gmail.com (wproxy.gmail.com [64.233.184.193]) by svr1.postgresql.org (Postfix) with ESMTP id 87FE25659E for ; Fri, 11 Mar 2005 14:47:43 +0000 (GMT) Received: by wproxy.gmail.com with SMTP id 70so971869wra for ; Fri, 11 Mar 2005 06:47:47 -0800 (PST) DomainKey-Signature: a=rsa-sha1; q=dns; c=nofws; s=beta; d=gmail.com; h=received:message-id:date:from:reply-to:to:subject:cc:in-reply-to:mime-version:content-type:content-transfer-encoding:references; b=sSWveRKxXYhmxzh/63FFdj3DxwCyj6v4KJUMswakJUp2icPc6BaOYdvT2J0+4ARE8mlfewzrfAYlegmeolMeZ9Bz6EdZMavSYicjGq4Mxm4Qs7otOHiR/L5x2sGPNHO/d9+0O+qmHsR0c+/kFtLasW/q4uiYPZezWk7UpCoC0kU= Received: by 10.54.40.2 with SMTP id n2mr2497178wrn; Fri, 11 Mar 2005 06:47:32 -0800 (PST) Received: by 10.54.18.53 with HTTP; Fri, 11 Mar 2005 06:47:31 -0800 (PST) Message-ID: <33c6269f050311064772fac861@mail.gmail.com> Date: Fri, 11 Mar 2005 09:47:31 -0500 From: Alex Turner Reply-To: Alex Turner To: Tom Lane Subject: Re: IO Timeout Cc: pgsql-admin@postgresql.org, Alex Turner In-Reply-To: <28300.1110514147@sss.pgh.pa.us> Mime-Version: 1.0 Content-Type: text/plain; charset=US-ASCII Content-Transfer-Encoding: 7bit References: <33c6269f050310111892ae175@mail.gmail.com> <27055.1110499469@sss.pgh.pa.us> <33c6269f05031018197ae6206b@mail.gmail.com> <28300.1110514147@sss.pgh.pa.us> X-Virus-Scanned: by amavisd-new at hub.org X-Spam-Status: No, hits=0.224 tagged_above=0 required=5 tests=AWL, DNS_FROM_RFC_ABUSE, RCVD_BY_IP X-Spam-Level: X-Archive-Number: 200503/178 X-Sequence-Number: 16676 Thanks very much Tom for you input - The guys at AMCC are suggesting that the firmware on the controller card crashed, causing the card to basicaly stop IO operations. This would explain why postgres could not recover and re-read WAL, because /dev/sdc and sdd were inaccessible at that time. I think this puzzle is mostly solved - all we need to do now is figured out what the heck happened on the controller card! Thanks, Alex Turner On Thu, 10 Mar 2005 23:09:07 -0500, Tom Lane wrote: > Alex Turner writes: > > Well - I am sort of trying to piece together exactly what happened. > > Here's what I know. > > > Around 02:52 I get messages in my syslog stating that there were > > problems writing to a controler channel: > > [ various hardware errors snipped ] > > > At around 07:30 all connections were failing giving the error: > > InternalError: FATAL: the database system is in recovery mode > > I think what happened here is that Postgres got a write error on WAL, > which would probably cause a PANIC, and then the ensuing database reboot > got hung up trying to re-read WAL. Client connection requests would be > refused with messages like the above until the recovery process > completed. The fact that this was still going on 4+ hours later shows > that Postgres is *not* timing out on stuck disk operations ... very much > the reverse in fact. > > You'd be best off to take the matter up with some kernel hackers. > If there's anything to be done to improve the behavior, it's at > the kernel device driver level. > > regards, tom lane > >