X-Original-To: pgsql-admin-postgresql.org@localhost.postgresql.org Received: from localhost (unknown [200.46.204.144]) by svr1.postgresql.org (Postfix) with ESMTP id 684645651C for ; Fri, 11 Mar 2005 02:19:25 +0000 (GMT) Received: from svr1.postgresql.org ([200.46.204.71]) by localhost (av.hub.org [200.46.204.144]) (amavisd-new, port 10024) with ESMTP id 41260-02 for ; Fri, 11 Mar 2005 02:19:16 +0000 (GMT) Received: from wproxy.gmail.com (wproxy.gmail.com [64.233.184.198]) by svr1.postgresql.org (Postfix) with ESMTP id 78CD15650F for ; Fri, 11 Mar 2005 02:19:15 +0000 (GMT) Received: by wproxy.gmail.com with SMTP id 68so854973wri for ; Thu, 10 Mar 2005 18:19:20 -0800 (PST) DomainKey-Signature: a=rsa-sha1; q=dns; c=nofws; s=beta; d=gmail.com; h=received:message-id:date:from:reply-to:to:subject:cc:in-reply-to:mime-version:content-type:content-transfer-encoding:references; b=XZg6Z58++g1EuiZMIUiUjkQB5yRCqEFhSfl6Eg0r/GyE+NgBFkGBBiU9fSvjPcgNWbETMdm5OcrhIzMeGw3vXi/Xav4vMGZyxzklpytKFdwfmbN4ixPp4ZAf2q4+6eiTqVd6AYrZlPlCI7BYJUIiU8Cgz5P7PFrADkU/SrIBg8A= Received: by 10.54.10.39 with SMTP id 39mr2232201wrj; Thu, 10 Mar 2005 18:19:19 -0800 (PST) Received: by 10.54.18.53 with HTTP; Thu, 10 Mar 2005 18:19:19 -0800 (PST) Message-ID: <33c6269f05031018197ae6206b@mail.gmail.com> Date: Thu, 10 Mar 2005 21:19:19 -0500 From: Alex Turner Reply-To: Alex Turner To: Tom Lane Subject: Re: IO Timeout Cc: pgsql-admin@postgresql.org, Alex Turner In-Reply-To: <27055.1110499469@sss.pgh.pa.us> Mime-Version: 1.0 Content-Type: text/plain; charset=US-ASCII Content-Transfer-Encoding: 7bit References: <33c6269f050310111892ae175@mail.gmail.com> <27055.1110499469@sss.pgh.pa.us> X-Virus-Scanned: by amavisd-new at hub.org X-Spam-Status: No, hits=0.222 tagged_above=0 required=5 tests=AWL, DNS_FROM_RFC_ABUSE, RCVD_BY_IP X-Spam-Level: X-Archive-Number: 200503/167 X-Sequence-Number: 16665 Well - I am sort of trying to piece together exactly what happened. Here's what I know. Around 02:52 I get messages in my syslog stating that there were problems writing to a controler channel: Mar 10 02:52:29 tsunami kernel: 3w-9xxx: scsi1: WARNING: (0x06:0x002C): Unit #1: Command (0x28) timed out, resetting card. Mar 10 02:52:29 tsunami kernel: 3w-9xxx: scsi1: ERROR: (0x06:0x001F): Microcontroller not ready during reset sequence. Mar 10 02:52:29 tsunami kernel: 3w-9xxx: scsi1: AEN: INFO (0x04:0x005E): :unit=0. ... Mar 10 02:58:41 tsunami kernel: 3w-9xxx: scsi1: ERROR: (0x03:0x0107): Duplicate request ID:RequestID=23. Mar 10 02:58:41 tsunami kernel: end_request: I/O error, dev sdd, sector 282528903 Mar 10 02:58:41 tsunami kernel: 3w-9xxx: scsi1: ERROR: (0x03:0x0107): Duplicate request ID:RequestID=23. Mar 10 02:58:41 tsunami kernel: end_request: I/O error, dev sdd, sector 282528903 Mar 10 02:58:41 tsunami kernel: 3w-9xxx: scsi1: ERROR: (0x03:0x0107): Duplicate request ID:RequestID=23. Mar 10 02:58:41 tsunami kernel: end_request: I/O error, dev sdc, sector 28837057 At around 07:30 all connections were failing giving the error: InternalError: FATAL: the database system is in recovery mode (pygresql - similar error in PHP also) I reboot the server, and one of the discs comes up as innaccesible (it's part of a RAID 10), but other than that, everything restarts as normal. Nothing significant in /var/log/pg_log which is where I have it logging to (my log leve is pretty low though). Alex Turner netEconomist On Thu, 10 Mar 2005 19:04:29 -0500, Tom Lane wrote: > Alex Turner writes: > > I have a question about IO timeouts: > > We are using the 3ware escalade 9500S series of cards, and we had a > > drive failure this morning. Apparnetly the card waits 30 seconds for > > the drive to respond, and if it doesn't, it put's the drive in a fail > > state. Postgres it seems didn't wait 30 seconds before it decided > > that the system was upset, and put the database in maintainence mode. > > > Is there a way to increase to IO wait timeout so this doesn't happen? > > Postgres hasn't got any "IO timeouts". Your concern would be better > directed to whatever kernel you're using; any sort of timeout on a disk > operation would be happening at the kernel level. > > For that matter, Postgres hasn't got any concept of "putting the > database in maintainence mode", so you haven't described what happened > very accurately at all. > > regards, tom lane >