X-Original-To: pgsql-admin-postgresql.org@localhost.postgresql.org Received: from localhost (unknown [200.46.204.144]) by svr1.postgresql.org (Postfix) with ESMTP id 5D94952A4D for ; Fri, 11 Mar 2005 04:09:29 +0000 (GMT) Received: from svr1.postgresql.org ([200.46.204.71]) by localhost (av.hub.org [200.46.204.144]) (amavisd-new, port 10024) with ESMTP id 70375-03 for ; Fri, 11 Mar 2005 04:09:20 +0000 (GMT) Received: from sss.pgh.pa.us (sss.pgh.pa.us [66.207.139.130]) by svr1.postgresql.org (Postfix) with ESMTP id 8837E554DD for ; Fri, 11 Mar 2005 04:09:20 +0000 (GMT) Received: from sss2.sss.pgh.pa.us (tgl@localhost [127.0.0.1]) by sss.pgh.pa.us (8.13.1/8.13.1) with ESMTP id j2B497dK028301; Thu, 10 Mar 2005 23:09:07 -0500 (EST) To: Alex Turner Cc: pgsql-admin@postgresql.org, Alex Turner Subject: Re: IO Timeout In-reply-to: <33c6269f05031018197ae6206b@mail.gmail.com> References: <33c6269f050310111892ae175@mail.gmail.com> <27055.1110499469@sss.pgh.pa.us> <33c6269f05031018197ae6206b@mail.gmail.com> Comments: In-reply-to Alex Turner message dated "Thu, 10 Mar 2005 21:19:19 -0500" Date: Thu, 10 Mar 2005 23:09:07 -0500 Message-ID: <28300.1110514147@sss.pgh.pa.us> From: Tom Lane X-Virus-Scanned: by amavisd-new at hub.org X-Spam-Status: No, hits=0.007 tagged_above=0 required=5 tests=AWL X-Spam-Level: X-Archive-Number: 200503/169 X-Sequence-Number: 16667 Alex Turner writes: > Well - I am sort of trying to piece together exactly what happened. > Here's what I know. > Around 02:52 I get messages in my syslog stating that there were > problems writing to a controler channel: > [ various hardware errors snipped ] > At around 07:30 all connections were failing giving the error: > InternalError: FATAL: the database system is in recovery mode I think what happened here is that Postgres got a write error on WAL, which would probably cause a PANIC, and then the ensuing database reboot got hung up trying to re-read WAL. Client connection requests would be refused with messages like the above until the recovery process completed. The fact that this was still going on 4+ hours later shows that Postgres is *not* timing out on stuck disk operations ... very much the reverse in fact. You'd be best off to take the matter up with some kernel hackers. If there's anything to be done to improve the behavior, it's at the kernel device driver level. regards, tom lane