From: Alex Turner <armtuk@gmail.com>
To: Tom Lane <tgl@sss.pgh.pa.us>
Cc: pgsql-admin@postgresql.org, Alex Turner <aturner@neteconomist.com>
Subject: Re: IO Timeout
Date: Thu, 10 Mar 2005 21:19:19 -0500
Message-ID: <33c6269f05031018197ae6206b@mail.gmail.com> (raw)
In-Reply-To: <27055.1110499469@sss.pgh.pa.us>
References: <33c6269f050310111892ae175@mail.gmail.com>
<27055.1110499469@sss.pgh.pa.us>
Well - I am sort of trying to piece together exactly what happened.
Here's what I know.
Around 02:52 I get messages in my syslog stating that there were
problems writing to a controler channel:
Mar 10 02:52:29 tsunami kernel: 3w-9xxx: scsi1: WARNING:
(0x06:0x002C): Unit #1: Command (0x28) timed out, resetting card.
Mar 10 02:52:29 tsunami kernel: 3w-9xxx: scsi1: ERROR: (0x06:0x001F):
Microcontroller not ready during reset sequence.
Mar 10 02:52:29 tsunami kernel: 3w-9xxx: scsi1: AEN: INFO
(0x04:0x005E): <NULL>:unit=0.
...
Mar 10 02:58:41 tsunami kernel: 3w-9xxx: scsi1: ERROR: (0x03:0x0107):
Duplicate request ID:RequestID=23.
Mar 10 02:58:41 tsunami kernel: end_request: I/O error, dev sdd,
sector 282528903
Mar 10 02:58:41 tsunami kernel: 3w-9xxx: scsi1: ERROR: (0x03:0x0107):
Duplicate request ID:RequestID=23.
Mar 10 02:58:41 tsunami kernel: end_request: I/O error, dev sdd,
sector 282528903
Mar 10 02:58:41 tsunami kernel: 3w-9xxx: scsi1: ERROR: (0x03:0x0107):
Duplicate request ID:RequestID=23.
Mar 10 02:58:41 tsunami kernel: end_request: I/O error, dev sdc, sector 28837057
At around 07:30 all connections were failing giving the error:
InternalError: FATAL: the database system is in recovery mode
(pygresql - similar error in PHP also)
I reboot the server, and one of the discs comes up as innaccesible
(it's part of a RAID 10), but other than that, everything restarts as
normal.
Nothing significant in /var/log/pg_log which is where I have it
logging to (my log leve is pretty low though).
Alex Turner
netEconomist
On Thu, 10 Mar 2005 19:04:29 -0500, Tom Lane <tgl@sss.pgh.pa.us> wrote:
> Alex Turner <armtuk@gmail.com> writes:
> > I have a question about IO timeouts:
> > We are using the 3ware escalade 9500S series of cards, and we had a
> > drive failure this morning. Apparnetly the card waits 30 seconds for
> > the drive to respond, and if it doesn't, it put's the drive in a fail
> > state. Postgres it seems didn't wait 30 seconds before it decided
> > that the system was upset, and put the database in maintainence mode.
>
> > Is there a way to increase to IO wait timeout so this doesn't happen?
>
> Postgres hasn't got any "IO timeouts". Your concern would be better
> directed to whatever kernel you're using; any sort of timeout on a disk
> operation would be happening at the kernel level.
>
> For that matter, Postgres hasn't got any concept of "putting the
> database in maintainence mode", so you haven't described what happened
> very accurately at all.
>
> regards, tom lane
>
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Reply to all the recipients from your mail client; the link fills in
the headers below: reply via email
Some clients, webmail especially, drop In-Reply-To and References from
a link, and the reply then starts a new thread: use the mbox below.
To: pgsql-admin@postgresql.org
Cc: armtuk@gmail.com, tgl@sss.pgh.pa.us, aturner@neteconomist.com
Subject: Re: IO Timeout
In-Reply-To: <33c6269f05031018197ae6206b@mail.gmail.com>
References: <33c6269f050310111892ae175@mail.gmail.com> <27055.1110499469@sss.pgh.pa.us> <33c6269f05031018197ae6206b@mail.gmail.com>
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
This inbox is served by DDX for PostgreSQL; see mirroring instructions
for how to clone and mirror all data and code used for this inbox