Received: from makus.postgresql.org (makus.postgresql.org [98.129.198.125]) by mail.postgresql.org (Postfix) with ESMTP id 2878E9ED582 for ; Wed, 7 Dec 2011 21:06:16 -0400 (AST) Received: from mail-bw0-f46.google.com ([209.85.214.46]) by makus.postgresql.org with esmtp (Exim 4.72) (envelope-from ) id 1RYSRC-0008Ly-TS for pgsql-admin@postgresql.org; Thu, 08 Dec 2011 01:06:15 +0000 Received: by bkbzv3 with SMTP id zv3so1115819bkb.19 for ; Wed, 07 Dec 2011 17:06:01 -0800 (PST) Received: by 10.205.132.6 with SMTP id hs6mr460118bkc.35.1323306361001; Wed, 07 Dec 2011 17:06:01 -0800 (PST) Received: from ayaki.localdomain (203-173-33-169.dyn.iinet.net.au. [203.173.33.169]) by mx.google.com with ESMTPS id jf4sm1026042bkc.5.2011.12.07.17.05.56 (version=SSLv3 cipher=OTHER); Wed, 07 Dec 2011 17:05:59 -0800 (PST) Message-ID: <4EE00D71.6090705@ringerc.id.au> Date: Thu, 08 Dec 2011 09:05:53 +0800 From: Craig Ringer User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:7.0.1) Gecko/20110930 Thunderbird/7.0.1 MIME-Version: 1.0 To: Kevin Grittner CC: pgsql-admin@postgresql.org, jkells Subject: Re: bad block problem References: <1OQDq.131420$034.90105@en-nntp-06.dc1.easynews.com> <4EDFA52F0200002500043A8A@gw.wicourts.gov> In-Reply-To: <4EDFA52F0200002500043A8A@gw.wicourts.gov> Content-Type: text/plain; charset=ISO-8859-1; format=flowed Content-Transfer-Encoding: 7bit X-Pg-Spam-Score: -1.9 (-) X-Archive-Number: 201112/58 X-Sequence-Number: 38816 On 12/08/2011 07:41 AM, Kevin Grittner wrote: > That sounds like your storage system is failing, quite independently > from PostgreSQL. Copy the entire data directory tree to some other > medium immediately, and preserve this copy. If you hit bad blocks, > retry if possible. If you find files you can't copy in their entirety, try using dd_rescue to copy it with a hole for the bad block. dd_rescue is an _incredibly_ useful tool for this, as it'll do bad-block-tolerant copies quickly and efficiently. Once you have a complete copy of your datadir, stop working on the faulty machine. Make your first copy read-only. Duplicate the copy and work on the duplicate when trying to restore. I'd start with enabling zero_damaged_pages to see if you can get a dump that way. Do **NOT** enable zero_damaged_pages on the original. Do it on the duplicate of the copied data. -- Craig Ringer