Received: from localhost (maia-2.hub.org [200.46.204.187]) by postgresql.org (Postfix) with ESMTP id 622839FB828 for ; Mon, 18 Jun 2007 17:55:52 -0300 (ADT) Received: from postgresql.org ([200.46.204.71]) by localhost (mx1.hub.org [200.46.204.187]) (amavisd-maia, port 10024) with ESMTP id 99243-02 for ; Mon, 18 Jun 2007 17:55:44 -0300 (ADT) X-Greylist: from auto-whitelisted by SQLgrey-1.7.5 Received: from mail01.enterprisedb.com (mail01.enterprisedb.com [63.246.7.168]) by postgresql.org (Postfix) with ESMTP id EFB5D9FB6F4 for ; Mon, 18 Jun 2007 17:55:47 -0300 (ADT) thread-index: Acex6xRuNU3PEtU+R1Kes9shWmBCdA== Received: from [192.168.0.4] ([85.211.203.94]) by mail01.enterprisedb.com with Microsoft SMTPSVC(6.0.3790.3959); Mon, 18 Jun 2007 16:56:00 -0400 Subject: Re: Explained by known hardware failures, or keep looking? From: "Simon Riggs" Content-Class: urn:content-classes:message Importance: normal X-MimeOLE: Produced By Microsoft MimeOLE V6.00.3790.4073 To: "Kevin Grittner" Cc: In-Reply-To: <467699B7.EE98.0025.0@wicourts.gov> References: <467699B7.EE98.0025.0@wicourts.gov> Content-Type: text/plain; charset="iso-8859-1" Date: Mon, 18 Jun 2007 21:52:53 +0100 Message-ID: <1182199973.6855.279.camel@silverbirch.site> MIME-Version: 1.0 X-Mailer: Evolution 2.6.0 Content-Transfer-Encoding: 7bit X-OriginalArrivalTime: 18 Jun 2007 20:56:00.0640 (UTC) FILETIME=[13871800:01C7B1EB] X-Virus-Scanned: Maia Mailguard 1.0.1 X-Archive-Number: 200706/154 X-Sequence-Number: 25822 On Mon, 2007-06-18 at 14:41 -0500, Kevin Grittner wrote: > [2007-06-14 11:31:05.986 CDT] 6781 LOG: redo starts at 1D2/6C739064 > [2007-06-14 11:31:46.533 CDT] 6781 WARNING: invalid page header in block 182566 of relation "1523860"; zeroing out page > [2007-06-14 11:31:46.533 CDT] 6781 CONTEXT: xlog redo split_r: rel 1663/16386/1523860; tid 182566/92; oth 182563; rgh 115741 > [2007-06-14 11:31:56.228 CDT] 6781 WARNING: invalid page header in block 182567 of relation "1523860"; zeroing out page > [2007-06-14 11:31:56.229 CDT] 6781 CONTEXT: xlog redo split_r: rel 1663/16386/1523860; tid 182567/94; oth 182128; rgh 114655 > [2007-06-14 11:32:04.964 CDT] 6781 WARNING: invalid page header in block 123644 of relation "1524189"; zeroing out page > [2007-06-14 11:32:04.964 CDT] 6781 CONTEXT: xlog redo split_r: rel 1663/16386/1524189; tid 123644/101; oth 123634; rgh 106665 > [2007-06-14 11:32:11.327 CDT] 6781 WARNING: invalid page header in block 356562 of relation "1524219"; zeroing out page > [2007-06-14 11:32:11.327 CDT] 6781 CONTEXT: xlog redo split_r: rel 1663/16386/1524219; tid 356562/58; oth 356549; rgh 34892 > [2007-06-14 11:32:14.795 CDT] 6781 LOG: record with zero length at 1D2/70C31890 > [2007-06-14 11:32:14.795 CDT] 6781 LOG: redo done at 1D2/70C31868 > [2007-06-14 11:32:33.833 CDT] 6781 LOG: database system is ready I can potentially believe that this could be caused by blocks that were written to, but not yet flushed at checkpoint. This could happen if the blocks were reasonably heavily used, say as right-edge of index for two connected tables at time of crash. I've got no diagnostics to back that up, however detailed the logs look. Other explanations welcome. > Could all of this be reasonably explained by the controller failure and/or the subsequent abrupt power loss, or should I be looking for another cause? Personally, as I look at this, I'm suspicious that either the controller didn't persist dirty pages in the June 14th failure or there is some ongoing hardware problem. Yes. The controller failure means data loss. PostgreSQL doesn't have a disk check utility because your data is never at risk from us when running with full transaction guarantees (ref new feature in 8,3), but the disk failure has meant stuff you thought was on disk wasn't really. So your DB has holes in it and you need to recover/failover/pull-hair. -- Simon Riggs EnterpriseDB http://www.enterprisedb.com