Received: from maia.hub.org (maia-2.hub.org [200.46.204.251]) by mail.postgresql.org (Postfix) with ESMTP id 63F41B5DBC4 for ; Sat, 16 Jul 2011 18:30:44 -0300 (ADT) Received: from mail.postgresql.org ([200.46.204.86]) by maia.hub.org (mx1.hub.org [200.46.204.251]) (amavisd-maia, port 10024) with ESMTP id 21504-09 for ; Sat, 16 Jul 2011 21:30:33 +0000 (UTC) X-Greylist: from auto-whitelisted by SQLgrey-1.7.6 Received: from sss.pgh.pa.us (sss.pgh.pa.us [66.207.139.130]) by mail.postgresql.org (Postfix) with ESMTP id 23A6DB5DBBE for ; Sat, 16 Jul 2011 18:30:33 -0300 (ADT) Received: from sss2.sss.pgh.pa.us (tgl@localhost [127.0.0.1]) by sss.pgh.pa.us (8.14.2/8.14.2) with ESMTP id p6GLUUNO023809; Sat, 16 Jul 2011 17:30:31 -0400 (EDT) To: Ken Caruso cc: "pgsql-admin@postgresql.org" Subject: Re: 9.0.4 Data corruption issue In-reply-to: References: <27386.1310831992@sss.pgh.pa.us> Comments: In-reply-to Ken Caruso message dated "Sat, 16 Jul 2011 12:42:47 -0700" Date: Sat, 16 Jul 2011 17:30:30 -0400 Message-ID: <23808.1310851830@sss.pgh.pa.us> From: Tom Lane X-Virus-Scanned: Maia Mailguard 1.0.1 X-Spam-Status: No, hits=-1.899 tagged_above=-10 required=5 tests=BAYES_00=-1.9, RP_MATCHES_RCVD=0.001 X-Spam-Level: X-Archive-Number: 201107/124 X-Sequence-Number: 37903 Ken Caruso writes: > Sorry, the actual error reported by CLUSTER is: > gpup=> cluster verbose tablename; > INFO: clustering "dbname.tablename" > WARNING: could not write block 12125253 of base/2651908/652397108 > DETAIL: Multiple failures --- write error might be permanent. > ERROR: could not open file "base/2651908/652397108.1" (target block > 12125253): No such file or directory > CONTEXT: writing block 12125253 of relation base/2651908/652397108 Hmm ... it looks like you've got a dirty buffer in shared memory that corresponds to a block that no longer exists on disk; in fact, the whole table segment it belonged to is gone. Or maybe the block or file number in the shared buffer header is corrupted somehow. I imagine you're seeing errors like this during each checkpoint attempt? I can't think of any very good way to clean that up. What I'd try here is a forced database shutdown (immediate-mode stop) and see if it starts up cleanly. It might be that whatever caused this has also corrupted the back WAL and so WAL replay will result in the same or similar error. In that case you'll be forced to do a pg_resetxlog to get the DB to come up again. If so, a dump and reload and some manual consistency checking would be indicated :-( regards, tom lane