X-Original-To: pgsql-admin-postgresql.org@localhost.postgresql.org Received: from localhost (mx2.hub.org [200.46.204.254]) by postgresql.org (Postfix) with ESMTP id 2717311F624F for ; Tue, 18 Apr 2006 14:30:50 -0300 (ADT) X-Virus-Scanned: by amavisd-new at hub.org X-Spam-Score: -2.499 X-Spam-Level: X-Spam-Status: No, score=-2.499 required=5 tests=[AWL=0.100, BAYES_00=-2.599] Received: from postgresql.org ([200.46.204.71]) by localhost (mx2.hub.org [200.46.204.254]) (amavisd-new, port 10024) with ESMTP id qrWqn9OJuaqv for ; Tue, 18 Apr 2006 14:30:44 -0300 (ADT) X-Greylist: from auto-whitelisted by SQLgrey- Received: from chilco.textdrive.com (chilco.textdrive.com [207.7.108.242]) by postgresql.org (Postfix) with ESMTP id DCCDC11F6238 for ; Tue, 18 Apr 2006 14:30:43 -0300 (ADT) Received: from [192.168.1.109] (unknown [209.149.57.250]) by chilco.textdrive.com (Postfix) with ESMTP id 14622DBCDC for ; Tue, 18 Apr 2006 17:31:54 +0000 (UTC) Mime-Version: 1.0 (Apple Message framework v749.3) Content-Transfer-Encoding: 7bit Message-Id: Content-Type: text/plain; charset=US-ASCII; delsp=yes; format=flowed To: pgsql-admin@postgresql.org From: "Thomas F. O'Connell" Subject: Invalid Page Headers Date: Tue, 18 Apr 2006 12:30:36 -0500 X-Mailer: Apple Mail (2.749.3) X-Archive-Number: 200604/166 X-Sequence-Number: 21335 I've got a database that reported this error yesterday morning after an UPDATE statement: ERROR: invalid page header in block 34 of relation "pg_toast_167245" Later in the day, it reported this one: ERROR: invalid page header in block 199 of relation "_pkey" this time after a SELECT. Before these errors had been brought to my attention, I was looking late that night at a long-running query on an unrelated table. There was a query that should've finished in seconds that was taking hours. I later heard from a developer that there were some deadlock-related issues with the query in question, and the development team wound up issuing: pg_ctl kill QUIT [process_id] to get rid of the problem query this morning, which was a relatively unimportant SELECT. Then, shortly thereafter, they began seeing yet another invalid page header: invalid page header in block 4369 of relation "" So there are currently three separate relations exhibiting invalid page errors. This box is a Debian 3.1 box running a custom Linux 2.6.10 #6 SMP kernel. Postgres 8.1.3 was compiled from source. pgpool 3.0.1, also built from source, is used by some parts of the application layer. The system is running on an ext3 filesystem, WAL is on a 4-disk RAID 10 running JFS, and data is on a 12-disk RAID 10 running JFS. I'm not seeing any signs of apparent kernel or hardware errors in the system and kernel logs. Also see nearby thread of a troubling error from the night before the first sight of invalid page headers: http://archives.postgresql.org/pgsql-general/2006-04/msg00746.php Is there any way in which this could be related to the invalid page headers? Based on this thread: http://archives.postgresql.org/pgsql-general/2005-11/msg01140.php I'm a little nervous about the prospects for analysis and recovery here. Any thoughts? Is there a risk that if we took postgres offline in this state that it would not come back up? -- Thomas F. O'Connell Database Architecture and Programming Sitening, LLC http://www.sitening.com/ 3004 B Poston Avenue Nashville, TN 37203-1314 615-260-0005 (cell) 615-469-5150 (office) 615-469-5151 (fax)