From: Thomas F. O'Connell <tfo@sitening.com>
To: pgsql-admin@postgresql.org
Subject: Invalid Page Headers
Date: Tue, 18 Apr 2006 12:30:36 -0500
Message-ID: <B8F30540-C0A5-4CF7-8DB2-867DFF25E58D@sitening.com> (raw)
I've got a database that reported this error yesterday morning after
an UPDATE statement:
ERROR: invalid page header in block 34 of relation "pg_toast_167245"
Later in the day, it reported this one:
ERROR: invalid page header in block 199 of relation "<table1>_pkey"
this time after a SELECT.
Before these errors had been brought to my attention, I was looking
late that night at a long-running query on an unrelated table. There
was a query that should've finished in seconds that was taking hours.
I later heard from a developer that there were some deadlock-related
issues with the query in question, and the development team wound up
issuing:
pg_ctl kill QUIT [process_id]
to get rid of the problem query this morning, which was a relatively
unimportant SELECT.
Then, shortly thereafter, they began seeing yet another invalid page
header:
invalid page header in block 4369 of relation "<table2>"
So there are currently three separate relations exhibiting invalid
page errors.
This box is a Debian 3.1 box running a custom Linux 2.6.10 #6 SMP
kernel. Postgres 8.1.3 was compiled from source. pgpool 3.0.1, also
built from source, is used by some parts of the application layer.
The system is running on an ext3 filesystem, WAL is on a 4-disk RAID
10 running JFS, and data is on a 12-disk RAID 10 running JFS. I'm not
seeing any signs of apparent kernel or hardware errors in the system
and kernel logs.
Also see nearby thread of a troubling error from the night before the
first sight of invalid page headers:
http://archives.postgresql.org/pgsql-general/2006-04/msg00746.php
Is there any way in which this could be related to the invalid page
headers?
Based on this thread:
http://archives.postgresql.org/pgsql-general/2005-11/msg01140.php
I'm a little nervous about the prospects for analysis and recovery
here. Any thoughts?
Is there a risk that if we took postgres offline in this state that
it would not come back up?
--
Thomas F. O'Connell
Database Architecture and Programming
Sitening, LLC
http://www.sitening.com/
3004 B Poston Avenue
Nashville, TN 37203-1314
615-260-0005 (cell)
615-469-5150 (office)
615-469-5151 (fax)
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Reply to all the recipients from your mail client; the link fills in
the headers below: reply via email
Some clients, webmail especially, drop In-Reply-To and References from
a link, and the reply then starts a new thread: use the mbox below.
To: pgsql-admin@postgresql.org
Cc: tfo@sitening.com
Subject: Re: Invalid Page Headers
In-Reply-To: <B8F30540-C0A5-4CF7-8DB2-867DFF25E58D@sitening.com>
References: <B8F30540-C0A5-4CF7-8DB2-867DFF25E58D@sitening.com>
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
This inbox is served by DDX for PostgreSQL; see mirroring instructions
for how to clone and mirror all data and code used for this inbox