pg.ddx.io  pgsql-admin@postgresql.org mailing list archive  
help / color / mirror / Atom feed
From: Thomas F. O'Connell <tfo@sitening.com>
To: pgsql-admin@postgresql.org
Subject: Invalid Page Headers
Date: Tue, 18 Apr 2006 12:30:36 -0500
Message-ID: <B8F30540-C0A5-4CF7-8DB2-867DFF25E58D@sitening.com> (raw)

I've got a database that reported this error yesterday morning after  
an UPDATE statement:

ERROR:  invalid page header in block 34 of relation "pg_toast_167245"

Later in the day, it reported this one:

ERROR:  invalid page header in block 199 of relation "<table1>_pkey"

this time after a SELECT.

Before these errors had been brought to my attention, I was looking  
late that night at a long-running query on an unrelated table. There  
was a query that should've finished in seconds that was taking hours.  
I later heard from a developer that there were some deadlock-related  
issues with the query in question, and the development team wound up  
issuing:

pg_ctl kill QUIT [process_id]

to get rid of the problem query this morning, which was a relatively  
unimportant SELECT.

Then, shortly thereafter, they began seeing yet another invalid page  
header:

invalid page header in block 4369 of relation "<table2>"

So there are currently three separate relations exhibiting invalid  
page errors.

This box is a Debian 3.1 box running a custom Linux 2.6.10 #6 SMP  
kernel. Postgres 8.1.3 was compiled from source. pgpool 3.0.1, also  
built from source, is used by some parts of the application layer.  
The system is running on an ext3 filesystem, WAL is on a 4-disk RAID  
10 running JFS, and data is on a 12-disk RAID 10 running JFS. I'm not  
seeing any signs of apparent kernel or hardware errors in the system  
and kernel logs.

Also see nearby thread of a troubling error from the night before the  
first sight of invalid page headers:

http://archives.postgresql.org/pgsql-general/2006-04/msg00746.php

Is there any way in which this could be related to the invalid page  
headers?

Based on this thread:

http://archives.postgresql.org/pgsql-general/2005-11/msg01140.php

I'm a little nervous about the prospects for analysis and recovery  
here. Any thoughts?

Is there a risk that if we took postgres offline in this state that  
it would not come back up?

--
Thomas F. O'Connell
Database Architecture and Programming
Sitening, LLC

http://www.sitening.com/
3004 B Poston Avenue
Nashville, TN 37203-1314
615-260-0005 (cell)
615-469-5150 (office)
615-469-5151 (fax)




view thread (10+ messages)  latest in thread

Message-ID: <B8F30540-C0A5-4CF7-8DB2-867DFF25E58D@sitening.com>
Permalink:  ../B8F30540-C0A5-4CF7-8DB2-867DFF25E58D@sitening.com/
Also on:    postgresql.org/message-id/B8F30540-C0A5-4CF7-8DB2-867DFF25E58D@sitening.com

 · 

reply

Reply instructions:

You may reply publicly to this message via plain-text email
using any one of the following methods:

* Reply to all the recipients from your mail client; the link fills in
  the headers below: reply via email
  Some clients, webmail especially, drop In-Reply-To and References from
  a link, and the reply then starts a new thread: use the mbox below.

  To: pgsql-admin@postgresql.org
  Cc: tfo@sitening.com
  Subject: Re: Invalid Page Headers
  In-Reply-To: <B8F30540-C0A5-4CF7-8DB2-867DFF25E58D@sitening.com>
  References: <B8F30540-C0A5-4CF7-8DB2-867DFF25E58D@sitening.com>

* Save the following mbox file, import it into your mail client,
  and reply-to-all from there: mbox

This inbox is served by DDX for PostgreSQL; see mirroring instructions
for how to clone and mirror all data and code used for this inbox