From: David Haube <dch@frolix.muddywaters.org>
To: pgsql-admin@postgresql.org
Subject: Database corruption
Date: Thu, 24 Jun 2010 11:28:31 -0700
Message-ID: <4C23A3CF.4080506@frolix.muddywaters.org> (raw)
Postgres version 8.4.4
Hardware:
12 cpu intel
sda 15K 600GB raid 1
sdb ssd 173GB raid 1 (most tables and indexes on sdb)
adaptec controller
24G of memory
shared buffers 12GB
Machine age 3 months
Normal load 2-4
200-300 Transactions per second
We had a database failure last night after one of the tables had a
corrupted block. After we noticed the corruption all available memory
was used up plus swap. The database died (or rather killed by the
kernel) with an out of memory error. We switched to the warm standby
which doesn't have any corruption.
On the postmortem I found 4 tables with corruption. Only thing that
links these tables was there was autovacuum (to prevent wraparound)
either running or had run on those tables. All tables are in the Gig
range or multi Gig range. The vacuum of some of the tables had been
going on for days.
The errors from the log file were in the form of :
ERROR: invalid page header in block 290125 of relation
pg_tblspc/16385/18674/205612
After an attempted vaccum we had this error:
2010-06-24 17:31:09 UTC [31766]: [36-1]WARNING: PD_ALL_VISIBLE flag was
incorrectly set in relation "org_crawl_page_scrape_result" page 128902
The first error was logged at 10:15pm
At 1 am a pg_dump was run from cron and failed after 20 minutes while
try ing to allocate an immenense amount of memory while attempting to
dump one of the corrupted tables.
At 2:00 am All memory was used up and cpu was maxed and a load average
of 56.
We transferred to the standby and rebooted the machine. At this time
the database is sitting there though we'll need to remake the database
and turn it into the warm standby.
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Reply to all the recipients using the --to and --cc options:
reply via email
To: pgsql-admin@postgresql.org
Cc: dch@frolix.muddywaters.org
Subject: Re: Database corruption
In-Reply-To: <4C23A3CF.4080506@frolix.muddywaters.org>
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
This inbox is served by DDX for PostgreSQL; see mirroring instructions
for how to clone and mirror all data and code used for this inbox