From: Sean Chittenden <sean@gigave.com>
To: Tom Lane <tgl@sss.pgh.pa.us>
Cc: pgsql-admin@postgresql.org
Subject: Re: Corrupted data, best course of repair?
Date: Mon, 22 Aug 2005 10:46:06 -0700
Message-ID: <20050822174606.GA46390@sean.gigave.com> (raw)
In-Reply-To: <15945.1124722814@sss.pgh.pa.us>
References: <20050822065149.GD71159@sean.gigave.com>
<15945.1124722814@sss.pgh.pa.us>
> They run with fsync off AND they like to toggle the power switch at
> random? I'd suggest finding other employment --- they couldn't
> possibly be paying you enough to justify cleaning up after stupidity
> as gross as that.
Colo-by-windows. If there weren't DBAs with Win32 admin tendencies,
I'd be out of work. :)
> Anyway, the errors appear to indicate that there are pages in the
> database with LSN (last WAL location) larger than the actual current
> end of WAL. The difference is pretty large though --- at least 85MB
> of WAL seems to have gone missing. My first thought was a corrupted
> LSN field. But seeing that there are at least two such pages, and
> given the antics you describe above, what seems more likely is that
> the LSNs were correct when written. I think some page of WAL never
> made it to disk during a period of heavy updates that was terminated
> by a power cycle, and during replay we stopped at the first point
> where the WAL data was detectably corrupt, and so a big chunk of WAL
> never got replayed. Which of course means there's probably a lot of
> stuff that needs to be fixed and did not get fixed, but in
> particular our idea of the current end-of-WAL address is a lot less
> than it should be. If you have the server log from just after the
> last postmaster restart, looking at what terminated the replay might
> confirm this.
Peachy.
> You could get the DB to stop complaining by doing a pg_resetxlog to
> push the WAL start address above the largest "flush request"
> mentioned in any of the messages. But my guess is that you'll find
> a lot of internal corruption after you do it. Going back to the
> dump might be a saner way to proceed.
Tons of corruption and a backup that's a few weeks old. *grin* The
most recent dump seems to have all of the data, but some rows are
there in duplicate. Thanks for the input. -sc
--
Sean Chittenden
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Reply to all the recipients from your mail client; the link fills in
the headers below: reply via email
Some clients, webmail especially, drop In-Reply-To and References from
a link, and the reply then starts a new thread: use the mbox below.
To: pgsql-admin@postgresql.org
Cc: sean@gigave.com, tgl@sss.pgh.pa.us
Subject: Re: Corrupted data, best course of repair?
In-Reply-To: <20050822174606.GA46390@sean.gigave.com>
References: <20050822065149.GD71159@sean.gigave.com> <15945.1124722814@sss.pgh.pa.us> <20050822174606.GA46390@sean.gigave.com>
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
This inbox is served by DDX for PostgreSQL; see mirroring instructions
for how to clone and mirror all data and code used for this inbox