To: Nathan Robertson <nathan.robertson@gmail.com>
cc: pgsql-admin@postgresql.org
Subject: Re: cache lookup failed for index 
In-reply-to: <AANLkTilNYQhofNEmz09HaXVbULXly13xA-XOYEQMBiMT@mail.gmail.com> 
References: <AANLkTilzUWnxCjomBRHkByi6tHeS8fDIRNqvU9WCsv1G@mail.gmail.com>
	<4C2B02040200002500032D63@gw.wicourts.gov>
	<AANLkTikYlHg1mVytsX_QQWSf58MTrw-aTIklyv5Zh2KY@mail.gmail.com>
	<4C2B0AEF0200002500032D75@gw.wicourts.gov>
	<AANLkTilfWzYJTo7J39shOutT9bvM9tyxAYa1ZI1rkWeT@mail.gmail.com>
	<4C2B21A00200002500032D8D@gw.wicourts.gov>
	<11712.1277914463@sss.pgh.pa.us>
	<AANLkTim5VFISQ6IVRrtmy9dr1aRJb5fu7n_LUmTVDkak@mail.gmail.com>
	<12643.1277917499@sss.pgh.pa.us>
	<AANLkTikVuefbecakovUG6BGz-pJVFJ04us4YKH8VDQA5@mail.gmail.com>
	<15355.1277925611@sss.pgh.pa.us>
	<AANLkTimfCVD15Wy0ifNx8KRO-DCP3pDTXn9zJKRd-yCd@mail.gmail.com>
	<16649.1277929815@sss.pgh.pa.us>
	<AANLkTilNYQhofNEmz09HaXVbULXly13xA-XOYEQMBiMT@mail.gmail.com>
Comments: In-reply-to Nathan Robertson <nathan.robertson@gmail.com>
	message dated "Wed, 30 Jun 2010 16:32:15 -0400"
Date: Wed, 30 Jun 2010 16:59:24 -0400
Message-ID: <17211.1277931564@sss.pgh.pa.us>
From: Tom Lane <tgl@sss.pgh.pa.us>

Nathan Robertson <nathan.robertson@gmail.com> writes:
> -bash-3.2$ postgres -O  -P -D /tmp/database -c log_error_verbosity=verbose
> webapp
> FATAL:  XX000: cache lookup failed for index 2662
> LOCATION:  RelationInitIndexAccessInfo, relcache.c:841

Um ... and you said this was current 8.1.x ... so it's dying here:

    tuple = SearchSysCache(INDEXRELID,
                           ObjectIdGetDatum(RelationGetRelid(relation)),
                           0, 0, 0);
    if (!HeapTupleIsValid(tuple))
        elog(ERROR, "cache lookup failed for index %u",
             RelationGetRelid(relation));

I interpret this to mean that it can't find the pg_index row for
pg_class_oid_index --- and since you're using -P, that doesn't just mean
corruption in pg_index's indexes, but that the tuple can't be found even
by seqscanning the whole catalog.  Probably the whole page it's in got
wiped out by your filesystem-level failure.

I hate to be the bearer of bad news, but I think this DB may be beyond
recovery.  It's very unlikely that there's just the one tuple gone.
If you're willing to throw money at the problem, there are various
people who offer consulting services that include trying to reconstruct
broken Postgres databases; but you might be best advised to just take
your lumps and go back to your last good backup.  At this point you're
looking at a significant investment of time with no guarantee of being
able to extract anything very useful.

What I'm taking from this is another horror story about the risks of
mounting databases across networks instead of locally :-(.  Postgres
is only as reliable as the storage it's using.

			regards, tom lane