To: "BJ Taylor" <btaylor@propertysolutions.com>
cc: pgsql-admin@postgresql.org
Subject: Re: missing chunk number 0 for toast value 
In-reply-to: <3d78fcfd0809250909h148daba8k2ad29e176d9a06fd@mail.gmail.com> 
References: <3d78fcfd0809241132y14a832cai8b32bc32e936699e@mail.gmail.com>
	<10862.1222287999@sss.pgh.pa.us>
	<3d78fcfd0809241446j326110c4p9907e04dbd10106b@mail.gmail.com>
	<5723.1222349048@sss.pgh.pa.us>
	<3d78fcfd0809250909h148daba8k2ad29e176d9a06fd@mail.gmail.com>
Comments: In-reply-to "BJ Taylor" <btaylor@propertysolutions.com>
	message dated "Thu, 25 Sep 2008 10:09:48 -0600"
Date: Thu, 25 Sep 2008 13:30:30 -0400
Message-ID: <9025.1222363830@sss.pgh.pa.us>
From: Tom Lane <tgl@sss.pgh.pa.us>

"BJ Taylor" <btaylor@propertysolutions.com> writes:
> Here are some recent logs from our system.  Unfortunately, I didn't think to
> grab the logs at the time I killed those processes, and now they are gone.
> I found those processes by using ps, and then I killed them with a simple
> kill *processid*.  Here are samples of our current log files:

Huh.  That would have generated SIGTERM, which should at least mostly be
safe.  We did recently fix a couple of problems with cleanup after
SIGTERM but they were both pretty low-probability risks.  Anyway,
you've definitely got issues now:

> PANIC:  right sibling's left-link doesn't match: block 175337 links to
> 243096 instead of expected 29675 in index "dbmail_headervalue_3"

Try reindexing that index, and any other ones that you see complaints
about.

> You also asked how it is being unstable.  It drops connections seemingly at
> random.  The error received when a connection is dropped is the following:

> WARNING:  terminating connection because of crash of another server process

Well, that's just the fallout from a PANIC in some other session; if you
weren't watching the session that actually crashed, you need to look in
the postmaster log to see what went wrong.

			regards, tom lane