Received: from magus.postgresql.org (magus.postgresql.org [87.238.57.229]) by mail.postgresql.org (Postfix) with ESMTP id A9BC943CEB8 for ; Thu, 19 Jan 2012 10:31:14 -0400 (AST) Received: from fuse-dedicated-74-83-143-254.fuse.net ([74.83.143.254] helo=relay.beechglen.com) by magus.postgresql.org with esmtp (Exim 4.72) (envelope-from ) id 1Rnt1E-0000c9-AG for pgsql-admin@postgresql.org; Thu, 19 Jan 2012 14:31:13 +0000 Received: from mail.beechglen.com (apache.beechglen.com [192.168.88.51]) by relay.beechglen.com (8.13.8/8.13.8) with ESMTP id q0JEUueb008201 (version=TLSv1/SSLv3 cipher=DHE-RSA-AES256-SHA bits=256 verify=NO); Thu, 19 Jan 2012 09:30:56 -0500 Received: from 204.10.222.10 (SquirrelMail authenticated user davidh) by mail.beechglen.com with HTTP; Thu, 19 Jan 2012 09:30:56 -0500 Message-ID: In-Reply-To: References: Date: Thu, 19 Jan 2012 09:30:56 -0500 Subject: Re: Help! PostgreSQL stuck at starting up after crash From: "David Hornsby" To: "Samuel Hwang" Cc: pgsql-admin@postgresql.org Reply-To: david@beechglen.com User-Agent: SquirrelMail/1.4.22 MIME-Version: 1.0 Content-Type: text/plain;charset=iso-8859-1 Content-Transfer-Encoding: 8bit X-Priority: 3 (Normal) Importance: Normal X-Pg-Spam-Score: -0.9 (/) X-Archive-Number: 201201/100 X-Sequence-Number: 39012 Sounds like you have a corrupt wal files that you will have to reset the wal logs with pgresetxlog. http://www.postgresql.org/docs/8.2/static/app-pgresetxlog.html This will result in missing transactions so before you do this shutdown postgres and make a copy of the database files first. That way if you don't like what happens you can always go back to the way things were. Also right now would be a good time to evaluate your backup strategy, which is a different topic for a different thread, but I can certainly help with that as well. -David Hornsby > version Postgresql 9.1.1 on centos5 x64 > > We experience slow performance and found the server is running 3 vacuum > process on the same db which use up 99% of CPU. > Then we kill -9 one of those process which cause postgresql to crash and > it > tried to restart after the crash > However when the starting process reach the last WAL files, it just stuck > there > > pg_controldata shows the db is in Archive Recovery mode and when using > psql > to connect the db, it says FATAL: the database system starting up. > > I took a chance and upgrade to PostgreSql 9.1.2 and see if anything > changed > it still stuck at the end of recovery. > pg_controldata shows db is in Crash recovery, but that probably different > wording I think > using psql to connect the db, it says FATAL: the database system is > starting up. > > I pretty much run out of idea here. > Can anyone help what to go from here? > > Samuel >