Received: from maia.hub.org (maia-2.hub.org [200.46.204.251]) by mail.postgresql.org (Postfix) with ESMTP id F3C25B5DBC4 for ; Mon, 10 Oct 2011 15:08:18 -0300 (ADT) Received: from mail.postgresql.org ([200.46.204.86]) by maia.hub.org (mx1.hub.org [200.46.204.251]) (amavisd-maia, port 10024) with ESMTP id 88009-01 for ; Mon, 10 Oct 2011 18:08:12 +0000 (UTC) X-Greylist: from auto-whitelisted by SQLgrey-1.8.0-rc2 Received: from EXCASHUBB.onmobile.com (unknown [124.30.188.115]) by mail.postgresql.org (Postfix) with ESMTP id E4E5CB5DBD8 for ; Mon, 10 Oct 2011 15:08:11 -0300 (ADT) Received: from MUSCA.onmobile.com ([172.16.28.141]) by EXCASHUBB.onmobile.com ([172.16.28.185]) with mapi; Mon, 10 Oct 2011 23:43:04 +0530 From: Akash Kodibail To: Kevin Grittner , "pgsql-admin@postgresql.org" Date: Mon, 10 Oct 2011 23:43:03 +0530 Subject: Re: Postgres Server crashed Thread-Topic: [ADMIN] Postgres Server crashed Thread-Index: AcyHcTq9XtwYa1epSpaa35HuSK4hZQAATLhw Message-ID: <04DA5C9AF2453F44ACD4541A08EFE9832D0CF8EEFE@MUSCA.onmobile.com> References: <04DA5C9AF2453F44ACD4541A08EFE9832D0CF8EE87@MUSCA.onmobile.com> <4E92E24E0200002500041CCA@gw.wicourts.gov> In-Reply-To: <4E92E24E0200002500041CCA@gw.wicourts.gov> Accept-Language: en-US Content-Language: en-US X-MS-Has-Attach: X-MS-TNEF-Correlator: acceptlanguage: en-US Content-Type: text/plain; charset="us-ascii" Content-Transfer-Encoding: quoted-printable MIME-Version: 1.0 X-Virus-Scanned: Maia Mailguard 1.0.1 X-Spam-Status: No, hits=-1.107 tagged_above=-10 required=5 tests=BAYES_00=-1.9, RDNS_NONE=0.793 X-Spam-Level: X-Archive-Number: 201110/46 X-Sequence-Number: 38498 Thank You Kevin. Details: --------------- Running "select version()" gives me "PostgreSQL 8.4.4 on x86_64-unknown-li= nux-gnu, compiled by GCC gcc (GCC) 3.4.6 20060404 (Red Hat 3.4.6-10), 64-bi= t" dmesg | head -10 -gives- "Linux version 2.6.18-92.el5 (mockbuild@builder10.= centos.org) (gcc version 4.1.2 20071124 (Red Hat 4.1.2-42)) #1 SMP Tue Jun = 10 18:51:06 EDT 2008" Database mounted on 285 GB NFS HD --------------- I am copying it to a path as and when I write this email. Regular backups is something I am afraid I have not been taking. But this i= ncident sure has taught me a lesson. pg_clog and pg_xlog are untouched. Thanks and Regards, Akash. -----Original Message----- From: Kevin Grittner [mailto:Kevin.Grittner@wicourts.gov] Sent: Monday, October 10, 2011 10:47 PM To: Akash Kodibail; pgsql-admin@postgresql.org Subject: Re: [ADMIN] Postgres Server crashed kash Kodibail wrote: > Lately the postgres server crashed due to space constraints, The very first thing to do is to copy the entire directory structure for the database cluster to somewhere you can keep it safe until the problem has been resolved. If you have not yet done so, I strongly recommend that you do so before trying anything else for recovery. > pg_control file got lost and the server wouldn't start. "got lost"? It seems to me that it would be a Very Bad Thing if an out of disk space situation allowed PostgreSQL to cause that. Should we be looking for a bug, or is there a chance someone deleted this in an attempt to free space? If the latter, is there any chance that other files critical to data integrity were deleted, like in the pg_xlog or pg_clog directories? > Had a old pg_control file placed in $PGDATA/global/ path. > > I was able to start the server, however, the data seems to be a > very old one (2011 Jan 07) (probably the time where pg_control I > had pasted was last updated). And normal maintenance may be viewing newer data as old, due to transaction wrap-around from the old pg_control file, and removing it as part of normal cleanup. So you may have destroyed some of your more recent data by doing that. > How do I recover the rest of the data? Well, as I said above, start by trying to preserve whatever you have now. After you've done that, please provide more information. We don't even know what version of PostgreSQL you're talking about, for example. http://wiki.postgresql.org/wiki/Guide_to_reporting_problems It's somewhat likely that you'll be best off going to your last known good backup, assuming you've been doing regular backups. -Kevin DISCLAIMER: The information in this message is confidential and may be lega= lly privileged. It is intended solely for the addressee. Access to this mes= sage by anyone else is unauthorized. If you are not the intended recipient,= any disclosure, copying, or distribution of the message, or any action or = omission taken by you in reliance on it, is prohibited and may be unlawful.= Please immediately contact the sender if you have received this message in= error. Further, this e-mail may contain viruses and all reasonable precaut= ion to minimize the risk arising there from is taken by OnMobile. OnMobile = is not liable for any damage sustained by you as a result of any virus in t= his e-mail. All applicable virus checks should be carried out by you before= opening this e-mail or any attachment thereto. Thank you - OnMobile Global Limited.