Received: from malur.postgresql.org ([217.196.149.56]) by arkaria.postgresql.org with esmtp (Exim 4.72) (envelope-from ) id 1TsOy1-0000AL-SG for pgsql-admin@arkaria.postgresql.org; Tue, 08 Jan 2013 02:31:06 +0000 Received: from localhost ([127.0.0.1] helo=postgresql.org) by malur.postgresql.org with smtp (Exim 4.72) (envelope-from ) id 1TsOy0-0005sg-Qa for pgsql-admin@arkaria.postgresql.org; Tue, 08 Jan 2013 02:31:04 +0000 Received: from makus.postgresql.org ([98.129.198.125]) by malur.postgresql.org with esmtp (Exim 4.72) (envelope-from ) id 1TsOxz-0005sa-ED for pgsql-admin@postgresql.org; Tue, 08 Jan 2013 02:31:03 +0000 Received: from [64.207.29.137] (helo=mail.illumen.com) by makus.postgresql.org with esmtp (Exim 4.72) (envelope-from ) id 1TsOxw-0006K6-Gn for pgsql-admin@postgresql.org; Tue, 08 Jan 2013 02:31:02 +0000 X-MimeOLE: Produced By Microsoft Exchange V6.5 Content-class: urn:content-classes:message MIME-Version: 1.0 Content-Type: multipart/alternative; boundary="----_=_NextPart_001_01CDED48.31DD43BA" Subject: Re: Possible database corruption Date: Mon, 7 Jan 2013 19:30:57 -0700 Message-ID: In-Reply-To: <50EB833C.4060206@2ndQuadrant.com> X-MS-Has-Attach: X-MS-TNEF-Correlator: Thread-Topic: [ADMIN] Possible database corruption Thread-Index: Ac3tRz8hNTx9H5y1R7Gurk2JPVfsmAAALXgA References: <50EB833C.4060206@2ndQuadrant.com> From: "Benjamin Krajmalnik" To: "Craig Ringer" Cc: X-Host-Lookup-Failed: Reverse DNS lookup failed for 64.207.29.137 (failed) X-Pg-Spam-Score: -1.1 (-) List-Archive: List-Help: List-ID: List-Owner: List-Post: List-Subscribe: List-Unsubscribe: X-Mailing-List: pgsql-admin Precedence: bulk Sender: pgsql-admin-owner@postgresql.org This is a multi-part message in MIME format. ------_=_NextPart_001_01CDED48.31DD43BA Content-Type: text/plain; charset="us-ascii" Content-Transfer-Encoding: quoted-printable Thanks for the reply - I posted an update that I had resolved the issue. When the partition with the WAL files filled up due to the missing fsm file (I wonder what caused that), the db panicked. After moving all 43GB of WAL files to a different partition, database came into recovery mode, and after about half an hour of processing the WAL files the server came back online. The only thing that is still pending is for the system to clean out all of the now unused wal files. Once this is done, I will move back the WAL files to their own spindle. =20 Since the database would not restart until the WAL files were moved I feared data corruption - which thankfully did not occur. =20 DB was Postgres 9.0.4 running on FreeBSD 8.1/amd64. Subsystem is dual RAID-1 SAS, OS/WAL on one set of spindles, data on the other. =20 =20 =20 From: Craig Ringer [mailto:craig@2ndQuadrant.com]=20 Sent: Monday, January 07, 2013 7:24 PM To: Benjamin Krajmalnik Cc: pgsql-admin@postgresql.org Subject: Re: [ADMIN] Possible database corruption =20 On 01/08/2013 05:22 AM, Benjamin Krajmalnik wrote: I have a situation where pg_xlog started growing until it filled up the disk drive. This should not ever cause corruption. If it has, there's a bug at work. A crash is reasonable (albeit undesirable; it'd be better to just report errors on connections) - but database corruption is not. Before doing ANYTHING else, read http://wiki.postgresql.org/wiki/Corruption and act on it. How big is the DB? What file system is it on? PostgreSQL 9.0.[what?] ? Host OS? Disk subsystem? --=20 Craig Ringer http://www.2ndQuadrant.com/ PostgreSQL Development, 24x7 Support, Training & Services ------_=_NextPart_001_01CDED48.31DD43BA Content-Type: text/html; charset="us-ascii" Content-Transfer-Encoding: quoted-printable

Thanks for the reply - I posted an update that I = had resolved the issue.

When the = partition with the WAL files filled up due to the missing fsm file (I = wonder what caused that), the db panicked.

After moving all 43GB of WAL files to a different = partition, database came into recovery mode, and after about half an = hour of processing the WAL files the server  came back = online.

The only thing that is = still pending is for the system to clean out all of the now unused wal = files.

Once this is done, I will = move back the WAL files to their own spindle.

 

Since = the database would not restart until the WAL files were moved I feared = data corruption - which thankfully did not occur.

 

DB was = Postgres 9.0.4 running on FreeBSD 8.1/amd64.  Subsystem is dual = RAID-1 SAS, OS/WAL on one set of spindles, data on the = other.

 

 

 

From: Craig Ringer [mailto:craig@2ndQuadrant.com]
Sent: = Monday, January 07, 2013 7:24 PM
To: Benjamin = Krajmalnik
Cc: pgsql-admin@postgresql.org
Subject: = Re: [ADMIN] Possible database = corruption

 

On = 01/08/2013 05:22 AM, Benjamin Krajmalnik = wrote:

I = have a situation where pg_xlog started growing until it filled up the = disk drive.

This = should not ever cause corruption. If it has, there's a bug at = work.

A crash is reasonable (albeit undesirable; it'd be better = to just report errors on connections) - but database corruption is = not.

Before doing ANYTHING else, read http://wiki.postgresq= l.org/wiki/Corruption and act on it.

How big is the = DB?

What file system is it on?

PostgreSQL 9.0.[what?] = ?

Host OS?

Disk subsystem?



-- =
 Craig =
Ringer           &=
nbsp;       http://www.2ndQuadrant.com/=
 PostgreSQL Development, 24x7 Support, Training & =
Services
------_=_NextPart_001_01CDED48.31DD43BA--