DomainKey-Signature: a=rsa-sha1; c=nofws; d=wpkg.org; h=message-id:date
	:from:mime-version:to:cc:subject:references:in-reply-to
	:content-type:content-transfer-encoding; q=dns; s=default; b=jaJ
	hF3TAmp27qOlYHtZ01sbfZDgnp0M9u0bkITc9bai8BqzWr6D7lAUQ58iX/ejYEWC
	sq5OSkYQDqFBk1ayIYizvP0KspDdmi+XLin8/+W40xdCXk1yOWMy1UwbsIn4Svan
	9ZB12G0zfKOmgGNjvj5cxZ9JDouuX3snAMys2/qw=
Message-ID: <4DC1B378.6050105@wpkg.org>
Date: Wed, 04 May 2011 22:13:44 +0200
From: Tomasz Chmielewski <mangoo@wpkg.org>
User-Agent: Mozilla/5.0 (X11; U; Linux x86_64; en-US;
	rv:1.9.2.15) Gecko/20110419 Lightning/1.0b2 Thunderbird/3.1.9
MIME-Version: 1.0
To: Scott Marlowe <scott.marlowe@gmail.com>
CC: Kevin Grittner <Kevin.Grittner@wicourts.gov>, pgsql-admin@postgresql.org
Subject: Re: 'SGT DETAIL: Could not open file "pg_clog/05DC": No such
	file or directory' - what to do now?
References: <4DC0F063.70306@wpkg.org>
	<4DC15137020000250003D295@gw.wicourts.gov>	<4DC19CD1.7070600@wpkg.org>
	<BANLkTikG3yN0xB_wTy1SVYOrA9E=FNBZLg@mail.gmail.com>
In-Reply-To: <BANLkTikG3yN0xB_wTy1SVYOrA9E=FNBZLg@mail.gmail.com>
Content-Type: text/plain; charset=ISO-8859-1
Content-Transfer-Encoding: 7bit

On 04.05.2011 21:50, Scott Marlowe wrote:

> Then another pg_clog file disappeared.
> 
> Is it possible there's some rogue process deleting files in pg_clog
> somehow?

I don't think.


> Have you run an fsck on this drive to make sure it's not got
> any file system errors?

Also, don't think there is any corruption here. AFAIR, this system never crashed.
Could be there is some silent corruption though - but if there really was one, we would likely see the kernel complaining, stale files elsewhere, and so on.

Without such clues on filesystem corruption, I can't afford downtime.


I didn't mention, but the application first talks to pgpool, which talks to two database servers (i.e. inserts to both).

The real fun begins here - this is from two different servers:

db10:/var/log/postgresql# zgrep "No such" *
postgresql_log:May  4 18:24:28 db10 postgres[15751]: [23-2] 2011-05-04 18:24:28 SGT DETAIL:  Could not open file "pg_clog/0601": No such file or directory.
postgresql_log:May  4 22:43:44 db10 postgres[15773]: [555-2] 2011-05-04 22:43:44 SGT DETAIL:  Could not open file "pg_clog/0601": No such file or directory.
postgresql_log:May  4 22:44:30 db10 postgres[15791]: [1841-2] 2011-05-04 22:44:30 SGT DETAIL:  Could not open file "pg_clog/0601": No such file or directory.
postgresql_log:May  4 22:55:53 db10 postgres[15741]: [4114-2] 2011-05-04 22:55:53 SGT DETAIL:  Could not open file "pg_clog/0601": No such file or directory.


db20:/var/log/postgresql# zgrep "No such" *
postgresql_log:May  4 18:24:28 db20 postgres[27114]: [2-2] 2011-05-04 18:24:28 SGT DETAIL:  Could not open file "pg_clog/0601": No such file or directory.
postgresql_log:May  4 22:43:44 db20 postgres[27116]: [2-2] 2011-05-04 22:43:44 SGT DETAIL:  Could not open file "pg_clog/0601": No such file or directory.
postgresql_log:May  4 22:44:30 db20 postgres[27138]: [2-2] 2011-05-04 22:44:30 SGT DETAIL:  Could not open file "pg_clog/0601": No such file or directory.
postgresql_log:May  4 22:55:53 db20 postgres[27104]: [2-2] 2011-05-04 22:55:53 SGT DETAIL:  Could not open file "pg_clog/0601": No such file or directory.


I can't exclude some corruption happened much earlier on db10; the whole database (as binary files) was copied to db20 almost 2 months ago.

Why would it start showing pg_clog files missing just 2 days ago, and not earlier? Hmm.


-- 
Tomasz Chmielewski
http://wpkg.org