Message-ID: <5021BCA6.7030000@ringerc.id.au>
Date: Wed, 08 Aug 2012 09:11:02 +0800
From: Craig Ringer <ringerc@ringerc.id.au>
User-Agent: Mozilla/5.0 (X11; Linux x86_64;
	rv:14.0) Gecko/20120717 Thunderbird/14.0
MIME-Version: 1.0
To: Terry Schmitt <tschmitt@schmittworks.com>
CC: pgsql-admin@postgresql.org
Subject: Re: Data corruption after SAN snapshot
References: 
 <CAOOcyswLYBfJDuvNBPMkiNCGNKgK=SiexUuTVHCh2O+Y1T-sLw@mail.gmail.com>
In-Reply-To: 
 <CAOOcyswLYBfJDuvNBPMkiNCGNKgK=SiexUuTVHCh2O+Y1T-sLw@mail.gmail.com>
Content-Type: text/plain; charset=ISO-8859-1; format=flowed
Content-Transfer-Encoding: 7bit

On 08/08/2012 06:23 AM, Terry Schmitt wrote:

> Anyone have a solid method to test if fdatasync is working correctly or
> thoughts on troubleshooting this?

Try diskchecker.pl

https://gist.github.com/3177656

The other obvious step is that you've changed three things, so start 
isolation testing.

- Test Postgres Plus Advanced Server 8.4, which you knew worked, on your 
new file system and OS.

- Test PP9.1 on your new OS but with ext3, which you knew worked

- Test PP9.1 on your new OS but with ext4, which should work if ext3 did

- Test PP9.1 on a copy of your *old* OS with the old file system setup.

- Test mainline PostgreSQL 9.1 on your new setup to see if it's PP specific.

Since each test sounds moderately time consuming, you'll probably need 
to find a way to automate. I'd first see if I could reproduce the 
problem when running PgBench against the same setup that's currently 
failing, and if that reproduces the fault you can use PgBench with the 
other tests.

--
Craig Ringer