DomainKey-Signature: a=rsa-sha1; c=nofws; d=gmail.com; s=gamma;
	h=mime-version:in-reply-to:references:date:message-id:subject:from:to
	:cc:content-type:content-transfer-encoding;
	b=Ehjxl0n0OnIM64BFY40cv4RQWeM70/pOwf6Ncg3Bhqh0luNDE5QpMmy/H9ooFNIbGJ
	6eJHDCAJEVtSTcBkHapcD2B8YHAAjfsU0fuo54/ILlKvzXSB541AJpXfd8SCfc4lh2BJ
	TKY4P/864wAv6AYrKFrEwRoNUE2KOzY8f8cBI=
MIME-Version: 1.0
In-Reply-To: <49A08309.1060809@cheapcomplexdevices.com>
References: <43a3a0bccd3684972f33e0051b45f2b4@webmail.zmi.at>
	<200902210943.27875@zmi.at>
	<dcc563d10902210140x22e9b4bckcf17ac02e4c91769@mail.gmail.com>
	<49A0110D.8020302@eldocomp.com>
	<49A08309.1060809@cheapcomplexdevices.com>
Date: Sat, 21 Feb 2009 19:43:11 -0700
Message-ID: <dcc563d10902211843k54b38175m615b9d83d30927f4@mail.gmail.com>
Subject: Re: 8.3.5 broken after power fail SOLVED
From: Scott Marlowe <scott.marlowe@gmail.com>
To: Ron Mayer <rm_pg@cheapcomplexdevices.com>
Cc: Naomi Walker <nwalker@eldocomp.com>,
	Michael Monnerie <michael.monnerie@is.it-management.at>,
	pgsql-admin@postgresql.org
Content-Type: text/plain; charset=ISO-8859-1
Content-Transfer-Encoding: 7bit

On Sat, Feb 21, 2009 at 3:41 PM, Ron Mayer
<rm_pg@cheapcomplexdevices.com> wrote:
> Naomi Walker wrote:
>> Other than disaster tests, how would I know if I have an system that
>> lies about fsync?
>
> Well, the linux kernel tries to detect it on bootup and
> will give messages like this:
>  %dmesg | grep 'disabling barriers'
>  JBD: barrier-based sync failed on md1 - disabling barriers
>  JBD: barrier-based sync failed on hda3 - disabling barriers
> when it detects certain types of unreliable fsync's. The command
>  %hdparm -I /dev/hdf | grep FLUSH_CACHE_EXT
> will give you clues if a hard drive itself even can support
> a non-lying fsync when it's internal cache is enabled.
>
>
> Sadly some filesystems (ext3) lie even above and beyond what
> Linux does - by only using the write barriers correctly
> when the inode itself is modified; not when the data is modified.
> A test program here:
> http://archives.postgresql.org/pgsql-performance/2008-08/msg00159.php
> can detect those cases where the kernel & drive don't lie
> about fsync but ext3 lies in spite of them; with more background
> info here:
>  http://article.gmane.org/gmane.linux.file-systems/21373
>  http://thread.gmane.org/gmane.linux.kernel/646040
>
>
> Elsewhere in the archives you can find programs that measure
> how fast fsyncs happen - but on your hardware, and you can
> try to see if those numbers approximately match how fast your
> disks spin.  But then you still need to make sure the test
> program used the same methods for syncing the drive that your
> postgres configuration files are choosing.
>
> I wonder if the only really safe way is to run a very
> write intensive database script and pull and kill your
> system in a number of ways, including yanking power to
> the system; to disk arrays, etc and see if your database died.

Well, you can't prove it's 100% safe but you can usually find most of
the not safe systems this way.  I usually setup a big pgbench db, run
500 or so concurrent, wait 5 or 10 minutes, run a checkpoint, and
halfway through it pull the plug.  It's still possible for a system to
fail after passing this test, but I feel a lot better knowing I've
done it a couple of times and the db came back up without problems.