From: Ron Mayer <rm_pg@cheapcomplexdevices.com>
To: Naomi Walker <nwalker@eldocomp.com>
Cc: Scott Marlowe <scott.marlowe@gmail.com>
Cc: Michael Monnerie <michael.monnerie@is.it-management.at>
Cc: pgsql-admin@postgresql.org
Subject: Re: 8.3.5 broken after power fail SOLVED
Date: Sat, 21 Feb 2009 14:41:13 -0800
Message-ID: <49A08309.1060809@cheapcomplexdevices.com> (raw)
In-Reply-To: <49A0110D.8020302@eldocomp.com>
References: <43a3a0bccd3684972f33e0051b45f2b4@webmail.zmi.at>
<200902210943.27875@zmi.at>
<dcc563d10902210140x22e9b4bckcf17ac02e4c91769@mail.gmail.com>
<49A0110D.8020302@eldocomp.com>
Naomi Walker wrote:
> Other than disaster tests, how would I know if I have an system that
> lies about fsync?
Well, the linux kernel tries to detect it on bootup and
will give messages like this:
%dmesg | grep 'disabling barriers'
JBD: barrier-based sync failed on md1 - disabling barriers
JBD: barrier-based sync failed on hda3 - disabling barriers
when it detects certain types of unreliable fsync's. The command
%hdparm -I /dev/hdf | grep FLUSH_CACHE_EXT
will give you clues if a hard drive itself even can support
a non-lying fsync when it's internal cache is enabled.
Sadly some filesystems (ext3) lie even above and beyond what
Linux does - by only using the write barriers correctly
when the inode itself is modified; not when the data is modified.
A test program here:
http://archives.postgresql.org/pgsql-performance/2008-08/msg00159.php
can detect those cases where the kernel & drive don't lie
about fsync but ext3 lies in spite of them; with more background
info here:
http://article.gmane.org/gmane.linux.file-systems/21373http://thread.gmane.org/gmane.linux.kernel/646040
Elsewhere in the archives you can find programs that measure
how fast fsyncs happen - but on your hardware, and you can
try to see if those numbers approximately match how fast your
disks spin. But then you still need to make sure the test
program used the same methods for syncing the drive that your
postgres configuration files are choosing.
I wonder if the only really safe way is to run a very
write intensive database script and pull and kill your
system in a number of ways, including yanking power to
the system; to disk arrays, etc and see if your database died.
>
>
>>
>> We preach this again and again. PostgreSQL can only survive a power
>> outage type failure ONLY if the hardware / OS / filesystem don't lie
>> about fsync. If they do, all bets are off, and this kind of failure
>> means you should really failover to another machine or restore a
>> backup.
>>
>> It's why you have to do possibly destructive tests to see if your
>> server stands at least some chance of surviving this kind of failure,
>> log shipping for recovery, and / or replication of another form (slony
>> etc...) to have a reliable server.
>>
>> The recommendations for recovery of data are just that, recovery
>> oriented. They can't fix a broken database at that point. You need
>> to take it offline after this kind of failure if you can't trust your
>> hardware.
>>
>> Usually when it finds something wrong it just won't start up.
>>
>>
>
>
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Reply to all the recipients using the --to and --cc options:
reply via email
To: pgsql-admin@postgresql.org
Cc: rm_pg@cheapcomplexdevices.com, nwalker@eldocomp.com, scott.marlowe@gmail.com, michael.monnerie@is.it-management.at
Subject: Re: 8.3.5 broken after power fail SOLVED
In-Reply-To: <49A08309.1060809@cheapcomplexdevices.com>
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
This inbox is served by DDX for PostgreSQL; see mirroring instructions
for how to clone and mirror all data and code used for this inbox