Date: Tue, 15 Aug 2006 14:15:05 -0500
From: "Jim C. Nasby" <jnasby@pervasive.com>
To: mark@mark.mielke.cc, pgsql-performance@postgresql.org
Subject: Re: Postgresql Performance on an HP DL385 and
Message-ID: <20060815191505.GH21363@pervasive.com>
References: <1155180553.13315.0.camel@amd64-gentoo-laptop>
	<20060810110935.GF2900@mathom.us>
	<20060814153841.GF27928@pervasive.com>
	<721b21dc0608140851v2332cdf9g59808004c983540f@mail.gmail.com>
	<20060814170545.GG27928@pervasive.com>
	<20060814170902.GN2900@mathom.us>
	<20060815162926.GS27928@pervasive.com>
	<20060815172645.GU2900@mathom.us>
	<20060815183327.GC12834@mark.mielke.cc>
	<20060815190254.GW2900@mathom.us>
Mime-Version: 1.0
Content-Type: text/plain; charset=us-ascii
Content-Disposition: inline
In-Reply-To: <20060815190254.GW2900@mathom.us>
User-Agent: Mutt/1.5.11

On Tue, Aug 15, 2006 at 03:02:56PM -0400, Michael Stone wrote:
> On Tue, Aug 15, 2006 at 02:33:27PM -0400, mark@mark.mielke.cc wrote:
> >On Tue, Aug 15, 2006 at 01:26:46PM -0400, Michael Stone wrote:
> >>On Tue, Aug 15, 2006 at 11:29:26AM -0500, Jim C. Nasby wrote:
> >>>Are 'we' sure that such a setup can't lose any data?
> >>Yes. If you check the archives, you can even find the last time this was 
> >>discussed...
> >
> >I looked last night (coincidence actually) and didn't find proof that
> >you cannot lose data.
> 
> You aren't going to find proof, any more than you'll find proof that you 
> won't lose data if you do lose a journalling fs. (Because there isn't 
> any.) Unfortunately, many people misunderstand the what a metadata 
> journal does for you, and overstate its importance in this type of 
> application.
> 
> >How do you deal with the file system structure being updated before the
> >data blocks are (re-)written?
> 
> *That's what the postgres log is for.* If the latest xlog entries don't 
> make it to disk, they won't be replayed; if they didn't make it to 
> disk, the transaction would not have been reported as commited. An 
> application that understands filesystem semantics can guarantee data 
> integrity without metadata journaling.
 
So what causes files to get 'lost' and get stuck in lost+found?
AFAIK that's because the file was written before the metadata. Now, if
fsync'ing a file also ensures that all the metadata is written, then
we're probably fine... if not, then we're at risk every time we create a
new file (every WAL segment if archiving is on, and every time a
relation passes a 1GB boundary).

FWIW, the way that FreeBSD gets around the need to fsck a dirty
filesystem before use without using a journal is to ensure that metadate
operations are always on the drive before the actual data is written.
There's still a need to fsck a dirty filesystem, but it can now be done
in the background, with the filesystem mounted and in use.

> >>The bottom line is that the only reason you need a metadata journalling 
> >>filesystem is to save the fsck time when you come up. On a little 
> >>partition like xlog, that's not an issue.
> >
> >fsck isn't only about time to fix. fsck is needed, because the file system
> >is broken. 
> 
> fsck is needed to reconcile the metadata with the on-disk allocations. 
> To do that, it reads all the inodes and their corresponding directory 
> entries. The time to do that is proportional to the size of the 
> filesystem, hence the comment about time. fsck is not needed "because 
> the filesystem is broken", it's needed because the filesystem is marked 
> dirty. 
> 
> Mike Stone
> 
> ---------------------------(end of broadcast)---------------------------
> TIP 5: don't forget to increase your free space map settings
> 

-- 
Jim C. Nasby, Sr. Engineering Consultant      jnasby@pervasive.com
Pervasive Software      http://pervasive.com    work: 512-231-6117
vcard: http://jim.nasby.net/pervasive.vcf       cell: 512-569-9461