pg.ddx.io pgsql-performance@postgresql.org mailing list archive
help / color / mirror / Atom feedFilesystem fragmentation (Re: Fragmentation of WAL files)
5+ messages / 5 participants
[nested] [flat]
* Filesystem fragmentation (Re: Fragmentation of WAL files)
@ 2007-04-26 13:14 Bill Moran <wmoran@collaborativefusion.com>
2007-04-26 16:35 ` Re: Filesystem fragmentation (Re: Fragmentation of WAL files) Craig A. James <cjames@modgraph-usa.com>
2007-04-27 07:39 ` Re: Filesystem fragmentation (Re: Fragmentation of WAL files) Florian Weimer <fweimer@bfk.de>
0 siblings, 2 replies; 5+ messages in thread
From: Bill Moran @ 2007-04-26 13:14 UTC (permalink / raw)
To: Heikki Linnakangas <heikki@enterprisedb.com>; +Cc: Jim Nasby <decibel@decibel.org>; pgsql-performance
In response to Heikki Linnakangas <heikki@enterprisedb.com>:
[snip]
> >> Can anyone else confirm this? I don't know if this is a windows-only
> >> issue, but I don't know of a way to check fragmentation in unix.
> >
> > I can confirm that it's only a Windows problem. No UNIX filesystem
> > that I'm aware of suffers from fragmentation.
>
> What do you mean by suffering? All filesystems fragment files at some
> point. When and how differs from filesystem to filesystem. And some
> filesystems might be smarter than others in placing the fragments.
To clarify my viewpoint:
To my knowledge, there is no Unix filesystem that _suffers_ from
fragmentation. Specifically, all filessytems have some degree of
fragmentation that occurs, but every Unix filesystem that I am aware of
has built-in mechanisms to mitigate this and prevent it from becoming
a performance issue.
> There's a tool for Linux in the e2fsprogs package called filefrag that
> shows the fragmentation of a file, but I've never used it myself.
Interesting. However, the existence of a tool does not particularly
indicated the _need_ for said tool. It might just have been something
cool that somebody wrote.
--
Bill Moran
Collaborative Fusion Inc.
http://people.collaborativefusion.com/~wmoran/
wmoran@collaborativefusion.com
Phone: 412-422-3463x4023
^ permalink raw reply [nested|flat] 5+ messages in thread
* Re: Filesystem fragmentation (Re: Fragmentation of WAL files)
2007-04-26 13:14 Filesystem fragmentation (Re: Fragmentation of WAL files) Bill Moran <wmoran@collaborativefusion.com>
@ 2007-04-26 16:35 ` Craig A. James <cjames@modgraph-usa.com>
2007-04-26 16:47 ` Re: Filesystem fragmentation (Re: Fragmentation of WAL files) Gregory Stark <stark@enterprisedb.com>
1 sibling, 1 reply; 5+ messages in thread
From: Craig A. James @ 2007-04-26 16:35 UTC (permalink / raw)
To: Bill Moran <wmoran@collaborativefusion.com>; +Cc: Heikki Linnakangas <heikki@enterprisedb.com>; pgsql-performance
Bill Moran wrote:
> In response to Heikki Linnakangas <heikki@enterprisedb.com>:
>>>> Can anyone else confirm this? I don't know if this is a windows-only
>>>> issue, but I don't know of a way to check fragmentation in unix.
>>> I can confirm that it's only a Windows problem. No UNIX filesystem
>>> that I'm aware of suffers from fragmentation.
>> What do you mean by suffering? All filesystems fragment files at some
>> point. When and how differs from filesystem to filesystem. And some
>> filesystems might be smarter than others in placing the fragments.
>
> To clarify my viewpoint:
> To my knowledge, there is no Unix filesystem that _suffers_ from
> fragmentation. Specifically, all filessytems have some degree of
> fragmentation that occurs, but every Unix filesystem that I am aware of
> has built-in mechanisms to mitigate this and prevent it from becoming
> a performance issue.
More specifically, this problem was solved on UNIX file systems way back in the 1970's and 1980's. No UNIX file system (including Linux) since then has had significant fragmentation problems, unless the file system gets close to 100% full. If you run below 90% full, fragmentation shouldn't ever be a significant performance problem.
The word "fragmentation" would have dropped from the common parlance if it weren't for MS Windoz.
Craig
^ permalink raw reply [nested|flat] 5+ messages in thread
* Re: Filesystem fragmentation (Re: Fragmentation of WAL files)
2007-04-26 13:14 Filesystem fragmentation (Re: Fragmentation of WAL files) Bill Moran <wmoran@collaborativefusion.com>
2007-04-26 16:35 ` Re: Filesystem fragmentation (Re: Fragmentation of WAL files) Craig A. James <cjames@modgraph-usa.com>
@ 2007-04-26 16:47 ` Gregory Stark <stark@enterprisedb.com>
2007-04-26 17:49 ` Re: Filesystem fragmentation (Re: Fragmentation of WAL files) Tom Lane <tgl@sss.pgh.pa.us>
0 siblings, 1 reply; 5+ messages in thread
From: Gregory Stark @ 2007-04-26 16:47 UTC (permalink / raw)
To: Craig A. James <cjames@modgraph-usa.com>; +Cc: Bill Moran <wmoran@collaborativefusion.com>; Heikki Linnakangas <heikki@enterprisedb.com>; pgsql-performance
"Craig A. James" <cjames@modgraph-usa.com> writes:
> More specifically, this problem was solved on UNIX file systems way back in the
> 1970's and 1980's. No UNIX file system (including Linux) since then has had
> significant fragmentation problems, unless the file system gets close to 100%
> full. If you run below 90% full, fragmentation shouldn't ever be a significant
> performance problem.
Note that the main technique used to avoid fragmentation -- paradoxically --
is to break the file up into reasonable sized chunks. This allows the
filesystem the flexibility to place the chunks efficiently.
In the case of a performance-critical file like the WAL that's always read
sequentially it may be to our advantage to defeat this technique and force it
to be allocated sequentially. I'm not sure whether any filesystems provide any
option to do so.
--
Gregory Stark
EnterpriseDB http://www.enterprisedb.com
^ permalink raw reply [nested|flat] 5+ messages in thread
* Re: Filesystem fragmentation (Re: Fragmentation of WAL files)
2007-04-26 13:14 Filesystem fragmentation (Re: Fragmentation of WAL files) Bill Moran <wmoran@collaborativefusion.com>
2007-04-26 16:35 ` Re: Filesystem fragmentation (Re: Fragmentation of WAL files) Craig A. James <cjames@modgraph-usa.com>
2007-04-26 16:47 ` Re: Filesystem fragmentation (Re: Fragmentation of WAL files) Gregory Stark <stark@enterprisedb.com>
@ 2007-04-26 17:49 ` Tom Lane <tgl@sss.pgh.pa.us>
0 siblings, 0 replies; 5+ messages in thread
From: Tom Lane @ 2007-04-26 17:49 UTC (permalink / raw)
To: Gregory Stark <stark@enterprisedb.com>; +Cc: Craig A. James <cjames@modgraph-usa.com>; Bill Moran <wmoran@collaborativefusion.com>; Heikki Linnakangas <heikki@enterprisedb.com>; pgsql-performance
Gregory Stark <stark@enterprisedb.com> writes:
> In the case of a performance-critical file like the WAL that's always read
> sequentially it may be to our advantage to defeat this technique and force it
> to be allocated sequentially. I'm not sure whether any filesystems provide any
> option to do so.
We more or less do that already by filling the entire file in one go
when it's created ...
regards, tom lane
^ permalink raw reply [nested|flat] 5+ messages in thread
* Re: Filesystem fragmentation (Re: Fragmentation of WAL files)
2007-04-26 13:14 Filesystem fragmentation (Re: Fragmentation of WAL files) Bill Moran <wmoran@collaborativefusion.com>
@ 2007-04-27 07:39 ` Florian Weimer <fweimer@bfk.de>
1 sibling, 0 replies; 5+ messages in thread
From: Florian Weimer @ 2007-04-27 07:39 UTC (permalink / raw)
To: Bill Moran <wmoran@collaborativefusion.com>; +Cc: Heikki Linnakangas <heikki@enterprisedb.com>; Jim Nasby <decibel@decibel.org>; pgsql-performance
* Bill Moran:
> To clarify my viewpoint:
> To my knowledge, there is no Unix filesystem that _suffers_ from
> fragmentation. Specifically, all filessytems have some degree of
> fragmentation that occurs, but every Unix filesystem that I am aware of
> has built-in mechanisms to mitigate this and prevent it from becoming
> a performance issue.
One database engine tends to create a huge number of fragments because
the files are written with holes in them. There is a significant
impact on sequential reads, but that doesn't matter much because the
engine doesn't implement fast, out-of-order B-tree scans anyway. 8-/
I still think that preallocating in reasonably sized chunks is
beneficial.
--
Florian Weimer <fweimer@bfk.de>
BFK edv-consulting GmbH http://www.bfk.de/
Kriegsstraße 100 tel: +49-721-96201-1
D-76133 Karlsruhe fax: +49-721-96201-99
^ permalink raw reply [nested|flat] 5+ messages in thread
end of thread, other threads:[~2007-04-27 07:39 UTC | newest]
Thread overview: 5+ messages (download: mbox mbox.gz follow: Atom feed)
-- links below jump to the message on this page --
2007-04-26 13:14 Filesystem fragmentation (Re: Fragmentation of WAL files) Bill Moran <wmoran@collaborativefusion.com>
2007-04-26 16:35 ` Craig A. James <cjames@modgraph-usa.com>
2007-04-26 16:47 ` Gregory Stark <stark@enterprisedb.com>
2007-04-26 17:49 ` Tom Lane <tgl@sss.pgh.pa.us>
2007-04-27 07:39 ` Florian Weimer <fweimer@bfk.de>
This inbox is served by DDX for PostgreSQL; see mirroring instructions
for how to clone and mirror all data and code used for this inbox