From: David Roussel <pgsql-performance@diroussel.xsmail.com>
To: David Lang <dlang@invendra.net>
Cc: pgsql-performance@postgresql.org
Subject: Re: filesystem performance with lots of files
Date: Tue, 20 Dec 2005 13:26:00 +0000
Message-ID: <43A80668.1040604@diroussel.xsmail.com> (raw)
In-Reply-To: <Pine.LNX.4.62.0512010616200.2807@qnivq.ynat.uz>
References: <Pine.LNX.4.62.0512010616200.2807@qnivq.ynat.uz>
David Lang wrote:
>
> ext3 has an option to make searching directories faster (htree), but
> enabling it kills performance when you create files. And this doesn't
> help with large files.
>
The ReiserFS white paper talks about the data structure he uses to store
directories (some kind of tree), and he says it's quick to both read and
write. Don't forget if you find ls slow, that could just be ls, since
it's ls, not the fs, that sorts this files into alphabetical order.
> how long would it take to do a tar-ftp-untar cycle with no smarts
Note that you can do the taring, zipping, copying and untaring
concurrentlt. I can't remember the exactl netcat command line options,
but it goes something like this
Box1:
tar czvf - myfiles/* | netcat myserver:12345
Box2:
netcat -listen 12345 | tar xzvf -
Not only do you gain from doing it all concurrently, but not writing a
temp file means that disk seeks a reduced too if you have a one spindle
machine.
Also condsider just copying files onto a network mount. May not be as
fast as the above, but will be faster than rsync, which has high CPU
usage and thus not a good choice on a LAN.
Hmm, sorry this is not directly postgres anymore...
David
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Reply to all the recipients using the --to and --cc options:
reply via email
To: pgsql-performance@postgresql.org
Cc: pgsql-performance@diroussel.xsmail.com, dlang@invendra.net
Subject: Re: filesystem performance with lots of files
In-Reply-To: <43A80668.1040604@diroussel.xsmail.com>
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
This inbox is served by DDX for PostgreSQL; see mirroring instructions
for how to clone and mirror all data and code used for this inbox