X-Original-To: pgsql-performance-postgresql.org@localhost.postgresql.org
Received: from localhost (av.hub.org [200.46.204.144])
	by postgresql.org (Postfix) with ESMTP id 26B829DCA21
	for <pgsql-performance-postgresql.org@localhost.postgresql.org>;
	Tue, 20 Dec 2005 09:26:06 -0400 (AST)
Received: from postgresql.org ([200.46.204.71])
	by localhost (av.hub.org [200.46.204.144]) (amavisd-new, port 10024)
	with ESMTP id 73941-09
	for <pgsql-performance-postgresql.org@localhost.postgresql.org>;
	Tue, 20 Dec 2005 09:26:09 -0400 (AST)
Received: from svr4.postgresql.org (svr4.postgresql.org [66.98.251.159])
	by postgresql.org (Postfix) with ESMTP id 56ADE9DC89E
	for <pgsql-performance@postgresql.org>;
	Tue, 20 Dec 2005 09:26:03 -0400 (AST)
Received: from out4.smtp.messagingengine.com (out4.smtp.messagingengine.com
	[66.111.4.28])
	by svr4.postgresql.org (Postfix) with ESMTP id 9D8555AF080
	for <pgsql-performance@postgresql.org>;
	Tue, 20 Dec 2005 13:26:07 +0000 (GMT)
Received: from frontend1.internal (mysql-sessions.internal [10.202.2.149])
	by frontend1.messagingengine.com (Postfix) with ESMTP id 5ADADD24423;
	Tue, 20 Dec 2005 08:26:04 -0500 (EST)
Received: from frontend2.messagingengine.com ([10.202.2.151])
	by frontend1.internal (MEProxy); Tue, 20 Dec 2005 08:26:04 -0500
X-Sasl-enc: b6+kA2bbCqnxPAJ+KlePLL3/ye/utufdHO43yWqcwkx9 1135085163
Received: from [149.182.235.91] (unknown [129.230.248.1])
	by frontend2.messagingengine.com (Postfix) with ESMTP id 63CCF5713C0;
	Tue, 20 Dec 2005 08:26:01 -0500 (EST)
Message-ID: <43A80668.1040604@diroussel.xsmail.com>
Date: Tue, 20 Dec 2005 13:26:00 +0000
From: David Roussel <pgsql-performance@diroussel.xsmail.com>
User-Agent: Mozilla Thunderbird 1.0.6 (Windows/20050716)
X-Accept-Language: en-us, en
MIME-Version: 1.0
To: David Lang <dlang@invendra.net>
Cc: pgsql-performance@postgresql.org
Subject: Re: filesystem performance with lots of files
References: <Pine.LNX.4.62.0512010616200.2807@qnivq.ynat.uz>
In-Reply-To: <Pine.LNX.4.62.0512010616200.2807@qnivq.ynat.uz>
Content-Type: multipart/alternative;
	boundary="------------050108030207020608010007"
X-Virus-Scanned: by amavisd-new at hub.org
X-Spam-Status: No, score=0.001 required=5 tests=[HTML_MESSAGE=0.001]
X-Spam-Score: 0.001
X-Spam-Level: 
X-Archive-Number: 200512/488
X-Sequence-Number: 16309

This is a multi-part message in MIME format.
--------------050108030207020608010007
Content-Type: text/plain; charset=ISO-8859-1; format=flowed
Content-Transfer-Encoding: 7bit

David Lang wrote:

>
>   ext3 has an option to make searching directories faster (htree), but 
> enabling it kills performance when you create files. And this doesn't 
> help with large files.
>
The ReiserFS white paper talks about the data structure he uses to store 
directories (some kind of tree), and he says it's quick to both read and 
write.  Don't forget if you find ls slow, that could just be ls, since 
it's ls, not the fs, that sorts this files into alphabetical order.

 > how long would it take to do a tar-ftp-untar cycle with no smarts

Note that you can do the taring, zipping, copying and untaring 
concurrentlt.  I can't remember the exactl netcat command line options, 
but it goes something like this

Box1:
tar czvf - myfiles/* | netcat myserver:12345

Box2:
netcat -listen 12345 | tar xzvf -

Not only do you gain from doing it all concurrently, but not writing a 
temp file means that disk seeks a reduced too if you have a one spindle 
machine.

Also condsider just copying files onto a network mount.  May not be as 
fast as the above, but will be faster than rsync, which has high CPU 
usage and thus not a good choice on a LAN.

Hmm, sorry this is not directly postgres anymore...

David

--------------050108030207020608010007
Content-Type: text/html; charset=ISO-8859-1
Content-Transfer-Encoding: 7bit

<!DOCTYPE html PUBLIC "-//W3C//DTD HTML 4.01 Transitional//EN">
<html>
<head>
  <meta content="text/html;charset=ISO-8859-1" http-equiv="Content-Type">
</head>
<body bgcolor="#ffffff" text="#333333">
David Lang wrote:<br>
<blockquote cite="midPine.LNX.4.62.0512010616200.2807@qnivq.ynat.uz"
 type="cite"><br>
&nbsp; ext3 has an option to make searching directories faster (htree), but
enabling it kills performance when you create files. And this doesn't
help with large files.
  <br>
  <br>
</blockquote>
The ReiserFS white paper talks about the data structure he uses to
store directories (some kind of tree), and he says it's quick to both
read and write.&nbsp; Don't forget if you find ls slow, that could just be
ls, since it's ls, not the fs, that sorts this files into alphabetical
order.<br>
<br>
&gt; how long would it take to do a tar-ftp-untar cycle with no smarts<br>
<br>
Note that you can do the taring, zipping, copying and untaring
concurrentlt.&nbsp; I can't remember the exactl netcat command line options,
but it goes something like this<br>
<br>
Box1:<br>
tar czvf - myfiles/* | netcat myserver:12345<br>
<br>
Box2:<br>
netcat -listen 12345 | tar xzvf -<br>
<br>
Not only do you gain from doing it all concurrently, but not writing a
temp file means that disk seeks a reduced too if you have a one spindle
machine.<br>
<br>
Also condsider just copying files onto a network mount.&nbsp; May not be as
fast as the above, but will be faster than rsync, which has high CPU
usage and thus not a good choice on a LAN.<br>
<br>
Hmm, sorry this is not directly postgres anymore...<br>
<br>
David<br>
</body>
</html>

--------------050108030207020608010007--