Received: from malur.postgresql.org ([217.196.149.56]) by arkaria.postgresql.org with esmtp (Exim 4.80) (envelope-from ) id 1UuSek-0004uF-OC for pgsql-hackers@arkaria.postgresql.org; Wed, 03 Jul 2013 19:23:58 +0000 Received: from localhost ([127.0.0.1] helo=postgresql.org) by malur.postgresql.org with smtp (Exim 4.80) (envelope-from ) id 1UuSek-0007mL-2v for pgsql-hackers@arkaria.postgresql.org; Wed, 03 Jul 2013 19:23:58 +0000 Received: from makus.postgresql.org ([2001:4800:7903:4::125]) by malur.postgresql.org with esmtp (Exim 4.80) (envelope-from ) id 1UuSej-0007mF-5o for pgsql-hackers@postgresql.org; Wed, 03 Jul 2013 19:23:57 +0000 Received: from mbx.knossos.net.nz ([202.160.48.10]) by makus.postgresql.org with esmtp (Exim 4.80) (envelope-from ) id 1UuSef-00014n-SZ for pgsql-hackers@postgresql.org; Wed, 03 Jul 2013 19:23:56 +0000 Received: from [10.1.1.3] (60-234-150-59.bitstream.orcon.net.nz [60.234.150.59]) (authenticated bits=0) by mbx.knossos.net.nz (8.14.4/8.14.4) with ESMTP id r63JNSmp023880 (version=TLSv1/SSLv3 cipher=DHE-RSA-AES256-SHA bits=256 verify=NOT); Thu, 4 Jul 2013 07:23:29 +1200 Message-ID: <51D47A17.6000809@archidevsys.co.nz> Date: Thu, 04 Jul 2013 07:23:03 +1200 From: Gavin Flower Organization: ArchiDevSys User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:17.0) Gecko/20130620 Thunderbird/17.0.7 MIME-Version: 1.0 To: Robert Haas CC: KONDO Mitsumasa , Heikki Linnakangas , PostgreSQL-development Subject: Re: Improvement of checkpoint IO scheduler for stable transaction responses References: <51B5AFB1.5030404@lab.ntt.co.jp> <51BDCB6C.1090507@vmware.com> <51BEB6F5.7070907@lab.ntt.co.jp> <51C40EF4.1080201@lab.ntt.co.jp> <51C9D03B.10107@vmware.com> <51C9FD5C.5050706@vmware.com> <51CD873A.5070705@lab.ntt.co.jp> <51D3DE55.6030901@lab.ntt.co.jp> In-Reply-To: Content-Type: multipart/alternative; boundary="------------000303020506060102090009" X-Pg-Spam-Score: -1.9 (-) List-Archive: List-Help: List-ID: List-Owner: List-Post: List-Subscribe: List-Unsubscribe: X-Mailing-List: pgsql-hackers Precedence: bulk Sender: pgsql-hackers-owner@postgresql.org This is a multi-part message in MIME format. --------------000303020506060102090009 Content-Type: text/plain; charset=ISO-8859-1; format=flowed Content-Transfer-Encoding: 7bit On 04/07/13 01:31, Robert Haas wrote: > On Wed, Jul 3, 2013 at 4:18 AM, KONDO Mitsumasa > wrote: >> I tested and changed segsize=0.25GB which is max partitioned table file size and >> default setting is 1GB in configure option (./configure --with-segsize=0.25). >> Because I thought that small segsize is good for fsync phase and background disk >> write in OS in checkpoint. I got significant improvements in DBT-2 result! > This is interesting. Unfortunately, it has a significant downside: > potentially, there will be a lot more files in the data directory. As > it is, the number of files that exist there today has caused > performance problems for some of our customers. I'm not sure off-hand > to what degree those problems have been related to overall inode > consumption vs. the number of files in the same directory. > > If the problem is mainly with number of of files in the same > directory, we could consider revising our directory layout. Instead > of: > > base/${DBOID}/${RELFILENODE}_{FORK} > > We could have: > > base/${DBOID}/${FORK}/${RELFILENODE} > > That would move all the vm and fsm forks to separate directories, > which would cut down the number of files in the main-fork directory > significantly. That might be worth doing independently of the issue > you're raising here. For large clusters, you'd even want one more > level to keep the directories from getting too big: > > base/${DBOID}/${FORK}/${X}/${RELFILENODE} > > ...where ${X} is two hex digits, maybe just the low 16 bits of the > relfilenode number. But this would be not as good for small clusters > where you'd end up with oodles of little-tiny directories, and I'm not > sure it'd be practical to smoothly fail over from one system to the > other. > 16 bits ==> 4 hex digits Could you perhaps start with 1 hex digit, and automagically increase it to 2, 3, .. as needed? There could be a status file at that level, that would indicate the current number of hex digits, plus a temporary mapping file when in transition. Cheers, Gavin --------------000303020506060102090009 Content-Type: text/html; charset=ISO-8859-1 Content-Transfer-Encoding: 7bit
On 04/07/13 01:31, Robert Haas wrote:
On Wed, Jul 3, 2013 at 4:18 AM, KONDO Mitsumasa
<kondo.mitsumasa@lab.ntt.co.jp> wrote:

I tested and changed segsize=0.25GB which is max partitioned table file size and
default setting is 1GB in configure option (./configure --with-segsize=0.25).
Because I thought that small segsize is good for fsync phase and background disk
write in OS in checkpoint. I got significant improvements in DBT-2 result!

This is interesting.  Unfortunately, it has a significant downside:
potentially, there will be a lot more files in the data directory.  As
it is, the number of files that exist there today has caused
performance problems for some of our customers.  I'm not sure off-hand
to what degree those problems have been related to overall inode
consumption vs. the number of files in the same directory.

If the problem is mainly with number of of files in the same
directory, we could consider revising our directory layout.  Instead
of:

base/${DBOID}/${RELFILENODE}_{FORK}

We could have:

base/${DBOID}/${FORK}/${RELFILENODE}

That would move all the vm and fsm forks to separate directories,
which would cut down the number of files in the main-fork directory
significantly.  That might be worth doing independently of the issue
you're raising here.  For large clusters, you'd even want one more
level to keep the directories from getting too big:

base/${DBOID}/${FORK}/${X}/${RELFILENODE}

...where ${X} is two hex digits, maybe just the low 16 bits of the
relfilenode number.  But this would be not as good for small clusters
where you'd end up with oodles of little-tiny directories, and I'm not
sure it'd be practical to smoothly fail over from one system to the
other.
16 bits ==> 4 hex digits

Could you perhaps start with 1 hex digit, and automagically increase it to 2, 3, .. as needed?  There could be a status file at that level, that would indicate the current number of hex digits, plus a temporary mapping file when in transition.

Cheers,
Gavin
 --------------000303020506060102090009--