Message-ID: <4B39515D.6050309@2ndquadrant.com>
Date: Mon, 28 Dec 2009 19:46:21 -0500
From: Greg Smith <greg@2ndquadrant.com>
User-Agent: Thunderbird 2.0.0.23 (X11/20090817)
MIME-Version: 1.0
To: Andres Freund <andres@anarazel.de>
CC: pgsql-hackers@postgresql.org, Tom Lane <tgl@sss.pgh.pa.us>,
	pgsql-performance@postgresql.org,
	Michael Clemmons <glassresistor@gmail.com>,
	Hannu Krosing <hannu@2ndquadrant.com>,
	Scott Marlowe <scott.marlowe@gmail.com>
Subject: Re: [HACKERS] Faster CREATE DATABASE by delaying fsync
	(was 8.4.1 ubuntu karmic slow createdb)
References: <4427a97a0912101235x3ade510blcc00d5ce9883c6b6@mail.gmail.com>
	<200912282354.51892.andres@anarazel.de>
	<3454.1262041588@sss.pgh.pa.us>
	<200912290020.35843.andres@anarazel.de>
In-Reply-To: <200912290020.35843.andres@anarazel.de>
Content-Type: text/plain; charset=ISO-8859-15; format=flowed
Content-Transfer-Encoding: 7bit

Andres Freund wrote:
> As I said the real benefit only occurred after adding posix_fadvise(.., 
> FADV_DONTNEED) which is somewhat plausible, because i.e. the directory entries 
> don't need to get scheduled for every file and because the kernel can reorder a 
> whole directory nearly sequentially. Without the advice it the kernel doesn't 
> know in time that it should write that data back and it wont do it for 5 
> seconds by default on linux or such...
>   
I know they just fiddled with the logic in the last release, but for 
most of the Linux kernels out there now pdflush wakes up every 5 seconds 
by default.  But typically it only worries about writing things that 
have been in the queue for 30 seconds or more until you've filled quite 
a bit of memory, so that's also an interesting number.  I tried to 
document the main tunables here and describe how they fit together at 
http://www.westnet.com/~gsmith/content/linux-pdflush.htm

It would be interesting to graph the "Dirty" and "Writeback" figures in 
/proc/meminfo over time with and without this patch in place.  That 
should make it obvious what the kernel is doing differently in the two 
cases.

-- 
Greg Smith    2ndQuadrant   Baltimore, MD
PostgreSQL Training, Services and Support
greg@2ndQuadrant.com  www.2ndQuadrant.com