DomainKey-Signature: a=rsa-sha1; c=nofws; d=gmail.com; s=gamma;
	h=from:to:subject:date:user-agent:mime-version:message-id
	:content-type:content-transfer-encoding;
	b=mBLRx7Sdc3MPNS9xqCfCfsnNPhr8Bf7n0ykPRmhF6BYT+jkZlMuhyY59p+ZDvlgLJl
	vDICZotRSgzcaC497mZDdtNjVnqYxHlLly2O96hxrdUjNdeemfwTpwfcpW/HF3dntZYe
	/bm7BLZd5T6kRCRQjFJ+iNiTnIsifIYGyX4yM=
From: Marc Cousin <cousinmarc@gmail.com>
To: pgsql-performance@postgresql.org
Subject: performance regression with Linux 2.6.33 and glibc 2.12
Date: Fri, 4 Jun 2010 15:39:03 +0200
User-Agent: KMail/1.13.3 (Linux/2.6.35-rc1-rc; KDE/4.4.3; x86_64; ; )
MIME-Version: 1.0
Message-Id: <201006041539.03868.cousinmarc@gmail.com>
Content-Type: text/plain;
  charset="us-ascii"
Content-Transfer-Encoding: 7bit

Hi.

I hope I'm not going to expose an already known problem, but I couldn't find 
it mailing list archives (I only found http://archives.postgresql.org/pgsql-
hackers/2009-12/msg01543.php).

On one of my (non production) machines, I've just seen a very big performance 
regression (I was doing a very simple insert test). I had an 'old' 8.4 
postgresql compiled a few month ago, performing very well, and my 'bleeding 
edge' 9.0, doing the same insert very slowly.

I managed to find the cause of the regression : with Linux 2.6.33, O_DSYNC is 
now available. With glibc 2.12, O_DSYNC is available in userspace. Having both 
(they are both very new, 2.12 isn't even official on glibc website), my new 
build defaulted to open_datasync. The problem is that it is much slower. I 
tested it on 2 small machines (no big raid, just basic machines, with SATA or 
software RAID).

Here is the trivial test :
The configuration is the default configuration, just after initdb

CREATE TABLE test (a int);
CREATE INDEX idxtest on test (a);


with wal_sync_method = open_datasync (new default)

marc=# INSERT INTO test SELECT generate_series(1,100000);
INSERT 0 100000
Time: 16083,912 ms

with wal_sync_method = fdatasync (old default)

marc=# INSERT INTO test SELECT generate_series(1,100000);
INSERT 0 100000
Time: 954,000 ms

Doing synthetic benchmarks with test_fsync:

open_datasync performance, glibc 2.12, 2.6.34, 1 SATA drive

Simple 8k write timing:
write                           0.037511

Compare file sync methods using one 8k write:
open_datasync write            56.998797
open_sync write               168.653995
write, fdatasync               55.359279
write, fsync                  166.854911

Compare file sync methods using two 8k writes:
open_datasync write, write    113.342738
open_sync write, write        339.066883
write, write, fdatasync        57.336820
write, write, fsync           166.847923

Compare open_sync sizes:                                                                                                                                                                       
16k open_sync write           169.423723                                                                                                                                               
2 8k open_sync writes         336.457119                                                                                                                                               

Compare fsync times on write() and new file descriptors (if the times
are similar, fsync() can sync data written on a different descriptor):
write, fsync, close           166.264048
write, close, fsync           168.702035

This is it, I just wanted to raise an alert on this: the degradation was 16-
fold with this test. We wont see linux 2.6.33 + glibc 2.12 in production 
before months (I hope), but shouldn't PostgreSQL use fdatasync by default with 
Linux, seeing the result ?

By the way, I re-did my tests with both 2.6.33, 2.6.34 and 2.6.35-rc1 and got 
the exact same result (O_DSYNC there, obviously, but also the performance 
degradation).

Cheers

Marc