pg.ddx.io pgsql-performance@postgresql.org mailing list archive
help / color / mirror / Atom feedperformance regression with Linux 2.6.33 and glibc 2.12
4+ messages / 3 participants
[nested] [flat]
* performance regression with Linux 2.6.33 and glibc 2.12
@ 2010-06-04 13:39 Marc Cousin <cousinmarc@gmail.com>
0 siblings, 1 reply; 4+ messages in thread
From: Marc Cousin @ 2010-06-04 13:39 UTC (permalink / raw)
To: pgsql-performance
Hi.
I hope I'm not going to expose an already known problem, but I couldn't find
it mailing list archives (I only found http://archives.postgresql.org/pgsql-
hackers/2009-12/msg01543.php).
On one of my (non production) machines, I've just seen a very big performance
regression (I was doing a very simple insert test). I had an 'old' 8.4
postgresql compiled a few month ago, performing very well, and my 'bleeding
edge' 9.0, doing the same insert very slowly.
I managed to find the cause of the regression : with Linux 2.6.33, O_DSYNC is
now available. With glibc 2.12, O_DSYNC is available in userspace. Having both
(they are both very new, 2.12 isn't even official on glibc website), my new
build defaulted to open_datasync. The problem is that it is much slower. I
tested it on 2 small machines (no big raid, just basic machines, with SATA or
software RAID).
Here is the trivial test :
The configuration is the default configuration, just after initdb
CREATE TABLE test (a int);
CREATE INDEX idxtest on test (a);
with wal_sync_method = open_datasync (new default)
marc=# INSERT INTO test SELECT generate_series(1,100000);
INSERT 0 100000
Time: 16083,912 ms
with wal_sync_method = fdatasync (old default)
marc=# INSERT INTO test SELECT generate_series(1,100000);
INSERT 0 100000
Time: 954,000 ms
Doing synthetic benchmarks with test_fsync:
open_datasync performance, glibc 2.12, 2.6.34, 1 SATA drive
Simple 8k write timing:
write 0.037511
Compare file sync methods using one 8k write:
open_datasync write 56.998797
open_sync write 168.653995
write, fdatasync 55.359279
write, fsync 166.854911
Compare file sync methods using two 8k writes:
open_datasync write, write 113.342738
open_sync write, write 339.066883
write, write, fdatasync 57.336820
write, write, fsync 166.847923
Compare open_sync sizes:
16k open_sync write 169.423723
2 8k open_sync writes 336.457119
Compare fsync times on write() and new file descriptors (if the times
are similar, fsync() can sync data written on a different descriptor):
write, fsync, close 166.264048
write, close, fsync 168.702035
This is it, I just wanted to raise an alert on this: the degradation was 16-
fold with this test. We wont see linux 2.6.33 + glibc 2.12 in production
before months (I hope), but shouldn't PostgreSQL use fdatasync by default with
Linux, seeing the result ?
By the way, I re-did my tests with both 2.6.33, 2.6.34 and 2.6.35-rc1 and got
the exact same result (O_DSYNC there, obviously, but also the performance
degradation).
Cheers
Marc
^ permalink raw reply [nested|flat] 4+ messages in thread
* Re: performance regression with Linux 2.6.33 and glibc 2.12
@ 2010-06-04 13:59 Tom Lane <tgl@sss.pgh.pa.us>
parent: Marc Cousin <cousinmarc@gmail.com>
0 siblings, 2 replies; 4+ messages in thread
From: Tom Lane @ 2010-06-04 13:59 UTC (permalink / raw)
To: Marc Cousin <cousinmarc@gmail.com>; +Cc: pgsql-performance
Marc Cousin <cousinmarc@gmail.com> writes:
> I hope I'm not going to expose an already known problem, but I couldn't find
> it mailing list archives (I only found http://archives.postgresql.org/pgsql-
> hackers/2009-12/msg01543.php).
You sure this isn't the well-known "ext4 actually implements fsync
where ext3 didn't" issue?
regards, tom lane
^ permalink raw reply [nested|flat] 4+ messages in thread
* Re: performance regression with Linux 2.6.33 and glibc 2.12
@ 2010-06-04 15:25 Andres Freund <andres@anarazel.de>
parent: Tom Lane <tgl@sss.pgh.pa.us>
1 sibling, 0 replies; 4+ messages in thread
From: Andres Freund @ 2010-06-04 15:25 UTC (permalink / raw)
To: pgsql-performance; +Cc: Tom Lane <tgl@sss.pgh.pa.us>; Marc Cousin <cousinmarc@gmail.com>
On Friday 04 June 2010 15:59:05 Tom Lane wrote:
> Marc Cousin <cousinmarc@gmail.com> writes:
> > I hope I'm not going to expose an already known problem, but I couldn't
> > find it mailing list archives (I only found
> > http://archives.postgresql.org/pgsql- hackers/2009-12/msg01543.php).
>
> You sure this isn't the well-known "ext4 actually implements fsync
> where ext3 didn't" issue?
I doubt it. It reads to me like he is testing the two methods on the same
installation with the same kernel
> > with wal_sync_method = open_datasync (new default)
> > marc=# INSERT INTO test SELECT generate_series(1,100000);
> > INSERT 0 100000
> > Time: 16083,912 ms
> >
> > with wal_sync_method = fdatasync (old default)
> >
> > marc=# INSERT INTO test SELECT generate_series(1,100000);
> > INSERT 0 100000
> > Time: 954,000 ms
Its not actually surprising that in such a open_datasync is hugely slower than
fdatasync. With open_datasync every single write will be synchronous, very
likely not reordered/batched/whatever. In contrast to that with fdatasync it
will only synced in way much bigger batches.
Or am I missing something?
I always thought the synchronous write methods to be a fallback kludge and
didnt realize its actually the preferred method...
Andres
^ permalink raw reply [nested|flat] 4+ messages in thread
* Re: performance regression with Linux 2.6.33 and glibc 2.12
@ 2010-06-04 15:29 Marc Cousin <cousinmarc@gmail.com>
parent: Tom Lane <tgl@sss.pgh.pa.us>
1 sibling, 0 replies; 4+ messages in thread
From: Marc Cousin @ 2010-06-04 15:29 UTC (permalink / raw)
To: Tom Lane <tgl@sss.pgh.pa.us>; +Cc: pgsql-performance
The Friday 04 June 2010 15:59:05, Tom Lane wrote :
> Marc Cousin <cousinmarc@gmail.com> writes:
> > I hope I'm not going to expose an already known problem, but I couldn't
> > find it mailing list archives (I only found
> > http://archives.postgresql.org/pgsql- hackers/2009-12/msg01543.php).
>
> You sure this isn't the well-known "ext4 actually implements fsync
> where ext3 didn't" issue?
>
> regards, tom lane
Everything is ext4. So I should have fsync working with write barriers on all
the tests.
I don't think this problem is of the same kind: I think it is really because
of O_DSYNC appearing on 2.6.33, and PostgreSQL using it by default now. If my
filesystem was lying to me about barriers, I should take no more performance
hit with open_datasync than with fdatasync, should I ?
^ permalink raw reply [nested|flat] 4+ messages in thread
end of thread, other threads:[~2010-06-04 15:29 UTC | newest]
Thread overview: 4+ messages (download: mbox mbox.gz follow: Atom feed)
-- links below jump to the message on this page --
2010-06-04 13:39 performance regression with Linux 2.6.33 and glibc 2.12 Marc Cousin <cousinmarc@gmail.com>
2010-06-04 13:59 ` Tom Lane <tgl@sss.pgh.pa.us>
2010-06-04 15:25 ` Andres Freund <andres@anarazel.de>
2010-06-04 15:29 ` Marc Cousin <cousinmarc@gmail.com>
This inbox is served by DDX for PostgreSQL; see mirroring instructions
for how to clone and mirror all data and code used for this inbox