pg.ddx.io pgsql-performance@postgresql.org mailing list archive
help / color / mirror / Atom feedPerformance increase with elevator=deadline
10+ messages / 9 participants
[nested] [flat]
* Performance increase with elevator=deadline
@ 2008-04-11 11:22 Albe Laurenz <laurenz.albe@wien.gv.at>
0 siblings, 2 replies; 10+ messages in thread
From: Albe Laurenz @ 2008-04-11 11:22 UTC (permalink / raw)
To: pgsql-performance
This refers to the performance problem reported in
http://archives.postgresql.org/pgsql-performance/2008-04/msg00052.php
After some time of trial and error we found that changing the I/O scheduling
algorithm to "deadline" improved I/O performance by a factor 4 (!) for
this specific load test.
It seems that the bottleneck in this case was actually in the Linux kernel.
Since performance statements are useless without a description of
the system and the type of load, I'll send a few details to make this
report more useful for the archives:
The machine is a PC with 8 AMD Opteron 885 CPUs and 32 GB RAM, attached to
a HP EVA 8100 storage system with 72 disks.
We are running 64-bit Linux 2.6.18-53.1.6.el5 from RedHat Enterprise 5.1.
The I/O queue depth is set to 64.
Our benchmark tools show a possible I/O performance of about 11000 transactions
per second for random scattered reads of 8k blocks.
PostgreSQL version is 8.2.4.
The database system is a cluster with 6 databases containing tables up
to a couple of GB in size. The whole database cluster takes about
200 GB of storage.
The database load is a set of read-only statements, several of which have
miserable execution plans and perform table and index scans.
With the default I/O scheduler we observe a performance of about
600 I/O transactions or 7 MB per second.
After booting with elevator=deadline both values increase by a factor
of up to 4 and the query response times sink drastically.
Yours,
Laurenz Albe
^ permalink raw reply [nested|flat] 10+ messages in thread
* Re: Performance increase with elevator=deadline
@ 2008-04-11 14:47 Gregory Stark <stark@enterprisedb.com>
parent: Albe Laurenz <laurenz.albe@wien.gv.at>
1 sibling, 1 reply; 10+ messages in thread
From: Gregory Stark @ 2008-04-11 14:47 UTC (permalink / raw)
To: Albe Laurenz <laurenz.albe@wien.gv.at>; +Cc: pgsql-performance
"Albe Laurenz" <laurenz.albe@wien.gv.at> writes:
> This refers to the performance problem reported in
> http://archives.postgresql.org/pgsql-performance/2008-04/msg00052.php
>
> After some time of trial and error we found that changing the I/O scheduling
> algorithm to "deadline" improved I/O performance by a factor 4 (!) for
> this specific load test.
What was the algorithm before?
--
Gregory Stark
EnterpriseDB http://www.enterprisedb.com
Ask me about EnterpriseDB's On-Demand Production Tuning
^ permalink raw reply [nested|flat] 10+ messages in thread
* Re: Performance increase with elevator=deadline
@ 2008-04-11 15:09 Jeff <threshar@torgo.978.org>
parent: Albe Laurenz <laurenz.albe@wien.gv.at>
1 sibling, 2 replies; 10+ messages in thread
From: Jeff @ 2008-04-11 15:09 UTC (permalink / raw)
To: Albe Laurenz <laurenz.albe@wien.gv.at>; +Cc: pgsql-performance
On Apr 11, 2008, at 7:22 AM, Albe Laurenz wrote:
> After some time of trial and error we found that changing the I/O
> scheduling
> algorithm to "deadline" improved I/O performance by a factor 4 (!) for
> this specific load test.
>
I was inspired once again to look into this - as I'm recently hitting
some glass ceilings with my machines.
I have a little app I wrote called pgiosim (its on pgfoundry - http://pgfoundry.org/projects/pgiosim)
that basically just opens some files, and does random seeks and 8kB
reads, much like what our beloved PG does.
Using 4 of these with a dataset of about 30GB across a few files
(Machine has 8GB mem) I went from around 100 io/sec to 330 changing to
noop. Quite an improvement. If you have a decent controller CFQ is
not what you want. I tried deadline as well and it was a touch
slower. The controller is a 3ware 9550sx with 4 disks in a raid10.
I'll be trying this out on the big array later today. I found it
suprising this info wasn't more widespread (the use of CFQ on a good
controller).
it also seems changing elevators on the fly works fine (echo
schedulername > /sys/block/.../queue/scheduler I admit I sat there
flipping back and forth going "disk go fast.. disk go slow.. disk go
fast... " :)
--
Jeff Trout <jeff@jefftrout.com>
http://www.stuarthamm.net/
http://www.dellsmartexitin.com/
^ permalink raw reply [nested|flat] 10+ messages in thread
* Re: Performance increase with elevator=deadline
@ 2008-04-11 16:40 Matthew <matthew@flymine.org>
parent: Jeff <threshar@torgo.978.org>
1 sibling, 1 reply; 10+ messages in thread
From: Matthew @ 2008-04-11 16:40 UTC (permalink / raw)
To: pgsql-performance
On Fri, 11 Apr 2008, Jeff wrote:
> Using 4 of these with a dataset of about 30GB across a few files (Machine has
> 8GB mem) I went from around 100 io/sec to 330 changing to noop. Quite an
> improvement. If you have a decent controller CFQ is not what you want. I
> tried deadline as well and it was a touch slower. The controller is a 3ware
> 9550sx with 4 disks in a raid10.
I ran Greg's fadvise test program a while back on a 12-disc array. The
three schedulers (deadline, noop, anticipatory) all performed pretty-much
the same, with the fourth (cfq, the default) being consistently slower.
> it also seems changing elevators on the fly works fine (echo schedulername >
> /sys/block/.../queue/scheduler I admit I sat there flipping back and forth
> going "disk go fast.. disk go slow.. disk go fast... " :)
Oh Homer Simpson, your legacy lives on.
Matthew
--
I suppose some of you have done a Continuous Maths course. Yes? Continuous
Maths? <menacing stares from audience> Whoah, it was like that, was it!
-- Computer Science Lecturer
^ permalink raw reply [nested|flat] 10+ messages in thread
* Re: Performance increase with elevator=deadline
@ 2008-04-11 18:03 Craig Ringer <craig@postnewspapers.com.au>
parent: Matthew <matthew@flymine.org>
0 siblings, 2 replies; 10+ messages in thread
From: Craig Ringer @ 2008-04-11 18:03 UTC (permalink / raw)
To: Matthew <matthew@flymine.org>; +Cc: pgsql-performance
Matthew wrote:
> On Fri, 11 Apr 2008, Jeff wrote:
>> Using 4 of these with a dataset of about 30GB across a few files
>> (Machine has 8GB mem) I went from around 100 io/sec to 330 changing to
>> noop. Quite an improvement. If you have a decent controller CFQ is
>> not what you want. I tried deadline as well and it was a touch
>> slower. The controller is a 3ware 9550sx with 4 disks in a raid10.
>
> I ran Greg's fadvise test program a while back on a 12-disc array. The
> three schedulers (deadline, noop, anticipatory) all performed
> pretty-much the same, with the fourth (cfq, the default) being
> consistently slower.
I use CFQ on some of my servers, despite the fact that it's often slower
in total throughput terms, because it delivers much more predictable I/O
latencies that help prevent light I/O processes being starved by heavy
I/O processes. In particular, an Linux terminal server used at work has
taken a lot of I/O tuning before it delivers even faintly acceptable I/O
latencies under any sort of load.
Bounded I/O latency at the expense of throughput is not what you usually
want on a DB server, where throughput is king, so I'm not at all
surprised that CFQ performs poorly for PostgreSQL. I've done no testing
on that myself, though, because with my DB size and the nature of my
queries most of them are CPU bound anyway.
Speaking of I/O performance with PostgreSQL, has anybody here done any
testing to compare results with LVM to results with the same filesystem
on a conventionally partitioned or raw volume? I'd probably use LVM even
at a performance cost because of its admin benefits, but I'd be curious
if there is any known cost for use with Pg. I've never been able to
measure one with other workloads.
--
Craig Ringer
^ permalink raw reply [nested|flat] 10+ messages in thread
* Re: Performance increase with elevator=deadline
@ 2008-04-11 19:56 Greg Smith <gsmith@gregsmith.com>
parent: Craig Ringer <craig@postnewspapers.com.au>
1 sibling, 0 replies; 10+ messages in thread
From: Greg Smith @ 2008-04-11 19:56 UTC (permalink / raw)
To: Craig Ringer <craig@postnewspapers.com.au>; +Cc: pgsql-performance
On Sat, 12 Apr 2008, Craig Ringer wrote:
> Speaking of I/O performance with PostgreSQL, has anybody here done any
> testing to compare results with LVM to results with the same filesystem on a
> conventionally partitioned or raw volume?
There was some chatter on this topic last year; a quick search finds
http://archives.postgresql.org/pgsql-performance/2007-06/msg00005.php
which is a fair statement of the situation. I don't recall any specific
benchmarks.
--
* Greg Smith gsmith@gregsmith.com http://www.gregsmith.com Baltimore, MD
^ permalink raw reply [nested|flat] 10+ messages in thread
* Re: Performance increase with elevator=deadline
@ 2008-04-15 13:27 Florian Weimer <fweimer@bfk.de>
parent: Jeff <threshar@torgo.978.org>
1 sibling, 1 reply; 10+ messages in thread
From: Florian Weimer @ 2008-04-15 13:27 UTC (permalink / raw)
To: Jeff <threshar@real.jefftrout.com>; +Cc: Albe Laurenz <laurenz.albe@wien.gv.at>; pgsql-performance
* Jeff:
> Using 4 of these with a dataset of about 30GB across a few files
> (Machine has 8GB mem) I went from around 100 io/sec to 330 changing to
> noop. Quite an improvement. If you have a decent controller CFQ is
> not what you want. I tried deadline as well and it was a touch
> slower. The controller is a 3ware 9550sx with 4 disks in a raid10.
>
> I'll be trying this out on the big array later today. I found it
> suprising this info wasn't more widespread (the use of CFQ on a good
> controller).
3ware might be a bit special because the controller has got very deep
queues on its own, so many assumptions of the kernel I/O schedulers do
not seem to apply. Toying with the kernel/controller queue depths
might help, but I haven't done real benchmarks to see if it's actually
a difference.
A few days ago, I experienced this: On a machine with a 3ware
controller, a simple getxattr call on a file in an uncontended
directory took several minutes because a PostgreSQL dump process was
running in the background (and some other activity of a legacy
database which caused frequent fdatasync calls). Clearly, this is
unacceptable, and I've since switched to the deadline scheduler, too.
So far, this particular behavior hasn't occurred again.
--
Florian Weimer <fweimer@bfk.de>
BFK edv-consulting GmbH http://www.bfk.de/
Kriegsstraße 100 tel: +49-721-96201-1
D-76133 Karlsruhe fax: +49-721-96201-99
^ permalink raw reply [nested|flat] 10+ messages in thread
* Re: Performance increase with elevator=deadline
@ 2008-04-15 19:08 david@lang.hm
parent: Florian Weimer <fweimer@bfk.de>
0 siblings, 0 replies; 10+ messages in thread
From: david@lang.hm @ 2008-04-15 19:08 UTC (permalink / raw)
To: Florian Weimer <fweimer@bfk.de>; +Cc: Jeff <threshar@threshar.is-a-geek.com>; Albe Laurenz <laurenz.albe@wien.gv.at>; pgsql-performance
On Tue, 15 Apr 2008, Florian Weimer wrote:
> * Jeff:
>
>> Using 4 of these with a dataset of about 30GB across a few files
>> (Machine has 8GB mem) I went from around 100 io/sec to 330 changing to
>> noop. Quite an improvement. If you have a decent controller CFQ is
>> not what you want. I tried deadline as well and it was a touch
>> slower. The controller is a 3ware 9550sx with 4 disks in a raid10.
>>
>> I'll be trying this out on the big array later today. I found it
>> suprising this info wasn't more widespread (the use of CFQ on a good
>> controller).
>
> 3ware might be a bit special because the controller has got very deep
> queues on its own, so many assumptions of the kernel I/O schedulers do
> not seem to apply. Toying with the kernel/controller queue depths
> might help, but I haven't done real benchmarks to see if it's actually
> a difference.
>
> A few days ago, I experienced this: On a machine with a 3ware
> controller, a simple getxattr call on a file in an uncontended
> directory took several minutes because a PostgreSQL dump process was
> running in the background (and some other activity of a legacy
> database which caused frequent fdatasync calls). Clearly, this is
> unacceptable, and I've since switched to the deadline scheduler, too.
> So far, this particular behavior hasn't occurred again.
one other thing to watch out for. up until very recent kernels (2.6.23 or
2.6.24) it was possible for one very busy block device to starve other
block devices. they added isolation of queues for different block devices,
but I've seen reports that the isolation can end up throttling high
performance devices as a result. I haven't had time to really dig into
this, but there are tuning knobs available to adjust the que space
available to different devices and the reports are significantly better
activity on a tuned system.
David Lang
^ permalink raw reply [nested|flat] 10+ messages in thread
* Re: Performance increase with elevator=deadline
@ 2008-04-16 05:27 Albe Laurenz <laurenz.albe@wien.gv.at>
parent: Gregory Stark <stark@enterprisedb.com>
0 siblings, 0 replies; 10+ messages in thread
From: Albe Laurenz @ 2008-04-16 05:27 UTC (permalink / raw)
To: Gregory Stark *EXTERN* <stark@enterprisedb.com>; +Cc: pgsql-performance
Gregory Stark wrote:
>> After some time of trial and error we found that changing the I/O scheduling
>> algorithm to "deadline" improved I/O performance by a factor 4 (!) for
>> this specific load test.
>
> What was the algorithm before?
The default algorithm, CFQ I think it is.
Yours,
Laurenz Albe
^ permalink raw reply [nested|flat] 10+ messages in thread
* Re: Performance increase with elevator=deadline
@ 2008-04-18 14:38 Enrico Sirola <enrico.sirola@gmail.com>
parent: Craig Ringer <craig@postnewspapers.com.au>
1 sibling, 0 replies; 10+ messages in thread
From: Enrico Sirola @ 2008-04-18 14:38 UTC (permalink / raw)
To: Craig Ringer <craig@postnewspapers.com.au>; +Cc: Matthew <matthew@flymine.org>; pgsql-performance
Hi,
Il giorno 11/apr/08, alle ore 20:03, Craig Ringer ha scritto:
>
> Speaking of I/O performance with PostgreSQL, has anybody here done
> any testing to compare results with LVM to results with the same
> filesystem on a conventionally partitioned or raw volume? I'd
> probably use LVM even at a performance cost because of its admin
> benefits, but I'd be curious if there is any known cost for use with
> Pg. I've never been able to measure one with other workloads.
I performed some tests some time ago. LVM is significantly slower.
The disk subsystem is a HP P400/512MB battery-backed controller with 4
disks in raid 10.
See the tests:
ext3 tests:
bonnie++ -s 16000 -u 0 -f -b
=
=
=
=
=
=
=
========================================================================
Version 1.03 ------Sequential Output------ --Sequential Input-
--Random-
-Per Chr- --Block-- -Rewrite- -Per Chr- --Block--
--Seeks--
Machine Size K/sec %CP K/sec %CP K/sec %CP K/sec %CP K/sec
%CP /sec %CP
16000M 153637 50 78895 17 204124
17 700.6 1
------Sequential Create------ --------Random
Create--------
-Create-- --Read--- -Delete-- -Create-- --Read---
-Delete--
files /sec %CP /sec %CP /sec %CP /sec %CP /sec
%CP /sec %CP
16 2233 10 +++++ +++ 2606 8 2255 10 +++++ ++
+ 2584 7
16000M,,,153637,50,78895,17,,,204124,17,700.6,1,16,2233,10,+++++,+++,
2606,8,2255,10,+++++,+++,2584,7
bonnie++ -s 16000 -u 0 -f
=
=
=
=
=
=
=
========================================================================
Version 1.03 ------Sequential Output------ --Sequential Input-
--Random-
-Per Chr- --Block-- -Rewrite- -Per Chr- --Block--
--Seeks--
Machine Size K/sec %CP K/sec %CP K/sec %CP K/sec %CP K/sec
%CP /sec %CP
16000M 162223 51 77277 17 207055
17 765.3 1
------Sequential Create------ --------Random
Create--------
-Create-- --Read--- -Delete-- -Create-- --Read---
-Delete--
files /sec %CP /sec %CP /sec %CP /sec %CP /sec
%CP /sec %CP
16 +++++ +++ +++++ +++ +++++ +++ +++++ +++ +++++ +++
+++++ +++
16000M,,,162223,51,77277,17,,,207055,17,765.3,1,16,+++++,+++,+++++,+++,
+++++,+++,+++++,+++,+++++,+++,+++++,+++
=
=
=
=
=
=
=
========================================================================
LVM tests:
bonnie++ -u 0 -f -s 16000 -b
Version 1.03 ------Sequential Output------ --Sequential Input-
--Random-
-Per Chr- --Block-- -Rewrite- -Per Chr- --Block--
--Seeks--
Machine Size K/sec %CP K/sec %CP K/sec %CP K/sec %CP K/sec
%CP /sec %CP
16000M 153768 52 53420 13 177414
15 699.8 1
------Sequential Create------ --------Random
Create--------
-Create-- --Read--- -Delete-- -Create-- --Read---
-Delete--
files /sec %CP /sec %CP /sec %CP /sec %CP /sec
%CP /sec %CP
16 2158 9 +++++ +++ 2490 7 2177 9 +++++ ++
+ 2460 7
16000M,,,153768,52,53420,13,,,177414,15,699.8,1,16,2158,9,+++++,+++,
2490,7,2177,9,+++++,+++,2460,7
bonnie++ -u 0 -f -s 16000
Version 1.03 ------Sequential Output------ --Sequential Input-
--Random-
-Per Chr- --Block-- -Rewrite- -Per Chr- --Block--
--Seeks--
Machine Size K/sec %CP K/sec %CP K/sec %CP K/sec %CP K/sec
%CP /sec %CP
16000M 161476 53 54904 13 171693
14 774.3 1
------Sequential Create------ --------Random
Create--------
-Create-- --Read--- -Delete-- -Create-- --Read---
-Delete--
files /sec %CP /sec %CP /sec %CP /sec %CP /sec
%CP /sec %CP
16 +++++ +++ +++++ +++ +++++ +++ +++++ +++ +++++ +++
+++++ +++
16000M,,,161476,53,54904,13,,,171693,14,774.3,1,16,+++++,+++,+++++,+++,
+++++,+++,+++++,+++,+++++,+++,+++++,+++
Bye,
e.
^ permalink raw reply [nested|flat] 10+ messages in thread
end of thread, other threads:[~2008-04-18 14:38 UTC | newest]
Thread overview: 10+ messages (download: mbox mbox.gz follow: Atom feed)
-- links below jump to the message on this page --
2008-04-11 11:22 Performance increase with elevator=deadline Albe Laurenz <laurenz.albe@wien.gv.at>
2008-04-11 14:47 ` Gregory Stark <stark@enterprisedb.com>
2008-04-16 05:27 ` Albe Laurenz <laurenz.albe@wien.gv.at>
2008-04-11 15:09 ` Jeff <threshar@torgo.978.org>
2008-04-11 16:40 ` Matthew <matthew@flymine.org>
2008-04-11 18:03 ` Craig Ringer <craig@postnewspapers.com.au>
2008-04-11 19:56 ` Greg Smith <gsmith@gregsmith.com>
2008-04-18 14:38 ` Enrico Sirola <enrico.sirola@gmail.com>
2008-04-15 13:27 ` Florian Weimer <fweimer@bfk.de>
2008-04-15 19:08 ` david@lang.hm
This inbox is served by DDX for PostgreSQL; see mirroring instructions
for how to clone and mirror all data and code used for this inbox